Repository navigation
Closed
Description
Status: done
Part of #3227.
Depends on: #3301, #3306
TL;DR
Goal: The follower installs the routing package of format 10 of a planner release with the installer of #3301, in a folder of its own, checks the disk for it, and keeps one runaway service from taking the memory of the route service.
Problem:
- The install of Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306 links or gunzips each file. A package of format 10 is a set of compressed cell objects that only the installer of Route package format 10: the contract, the writer and the installer #3301 decodes, and an install of 150 to 175 GB (world) takes long enough that it must go on after a stop.
- On the branch,
routing/of a release holds format 9 andoverlays.sqlitebeside the files of format 10 (planner-release.md#L56), and the layer format stays 9 until Format 10 becomes the only format: delete the old engine and format 9, merge routing-overlay into develop #3311. - An install writes 150 to 175 GB through a page cache of 32 to 64 GB in the same unit, which evicts the files of the serving release.
- With memory maps, the RSS of the route service counts its mapped file pages. When the unit runs out of memory, the kernel kills the process with the largest RSS: the route service, and not the service that grows.
Decision: - A planner release whose file list has
routing/directory.binhas a package of format 10. The follower gives the installer of Route package format 10: the contract, the writer and the installer #3301 onlyrouting/package.json,routing/directory.binand the objectsrouting/objects/*, with all cells and all profile groups, and the target<planner folder>/route-cells/. The other files install as in Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306. - A planner folder without the marker of Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306 keeps
route-cells/, and the follower calls the installer again: it goes on from its complete files. - The disk check fetches
package.jsonanddirectory.binfirst.needadds theobject_bytesof the objects that the store lacks and theinstalled_bytesof the parts thatroute-cells/lacks. - The install runs on 2 threads at nice 10, and the installer drops each file from the page cache after its sync.
- The object of a cell goes from the store once its install has the marker of Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306, because the installer decodes it.
- The route service gets the limit
RLIMIT_DATA = (CPUs + 1) GiBonly when it readsroute-cells/; before that, on the branch, it runs the engine of format 9 with the graph in its heap. Search and downloads getoom_score_adj = 1000, so the kernel kills one of them first. - The server sets
vm.max_map_count = 1048576through a sysctl file oftools/vps/, which the deploy installs and the setup check ofobc data deploychecks. When an install has more files than the limit allows, the route service refuses to start (Route engine: the data layer on format 10 and the host interface #3302), and the follower gives that message as the reason of the release failure. The limit of open files stays.
Rejected: - The map limit set by the follower or the unit. It is a setting of the kernel for the whole machine; the follower runs without root, and systemd has no setting for it.
- A map limit computed from the release. The kernel counts a map in its own memory only when a process holds it, so one fixed value that holds the largest download level of Proof on Germany: the bake, the level sizes, the download level, the old engine against the new one #3310 with room costs nothing.
routing/as the target of the installer. The installer deletes only incells/, but the folder then mixes two formats, and the format 9 reader and the installer write in one folder.- A trigger on the layer format. It stays 9 until Format 10 becomes the only format: delete the old engine and format 9, merge routing-overlay into develop #3311, so the trigger never fires on the branch.
RLIMIT_DATAfor each service. Node sizes its heap from the RAM and aborts at the limit, and its parser child inherits it (Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306). The search service needs no cap of its own once the kernel picks it first.--max-old-space-sizefor Node. It caps only the V8 heap, not SQLite and the model, and the kernel choice already protects the route service.- A warm-up of the page cache before
ready. The first requests fill the cache; a warm-up reads files that no rider asks for.
Systems: obc-data
Contracts:specs/obc-data.md(Serve: the routing install, the limits)
Size: S (1 system, about 250 changed lines)
Branch:routing-overlay
Facts
- The installer of Route package format 10: the contract, the writer and the installer #3301 (
planner/router/src/install.rs):plan(directory, cells, groups),installed_bytes(plan), andinstall(target, directory, info, cells, groups, threads, fetch), wherefetchgives a reader of an object by its SHA-256. It writesNAME.part, checks the object and the part, syncs and renames, so a second call goes on after a stop. It writesinstall.binlast, deletes only incells/, takes a lock on the target, and refuses a target with theinstall.binof another directory. A reader opens only a folder withinstall.bin(Route package format 10: the contract, the writer and the installer #3301, rule 4). Each part entry of the directory hasobject_bytesandbytes. - Route package format 10: the contract, the writer and the installer #3301 lists the objects as
identityfilesrouting/objects/SHA256, besiderouting/package.jsonandrouting/directory.bin, and keeps the layer format at 9 until Format 10 becomes the only format: delete the old engine and format 9, merge routing-overlay into develop #3311. - Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306 deletes the content of a planner folder without its marker, and keeps each
identityobject while a wanted release names it, because a link takes no space. RLIMIT_DATAcounts private writable maps (since Linux 4.7), not maps that are read-only or shared. The epic gives the route service of format 10 less than 1 GB of its own memory for each worker (Done when, Proof on Germany: the bake, the level sizes, the download level, the old engine against the new one #3310).try_reserveof the engine givesError::Memoryat the limit, which Route service: request classes, the limit error as status 422, the probe radius #3308 answers with503 busy; another allocation aborts the process, and Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306 starts it again.- A process may raise its own
oom_score_adjwith no privilege. The kernel picks the process with the largest RSS plusoom_score_adjthousandths of the memory limit, and the RSS of the route service counts its mapped pages, which can approach the unit limit; only 1000 puts search and downloads first. - The host of Route engine: the data layer on format 10 and the host interface #3302 maps one file for each part of each cell, read-only and shared, at its first use, closes the file, and keeps the map (#3302, the host interface). So the maps of one route service grow to the file count of its install, and its open files stay few.
Maps::newrefuses an install whose files plus 4,096 pass/proc/sys/vm/max_map_count(Route engine: the data layer on format 10 and the host interface #3302). Files: about 9,000 for Europe and 35,000 for the world at the default download level; Proof on Germany: the bake, the level sizes, the download level, the old engine against the new one #3310 chooses the lowest download level whose estimated world file count is at most 250,000. The Linux default limit is 65,530 maps for each process; the systemd default soft limit of open files is 1,024. - The installer of Route package format 10: the contract, the writer and the installer #3301 holds about two files open for each thread.
obc data deploychecks the one-time setup of the VPS with one SSH probe and prints a fix for each missing step (checks.rs#L204-L214, #L235-L290).deploy-services.shinstalls the unit of its commit as root at each deploy (deploy-services.sh#L126-L135).
Plan
host/obc-data/src/serve/services.rs,fetch: when the runtime files haverouting/directory.bin, the first phase after the step indexes downloadsrouting/package.jsonandrouting/directory.bin. The disk check of Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306 then counts, for the routing package, theobject_bytesof each object ofinstall::plan(all cells, all groups)that the store lacks, plusinstall::installed_bytesof the parts whose fileroute-cells/cells/lacks with its final name. The other runtime files count as in Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306.services.rs,install: the filesrouting/package.json,routing/directory.binandrouting/objects/*are not linked intorouting/. After the other files, the fetch thread sets its nice to 10 (libc::setpriorityon its thread). The installer of Route package format 10: the contract, the writer and the installer #3301 spawns its own threads in the call, and only threads made after the change inherit nice 10; a thread pool made before it would keep its priority and callsinstall::installwith the target<planner folder>/route-cells/, all cells, all profile groups, 2 threads, and afetchthat opens the store object. An error of the installer fails the install with its message. The marker of Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306 comes after it.services.rs,fetch: a planner folder without the marker of Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306 loses its content exceptroute-cells/.services.rs,prune: an object ofrouting/objects/goes once the install of its release has the marker of Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306.services.rs,start: the route service getsROUTE_CELLS=<planner folder>/route-cellswhen the folder exists, beside the folderrouting/of today. When the folder exists andplanner-service --formatslistsplanner/routing-cells(Route engine: the overlay search with the cost tables, and unpacking of the chosen legs only #3305 adds that entry when its service readsROUTE_CELLS),spawnsetsRLIMIT_DATAof the route service to(available_parallelism() + 1) GiBwithlibc::setrlimitinpre_exec: 1 GiB for each worker (the gate of the epic) and 1 GiB for the shared part. Else the route service has no cap. Search and downloads write1000to/proc/self/oom_score_adjinpre_execwithlibc::openandlibc::write.- The map limit.
tools/vps/obc-data.sysctl(new):vm.max_map_count = 1048576. Its comment: the route service maps one file for each part of each cell and keeps the maps; 1,048,576 holds the largest install of Proof on Germany: the bake, the level sizes, the download level, the old engine against the new one #3310 (250,000 files) plus 4,096 four times, and a map costs kernel memory only while a process holds it.tools/vps/deploy-services.sh, the switch step: install it as/etc/sysctl.d/90-obc-data.confand runsysctl -pon it, before the restart of the follower.host/obc-data/src/cli/deploy_cli/checks.rs:PROBEprintsmapswhensysctl -n vm.max_map_countis at least 1,048,576, andservicesgets the checkvm.max_map_countwith the fixscp tools/vps/obc-data.sysctl HOST:/etc/sysctl.d/90-obc-data.confandsysctl --system. The checks come from the probe lines through a function that a test calls with no SSH. When the limit is too low, the route service refuses to start with the message of Route engine: the data layer on format 10 and the host interface #3302; the follower gives that message as the reason of the release failure, and the back-off of Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306 starts it with no download once the limit is raised. - The open files: no change. The host of Route engine: the data layer on format 10 and the host interface #3302 closes each file after its map, and the installer holds a few files, so the soft limit of 1,024 holds.
specs/obc-data.md, Serve: the folderroute-cells/and its installer, the disk check of format 10, the eviction from the cache, the limit of the route service, the order in which the kernel kills a service, and the map limit.tools/vps/README.md: the sysctl file and the failure when the limit is too low.
Files
host/obc-data/src/serve/services.rs: change: the routing install, its disk check, its resume, the prune of its objects,ROUTE_CELLS, the limits.tools/vps/obc-data.sysctl: create: the map limit.tools/vps/deploy-services.sh,tools/vps/README.md: change: install the sysctl file; the reference table.host/obc-data/src/cli/deploy_cli/checks.rs: change: the probe, the check of the map limit, a test.specs/obc-data.md: change: Serve.
Done when
a_routing_package_of_format_10_installs_and_goes_on_after_a_stop, inservices.rs: a small package of Route package format 10: the contract, the writer and the installer #3301 in a planner release installs intoroute-cells/with theinstall.binof Route package format 10: the contract, the writer and the installer #3301 and the marker of Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306, androuting/holds no object. An install that stops after its first cell keeps that cell, and the next fetch writes only the other cells.a_vps_without_the_map_limit_gets_its_fix, inchecks.rs: probe lines withoutmapsgive the failed checkvm.max_map_countwith its fix; withmapsit passes.- Owner, after the deploy of Format 10 becomes the only format: delete the old engine and format 9, merge routing-overlay into develop #3311 to the production VPS (production stays on
developuntil then):obc data deployshows the checkvm.max_map_countas passed. obc test -p obc-dataandobc exec cargo clippy -p obc-data --all-targets -- -D warnings.
Out of scope
- The bytes of format 10 and the installer, with its eviction from the page cache after each sync: Route package format 10: the contract, the writer and the installer #3301. The engine that reads
route-cells/and the maps: Route engine: the data layer on format 10 and the host interface #3302. The use ofROUTE_CELLSby the route service, and its entryplanner/routing-cellsin--formats: Route engine: the overlay search with the cost tables, and unpacking of the chosen legs only #3305. Route engine: the data layer on format 10 and the host interface #3302 maps the files read-only and shared, because a private writable map counts againstRLIMIT_DATA. - The record of the memory of the route service under load, without the cap, with the count of aborts: Proof on Germany: the bake, the level sizes, the download level, the old engine against the new one #3310.
- The start time of the route service with a cold page cache on the Germany package: Proof on Germany: the bake, the level sizes, the download level, the old engine against the new one #3310.
fioon the candidate server. The owner records it on Routing: hierarchical search and disk storage #3227 before the server order: random reads of 4, 16 and 64 KiB at queue depth 1 and 32, at several hours, and one sequential write. It changes no line of Follower: install format 10, and limits for memory maps #3307. What each outcome changes:- Reads per second at depth 32 at least 4 times those at depth 1: the prefetch of all ranges of a path (Route engine: the overlay search with the cost tables, and unpacking of the chosen legs only #3305, through the
announcecall of the host of Route engine: the data layer on format 10 and the host interface #3302) is the lever. - Less than 1.5 times: the count of reads decides a cold route, so the layout of Route package format 10: the contract, the writer and the installer #3301 and the server choice are owner questions before Format 10 becomes the only format: delete the old engine and format 9, merge routing-overlay into develop #3311.
- A P99 at depth 1 above 5 ms at any hour: a cold leg of 2,000 km reads about 2,900 cells at the worst one after another; an owner question before the server order.
- The sequential write rate divided into the installed bytes of a world release is the install time of Follower: install format 10, and limits for memory maps #3307.
- Reads per second at depth 32 at least 4 times those at depth 1: the prefetch of all ranges of a path (Route engine: the overlay search with the cost tables, and unpacking of the chosen legs only #3305, through the
Built differently
Pull request #3398 built this design, into routing-overlay, with these changes:
- The installer runs as a child process,
planner-service <install>/route-cells --install(and--missing), not inside the follower.host/obc-datais MIT/Apache andtools/check_obc_data_licence.pystops it from linking the GPL crateplanner-router; the follower on the server is built fromobc-dataalone. The child runs at nice 10 and stops when the follower stops or dies. - The worker count has one source.
planner-service --workersprints the count by the rule of the service, and the follower sets the data limit of the routing service to that count plus 1 GiB. - A damaged object is fetched again. The installer exits with 3 and names the object; the follower deletes that one object from its store. It deletes one object for each failed install, and each retry waits longer.
- The prune also deletes
routing/directory.binandrouting/package.jsonfrom the store, because the installer keeps a copy of each inroute-cells/. - When a service or the installer fails, the reason ends with its last line of standard error, by the rule of
engine::ended: no line after a signal and no empty line. - The deploy installs the sysctl file and the unit before the switch of
current, and applies the sysctl value only when the kernel has a lower one. - The test of "Done when" is in two parts: a follower test with a script in place of the routing program, and a
planner-servicetest with the real installer on a small package. - Open, for the owner on the server after the deploy:
obc data deployshows the checkvm.max_map_countas passed; the routing service has its data limit and search and downloads haveoom_score_adj1000; thefiorun on the candidate server. The data limit takes effect when Route engine: the overlay search with the cost tables, and unpacking of the chosen legs only #3305 addsplanner/routing-cellsto--formats.
Metadata
Metadata
Assignees
Labels
No labels
Activity