Repository navigation
Closed
Description
Status: done
Part of #3227.
Depends on: none
TL;DR
Goal: A restart of the follower, or a stop of one service, installs nothing again. The follower runs the services of at most two planner releases and fetches one release at a time. The page cache of the server can use its RAM.
Problem:
- The follower deletes every installed file at each start and stop, and when one service of a planner release stops. Then it installs everything again on its poll thread. EPIC — Scale readiness: 100 concurrent users, long routes, from one region to the world #3000 estimates 30 minutes of recovery at the target coverage. With format 10 the same code writes 45 to 73 GB (Europe) or 150 to 175 GB (world) again for each release at each start.
- It runs the services of three planner releases and keeps their files. At world size, three installed releases and the transport of the third need about 800 to 870 GB of a disk of about 930 GB (see Facts).
- It downloads one object after another: EPIC — Scale readiness: 100 concurrent users, long routes, from one region to the world #3000 estimates 3 to 6 h for an update of 100 GB.
MemoryMax=9216Mcounts the page cache of memory maps, so the cache of all services together cannot pass 9 GB, whatever the RAM of the server.
Decision:- An install lives in
<store>/serve/<target>/<planner id>/and ends with a marker that the fetch thread writes last. The follower deletes an install only when no wanted release, no current release and no running service names its planner release. A start deletes only the content of a folder without a marker. - One fetch runs at a time, for the first release of the list current, serving, next, rollback that has no install. While
planner/next.jsonnames a release, the rollback does not fetch. The fetch downloads with 8 requests in parallel, installs, and writes the marker; it stops between two files when no release wants it or serve stops. The poll thread only starts processes. - The services of the first two different planner releases of the same list that have an install run. The files of the rollback stay while the disk check allows them.
- Before an install, a disk check compares the free space with the bytes that the install needs. When they do not fit, the follower deletes the files of the rollback; when they still do not fit, the release fails with both numbers.
- Each service has its own state and back-off. A search fault does not stop routing, and no stop deletes files. While a service of a known release does not answer, its API answers
503 busy. - The unit:
MemoryMax=90%of the RAM,OOMPolicy=continue, noCPUQuota, and theTasksMaxof the systemd default.
Rejected: - Keep three installed releases always. At world size they need about 800 to 870 GB during an install, with too little margin on a disk of about 930 GB.
- Keep two always. At Europe size three need 300 to 390 GB, and the third gives a rollback after a failed update with no download. The disk check decides at each install from the real sizes, not from a constant for each region.
- Fetches in parallel for several releases (today). Two disk checks then count the same free space, and a rollback that the check deleted fetches again beside the next release.
- A memory limit for each service process (
RLIMIT_DATA) ondevelop. Node sizes its heap from the RAM and aborts at the limit, its parser child inherits the limit, and the limits of two releases are no partition of the RAM. Ondevelopthe graph is in the heap of the routing service, so the kernel picks the process with the most memory, which is the one that grows. Follower: install format 10, and limits for memory maps #3307 decides a cap for the case of memory maps. MemoryHighnear the RAM. The page cache of read-only maps is clean and cheap to reclaim;MemoryHighadds throttling and gives no protection thatMemoryMaxdoes not give.- Install on the poll thread (today). The poll thread then stops all checks for the time of an install.
Systems: obc-data
Contracts:specs/obc-data.md(Serve, Router ready)
Size: M (1 system, a contract change, about 600 changed lines)
Branch:develop
Facts
serveremoves<store>/serve/<target>/when it starts and when it stops (serve_cli.rs#L139-L141, #L206).Processesremoves the folder of its planner release when it drops (services.rs#L272-L277).pollcallsLoad::starton the poll thread when a download ends;startinstalls (link or gunzip, then a SHA-256 of each file) and spawns (serve.rs#L295-L301, services.rs#L173-L214, #L428-L444). After a stop, serve waitsGRACE(10 s) for the follower and then ends without it (serve_cli.rs#L194-L204);prunewaits for each running or abandoned download (serve.rs#L353-L359).- One stopped service fails the whole planner release, and all its processes stop (serve.rs#L324-L327). The back-off is 60 s, doubling to 30 minutes (serve.rs#L28-L32). A service must answer within 120 s (serve.rs#L26).
- The follower fetches and runs each of serving, rollback, next and current, each fetch on its own thread (serve.rs#L281-L294).
fetchchecks the formats first, then downloads one object after another (services.rs#L113-L133). Each object download retries 4 times and resumes its.partfile with aRangerequest (fetch/http.rs#L1-L2, #L48-L51).prunekeeps the objects and the.partfiles of the wanted releases (services.rs#L157-L169), and the store<store>/follower/<target>/survives a restart.- A file entry has
transport: {bytes, sha256, encoding}(planner-release.md);Entryreads onlysha256andencoding(services.rs#L317-L328). The store and the install folders are both under<store>, so anidentityfile is a hard link and takes no more space (target.rs#L307-L322). apianswers 404 when a service of a planner release does not answer (serve/http.rs#L117-L130), and the phone reads 404 as a release that is gone (PlannerService.swift#L585).obc data rollbackasks thereadyof the rollback once and refuses on any answer but 200 (rollback_cli.rs#L118-L127). An update holds the lock of the target whileplanner/next.jsonnames its release, so no rollback runs during an update.- The unit has
MemoryMax=9216M,CPUQuota=1800%andTasksMax=576, three times the legacy units of three services (obc-data-serve.service#L17-L22). systemd acceptsMemoryMaxas a percentage of the physical RAM. The defaultOOMPolicy=stopstops the whole unit when the kernel kills one of its processes. The routing service takes its worker count fromavailable_parallelism(), which obeys aCPUQuota(main.rs#L20-L24). - Disk for each release, from the epic: installed routing 45 to 73 GB (Europe), 150 to 175 GB (world); published objects about 100 B for each road (Europe about 30 GB, world about 100 GB). Search: 51 to 56 GB at the target coverage of EPIC — Scale readiness: 100 concurrent users, long routes, from one region to the world #3000, plus 20 to 22 GB of gzip transport. World, with about 60 GB of search: three releases 630 to 705 GB, plus the transport of the third 120 GB, plus a reserve of 5 %: about 800 to 870 GB. Two releases with the transport of the second: 540 to 590 GB. Europe, three releases: 300 to 390 GB.
Plan
host/obc-data/src/serve/services.rs: the install.install(store, dir, served, go)writes intodir=<store>/serve/<target>/<planner id>/as today, then copies the document torelease.jsonin it, callslibc::syncfsonce on the folder, and writes the markerinstalledlast (a temporary name,fsync, rename,fsyncof the folder). The marker holds the planner id. Between two files,installstops with an error whengo()is false.fn installed(dir) -> boolis true only with a marker that names the folder.Processesno longer owns the folder; itsDroponly stops the child. The region comes from therelease.jsonof the install, so a start reads no store object.services.rs:fetchdownloads and installs, on the fetch thread. It callsreadablefirst, as today. Then a folder with a valid marker returns at once. A folder without a marker loses its content, the missing objects download, andinstallruns.gois true whilewantedis true andcrate::fetch::httpis not stopped (a newpub fn stopped() -> boolbesidestop). Each phase ofobjectsdownloads its missing objects withPARALLEL = 8threads (std::thread::scope) that take the next object from a shared list; the first error stops the others at their next object and is the result. Comment ofPARALLEL: it hides the request time of objects of a few MB; 8 requests fill one link.services.rs: the disk check, infetchafter the document and the step indexes are in the store.need= for each runtime file,transport.byteswhen the store lacks the object, plusbyteswhen the encoding is notidentity; plus a reserve of 5 % of the file system (a design choice: logs, the downloads cache and the working space of the file system).free=f_bavail × f_frsizeoflibc::statvfson<store>. Whenneed > free,fetchreturnsFetched::NoRoom { need, free }and downloads nothing.Entry.transportgetsbytes: u64.host/obc-data/src/serve.rs: one fetch at a time.pollstarts a fetch only when no fetch runs and no abandoned fetch is left. It takes the first planner release of the list current, serving, next, rollback that has no install and does not wait for its back-off; whilewanted.nextis set, it skips the rollback. A fetch of the rollback that runs whenwanted.nextappears is abandoned (itswantedgoes false); its.partfiles and its folder without a marker stay, and it goes on after the next release.Load::fetchreturns a thread ofenum Fetched { Installed, NoRoom { need: u64, free: u64 }, Failed(String) }.serve.rs:NoRoom. When the release is not the rollback and an install exists whose planner release only the rollback names,pollstops the services of that install, deletes its folder, writesWARNING: no rollback during the install of <N>: deleted the files of planner release <R>; the disk has <free> GB free and the install needs <need> GB, and fetches again at its next poll. Else the release fails withthe disk has <free> GB free; the install needs <need> GB, with the back-off.serve.rs: per-service states.Stateof a planner release becomesFetching { worker, backoff, wanted } | Installed { services: [Service; 3] } | Failed { reason, retry, backoff }. AServiceisStopped | Starting { process, since } | Ready { process, since } | Failed { reason, retry }, each with itsbackoff.Load::start(served, service: usize) -> Result<S, String>spawns one process; the traitServicesbecomesProcesswithready() -> Result<bool, String>,stopped() -> Option<String>andport() -> u16, implemented for one child inservices.rs. A service that stops 60 s or more after its first answer starts again at once, with its back-off set to 60 s. A service that stops earlier, exits while it starts, or does not answer withinREADY, waits its back-off, which then doubles up to 30 minutes.serve.rs: two running releases. The first two different planner releases of the list current, serving, next, rollback that have an install run their services. The services of each other installed planner release stop (Stopped), and its files stay. Clients get the serving release once its three services answer, as today.serve.rs: deletion.prunedeletes each folder in<store>/serve/<target>/whose planner release is not inplannersand that no wanted release names, after its services stopped.pruneruns only after a look that read the target, so a start without a network deletes nothing.services::prunekeeps each object that a wanted release names, except agzipobject of a release with a marker: anidentityobject is a link that takes no space, and the next release can need it.host/obc-data/src/cli/serve_cli.rs: delete the removal of<store>/serve/<target>/at the start and the stop.Loader::fetchpasses the install folder.host/obc-data/src/serve/http.rs:Viewgives for each planner release its state and the port of each service that answers.apiforwards to a service that answers, also while another service of its release fails. For a planner release that the view has, a service that does not answer gives 503{"code": "busy", "message": "The <service> service restarts. Try again shortly."}; 404 stays for a planner release that the view does not have.readyanswers 200 when all three services answer; 503{"state": "starting"}while it downloads or a service starts; 503{"state": "installed"}while its services do not run; 503{"state": "failed", "reason"}with the reason of each failed service or of the release.tools/vps/obc-data-serve.service:MemoryMax=90%andOOMPolicy=continue; deleteCPUQuotaandTasksMax. The comment: the page cache of the services counts for the unit, so the unit may use the RAM of the machine less a tenth for the system (sshd, Caddy, journald); when the kernel kills one process, only that service stops and the follower starts it again; the routing workers follow the CPUs of the machine; the tasks follow the systemd default.specs/obc-data.md, Serve and Router ready: the install folder and its marker, the start, one fetch at a time, the two running releases, the disk check and its warning, the per-service failures and back-off,503 busyand the stateinstalled, the 8 parallel downloads.tools/vps/README.md: the follower keeps the installed files in/var/lib/obc-data/store/serve/prod/; a rollback after a failed update downloads nothing while the disk kept its files.
Files
host/obc-data/src/serve.rs: change: one fetch at a time,Fetched, per-service states, the two running releases, the disk response, the deletion, the tests.host/obc-data/src/serve/services.rs: change: the install with its marker and stops, the fetch with 8 parallel downloads, the disk check, one process for each service, the prune rule.host/obc-data/src/serve/http.rs: change: the view,apiandready.host/obc-data/src/fetch/http.rs: change:stopped().host/obc-data/src/cli/serve_cli.rs: change: no removal of the install folder; the loader.tools/vps/obc-data-serve.service: change: the limits and their comment.tools/vps/README.md,specs/obc-data.md: change.
Done when
- New tests in
serve.rs, each through thePublishedcase with the real install:an_install_stays_across_a_restart_and_only_an_unmarked_one_installs_again: a secondFolloweron the same store starts the release with no object request and no write; a folder without a marker installs again.a_stopped_service_restarts_alone_and_keeps_its_files: the search service exits; routing keeps answering throughapi; search answers503 busyand then starts again; the folder stays. A second exit within 60 s waits the back-off.during_an_update_the_rollback_stops_and_keeps_its_files: with a next release, the services of serving and next run, those of the rollback stop with the stateinstalled, and the rollback does not fetch; whenplanner/next.jsongoes, the rollback starts with no object request.an_install_that_does_not_fit_deletes_the_rollback_once: a fakefreemakes the next release fit only without the rollback; the rollback folder goes, the warning names both sizes, the next release installs, and the rollback does not fetch while the next release is named. When a release does not fit with no rollback, it fails with that reason.
- The existing tests of
serve.rsandservices.rspass; they change only where a stop deleted files or two fetches ran at once. obc test -p obc-dataandobc exec cargo clippy -p obc-data --all-targets -- -D warnings.- The pull request records, on the owner PC under
nicewithobc data serve localand a Central Switzerland release: the time from the start of serve toready200, the first time and after a restart, before and after the change. - Owner: after
obc data deploy services,journalctl -u obc-data-serveshows one start of each service, and no install after a secondsystemctl restart obc-data-serve.
Out of scope
- The routing install of format 10, and a memory cap for one service with memory maps: Follower: install format 10, and limits for memory maps #3307.
- The worker count of the routing service and the request classes: Route service: request classes, the limit error as status 422, the probe radius #3308.
- The gzip copy of the routing packs (
tools/planner_offline.py): the packs go in Format 10 becomes the only format: delete the old engine and format 9, merge routing-overlay into develop #3311.
Metadata
Metadata
Assignees
Labels
No labels
Activity