Skip to content

Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306

Closed

Description

@timohueser

Status: done

Part of #3227.
Depends on: none

TL;DR

Goal: A restart of the follower, or a stop of one service, installs nothing again. The follower runs the services of at most two planner releases and fetches one release at a time. The page cache of the server can use its RAM.
Problem:

  • The follower deletes every installed file at each start and stop, and when one service of a planner release stops. Then it installs everything again on its poll thread. EPIC — Scale readiness: 100 concurrent users, long routes, from one region to the world #3000 estimates 30 minutes of recovery at the target coverage. With format 10 the same code writes 45 to 73 GB (Europe) or 150 to 175 GB (world) again for each release at each start.
  • It runs the services of three planner releases and keeps their files. At world size, three installed releases and the transport of the third need about 800 to 870 GB of a disk of about 930 GB (see Facts).
  • It downloads one object after another: EPIC — Scale readiness: 100 concurrent users, long routes, from one region to the world #3000 estimates 3 to 6 h for an update of 100 GB.
  • MemoryMax=9216M counts the page cache of memory maps, so the cache of all services together cannot pass 9 GB, whatever the RAM of the server.
    Decision:
  • An install lives in <store>/serve/<target>/<planner id>/ and ends with a marker that the fetch thread writes last. The follower deletes an install only when no wanted release, no current release and no running service names its planner release. A start deletes only the content of a folder without a marker.
  • One fetch runs at a time, for the first release of the list current, serving, next, rollback that has no install. While planner/next.json names a release, the rollback does not fetch. The fetch downloads with 8 requests in parallel, installs, and writes the marker; it stops between two files when no release wants it or serve stops. The poll thread only starts processes.
  • The services of the first two different planner releases of the same list that have an install run. The files of the rollback stay while the disk check allows them.
  • Before an install, a disk check compares the free space with the bytes that the install needs. When they do not fit, the follower deletes the files of the rollback; when they still do not fit, the release fails with both numbers.
  • Each service has its own state and back-off. A search fault does not stop routing, and no stop deletes files. While a service of a known release does not answer, its API answers 503 busy.
  • The unit: MemoryMax=90% of the RAM, OOMPolicy=continue, no CPUQuota, and the TasksMax of the systemd default.
    Rejected:
  • Keep three installed releases always. At world size they need about 800 to 870 GB during an install, with too little margin on a disk of about 930 GB.
  • Keep two always. At Europe size three need 300 to 390 GB, and the third gives a rollback after a failed update with no download. The disk check decides at each install from the real sizes, not from a constant for each region.
  • Fetches in parallel for several releases (today). Two disk checks then count the same free space, and a rollback that the check deleted fetches again beside the next release.
  • A memory limit for each service process (RLIMIT_DATA) on develop. Node sizes its heap from the RAM and aborts at the limit, its parser child inherits the limit, and the limits of two releases are no partition of the RAM. On develop the graph is in the heap of the routing service, so the kernel picks the process with the most memory, which is the one that grows. Follower: install format 10, and limits for memory maps #3307 decides a cap for the case of memory maps.
  • MemoryHigh near the RAM. The page cache of read-only maps is clean and cheap to reclaim; MemoryHigh adds throttling and gives no protection that MemoryMax does not give.
  • Install on the poll thread (today). The poll thread then stops all checks for the time of an install.
    Systems: obc-data
    Contracts: specs/obc-data.md (Serve, Router ready)
    Size: M (1 system, a contract change, about 600 changed lines)
    Branch: develop

Facts

  • serve removes <store>/serve/<target>/ when it starts and when it stops (serve_cli.rs#L139-L141, #L206). Processes removes the folder of its planner release when it drops (services.rs#L272-L277).
  • poll calls Load::start on the poll thread when a download ends; start installs (link or gunzip, then a SHA-256 of each file) and spawns (serve.rs#L295-L301, services.rs#L173-L214, #L428-L444). After a stop, serve waits GRACE (10 s) for the follower and then ends without it (serve_cli.rs#L194-L204); prune waits for each running or abandoned download (serve.rs#L353-L359).
  • One stopped service fails the whole planner release, and all its processes stop (serve.rs#L324-L327). The back-off is 60 s, doubling to 30 minutes (serve.rs#L28-L32). A service must answer within 120 s (serve.rs#L26).
  • The follower fetches and runs each of serving, rollback, next and current, each fetch on its own thread (serve.rs#L281-L294).
  • fetch checks the formats first, then downloads one object after another (services.rs#L113-L133). Each object download retries 4 times and resumes its .part file with a Range request (fetch/http.rs#L1-L2, #L48-L51). prune keeps the objects and the .part files of the wanted releases (services.rs#L157-L169), and the store <store>/follower/<target>/ survives a restart.
  • A file entry has transport: {bytes, sha256, encoding} (planner-release.md); Entry reads only sha256 and encoding (services.rs#L317-L328). The store and the install folders are both under <store>, so an identity file is a hard link and takes no more space (target.rs#L307-L322).
  • api answers 404 when a service of a planner release does not answer (serve/http.rs#L117-L130), and the phone reads 404 as a release that is gone (PlannerService.swift#L585).
  • obc data rollback asks the ready of the rollback once and refuses on any answer but 200 (rollback_cli.rs#L118-L127). An update holds the lock of the target while planner/next.json names its release, so no rollback runs during an update.
  • The unit has MemoryMax=9216M, CPUQuota=1800% and TasksMax=576, three times the legacy units of three services (obc-data-serve.service#L17-L22). systemd accepts MemoryMax as a percentage of the physical RAM. The default OOMPolicy=stop stops the whole unit when the kernel kills one of its processes. The routing service takes its worker count from available_parallelism(), which obeys a CPUQuota (main.rs#L20-L24).
  • Disk for each release, from the epic: installed routing 45 to 73 GB (Europe), 150 to 175 GB (world); published objects about 100 B for each road (Europe about 30 GB, world about 100 GB). Search: 51 to 56 GB at the target coverage of EPIC — Scale readiness: 100 concurrent users, long routes, from one region to the world #3000, plus 20 to 22 GB of gzip transport. World, with about 60 GB of search: three releases 630 to 705 GB, plus the transport of the third 120 GB, plus a reserve of 5 %: about 800 to 870 GB. Two releases with the transport of the second: 540 to 590 GB. Europe, three releases: 300 to 390 GB.

Plan

  1. host/obc-data/src/serve/services.rs: the install. install(store, dir, served, go) writes into dir = <store>/serve/<target>/<planner id>/ as today, then copies the document to release.json in it, calls libc::syncfs once on the folder, and writes the marker installed last (a temporary name, fsync, rename, fsync of the folder). The marker holds the planner id. Between two files, install stops with an error when go() is false. fn installed(dir) -> bool is true only with a marker that names the folder. Processes no longer owns the folder; its Drop only stops the child. The region comes from the release.json of the install, so a start reads no store object.
  2. services.rs: fetch downloads and installs, on the fetch thread. It calls readable first, as today. Then a folder with a valid marker returns at once. A folder without a marker loses its content, the missing objects download, and install runs. go is true while wanted is true and crate::fetch::http is not stopped (a new pub fn stopped() -> bool beside stop). Each phase of objects downloads its missing objects with PARALLEL = 8 threads (std::thread::scope) that take the next object from a shared list; the first error stops the others at their next object and is the result. Comment of PARALLEL: it hides the request time of objects of a few MB; 8 requests fill one link.
  3. services.rs: the disk check, in fetch after the document and the step indexes are in the store. need = for each runtime file, transport.bytes when the store lacks the object, plus bytes when the encoding is not identity; plus a reserve of 5 % of the file system (a design choice: logs, the downloads cache and the working space of the file system). free = f_bavail × f_frsize of libc::statvfs on <store>. When need > free, fetch returns Fetched::NoRoom { need, free } and downloads nothing. Entry.transport gets bytes: u64.
  4. host/obc-data/src/serve.rs: one fetch at a time. poll starts a fetch only when no fetch runs and no abandoned fetch is left. It takes the first planner release of the list current, serving, next, rollback that has no install and does not wait for its back-off; while wanted.next is set, it skips the rollback. A fetch of the rollback that runs when wanted.next appears is abandoned (its wanted goes false); its .part files and its folder without a marker stay, and it goes on after the next release. Load::fetch returns a thread of enum Fetched { Installed, NoRoom { need: u64, free: u64 }, Failed(String) }.
  5. serve.rs: NoRoom. When the release is not the rollback and an install exists whose planner release only the rollback names, poll stops the services of that install, deletes its folder, writes WARNING: no rollback during the install of <N>: deleted the files of planner release <R>; the disk has <free> GB free and the install needs <need> GB, and fetches again at its next poll. Else the release fails with the disk has <free> GB free; the install needs <need> GB, with the back-off.
  6. serve.rs: per-service states. State of a planner release becomes Fetching { worker, backoff, wanted } | Installed { services: [Service; 3] } | Failed { reason, retry, backoff }. A Service is Stopped | Starting { process, since } | Ready { process, since } | Failed { reason, retry }, each with its backoff. Load::start(served, service: usize) -> Result<S, String> spawns one process; the trait Services becomes Process with ready() -> Result<bool, String>, stopped() -> Option<String> and port() -> u16, implemented for one child in services.rs. A service that stops 60 s or more after its first answer starts again at once, with its back-off set to 60 s. A service that stops earlier, exits while it starts, or does not answer within READY, waits its back-off, which then doubles up to 30 minutes.
  7. serve.rs: two running releases. The first two different planner releases of the list current, serving, next, rollback that have an install run their services. The services of each other installed planner release stop (Stopped), and its files stay. Clients get the serving release once its three services answer, as today.
  8. serve.rs: deletion. prune deletes each folder in <store>/serve/<target>/ whose planner release is not in planners and that no wanted release names, after its services stopped. prune runs only after a look that read the target, so a start without a network deletes nothing. services::prune keeps each object that a wanted release names, except a gzip object of a release with a marker: an identity object is a link that takes no space, and the next release can need it.
  9. host/obc-data/src/cli/serve_cli.rs: delete the removal of <store>/serve/<target>/ at the start and the stop. Loader::fetch passes the install folder.
  10. host/obc-data/src/serve/http.rs: View gives for each planner release its state and the port of each service that answers. api forwards to a service that answers, also while another service of its release fails. For a planner release that the view has, a service that does not answer gives 503 {"code": "busy", "message": "The <service> service restarts. Try again shortly."}; 404 stays for a planner release that the view does not have. ready answers 200 when all three services answer; 503 {"state": "starting"} while it downloads or a service starts; 503 {"state": "installed"} while its services do not run; 503 {"state": "failed", "reason"} with the reason of each failed service or of the release.
  11. tools/vps/obc-data-serve.service: MemoryMax=90% and OOMPolicy=continue; delete CPUQuota and TasksMax. The comment: the page cache of the services counts for the unit, so the unit may use the RAM of the machine less a tenth for the system (sshd, Caddy, journald); when the kernel kills one process, only that service stops and the follower starts it again; the routing workers follow the CPUs of the machine; the tasks follow the systemd default.
  12. specs/obc-data.md, Serve and Router ready: the install folder and its marker, the start, one fetch at a time, the two running releases, the disk check and its warning, the per-service failures and back-off, 503 busy and the state installed, the 8 parallel downloads. tools/vps/README.md: the follower keeps the installed files in /var/lib/obc-data/store/serve/prod/; a rollback after a failed update downloads nothing while the disk kept its files.

Files

  • host/obc-data/src/serve.rs: change: one fetch at a time, Fetched, per-service states, the two running releases, the disk response, the deletion, the tests.
  • host/obc-data/src/serve/services.rs: change: the install with its marker and stops, the fetch with 8 parallel downloads, the disk check, one process for each service, the prune rule.
  • host/obc-data/src/serve/http.rs: change: the view, api and ready.
  • host/obc-data/src/fetch/http.rs: change: stopped().
  • host/obc-data/src/cli/serve_cli.rs: change: no removal of the install folder; the loader.
  • tools/vps/obc-data-serve.service: change: the limits and their comment.
  • tools/vps/README.md, specs/obc-data.md: change.

Done when

  • New tests in serve.rs, each through the Published case with the real install:
    • an_install_stays_across_a_restart_and_only_an_unmarked_one_installs_again: a second Follower on the same store starts the release with no object request and no write; a folder without a marker installs again.
    • a_stopped_service_restarts_alone_and_keeps_its_files: the search service exits; routing keeps answering through api; search answers 503 busy and then starts again; the folder stays. A second exit within 60 s waits the back-off.
    • during_an_update_the_rollback_stops_and_keeps_its_files: with a next release, the services of serving and next run, those of the rollback stop with the state installed, and the rollback does not fetch; when planner/next.json goes, the rollback starts with no object request.
    • an_install_that_does_not_fit_deletes_the_rollback_once: a fake free makes the next release fit only without the rollback; the rollback folder goes, the warning names both sizes, the next release installs, and the rollback does not fetch while the next release is named. When a release does not fit with no rollback, it fails with that reason.
  • The existing tests of serve.rs and services.rs pass; they change only where a stop deleted files or two fetches ran at once.
  • obc test -p obc-data and obc exec cargo clippy -p obc-data --all-targets -- -D warnings.
  • The pull request records, on the owner PC under nice with obc data serve local and a Central Switzerland release: the time from the start of serve to ready 200, the first time and after a restart, before and after the change.
  • Owner: after obc data deploy services, journalctl -u obc-data-serve shows one start of each service, and no install after a second systemctl restart obc-data-serve.

Out of scope

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions