Skip to content

Follower: install format 10, and limits for memory maps #3307

Closed

Description

@timohueser

Status: done

Part of #3227.
Depends on: #3301, #3306

TL;DR

Goal: The follower installs the routing package of format 10 of a planner release with the installer of #3301, in a folder of its own, checks the disk for it, and keeps one runaway service from taking the memory of the route service.
Problem:

Facts

Plan

  1. host/obc-data/src/serve/services.rs, fetch: when the runtime files have routing/directory.bin, the first phase after the step indexes downloads routing/package.json and routing/directory.bin. The disk check of Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306 then counts, for the routing package, the object_bytes of each object of install::plan(all cells, all groups) that the store lacks, plus install::installed_bytes of the parts whose file route-cells/cells/ lacks with its final name. The other runtime files count as in Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306.
  2. services.rs, install: the files routing/package.json, routing/directory.bin and routing/objects/* are not linked into routing/. After the other files, the fetch thread sets its nice to 10 (libc::setpriority on its thread). The installer of Route package format 10: the contract, the writer and the installer #3301 spawns its own threads in the call, and only threads made after the change inherit nice 10; a thread pool made before it would keep its priority and calls install::install with the target <planner folder>/route-cells/, all cells, all profile groups, 2 threads, and a fetch that opens the store object. An error of the installer fails the install with its message. The marker of Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306 comes after it.
  3. services.rs, fetch: a planner folder without the marker of Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306 loses its content except route-cells/.
  4. services.rs, prune: an object of routing/objects/ goes once the install of its release has the marker of Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306.
  5. services.rs, start: the route service gets ROUTE_CELLS=<planner folder>/route-cells when the folder exists, beside the folder routing/ of today. When the folder exists and planner-service --formats lists planner/routing-cells (Route engine: the overlay search with the cost tables, and unpacking of the chosen legs only #3305 adds that entry when its service reads ROUTE_CELLS), spawn sets RLIMIT_DATA of the route service to (available_parallelism() + 1) GiB with libc::setrlimit in pre_exec: 1 GiB for each worker (the gate of the epic) and 1 GiB for the shared part. Else the route service has no cap. Search and downloads write 1000 to /proc/self/oom_score_adj in pre_exec with libc::open and libc::write.
  6. The map limit. tools/vps/obc-data.sysctl (new): vm.max_map_count = 1048576. Its comment: the route service maps one file for each part of each cell and keeps the maps; 1,048,576 holds the largest install of Proof on Germany: the bake, the level sizes, the download level, the old engine against the new one #3310 (250,000 files) plus 4,096 four times, and a map costs kernel memory only while a process holds it. tools/vps/deploy-services.sh, the switch step: install it as /etc/sysctl.d/90-obc-data.conf and run sysctl -p on it, before the restart of the follower. host/obc-data/src/cli/deploy_cli/checks.rs: PROBE prints maps when sysctl -n vm.max_map_count is at least 1,048,576, and services gets the check vm.max_map_count with the fix scp tools/vps/obc-data.sysctl HOST:/etc/sysctl.d/90-obc-data.conf and sysctl --system. The checks come from the probe lines through a function that a test calls with no SSH. When the limit is too low, the route service refuses to start with the message of Route engine: the data layer on format 10 and the host interface #3302; the follower gives that message as the reason of the release failure, and the back-off of Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306 starts it with no download once the limit is raised.
  7. The open files: no change. The host of Route engine: the data layer on format 10 and the host interface #3302 closes each file after its map, and the installer holds a few files, so the soft limit of 1,024 holds.
  8. specs/obc-data.md, Serve: the folder route-cells/ and its installer, the disk check of format 10, the eviction from the cache, the limit of the route service, the order in which the kernel kills a service, and the map limit. tools/vps/README.md: the sysctl file and the failure when the limit is too low.

Files

  • host/obc-data/src/serve/services.rs: change: the routing install, its disk check, its resume, the prune of its objects, ROUTE_CELLS, the limits.
  • tools/vps/obc-data.sysctl: create: the map limit.
  • tools/vps/deploy-services.sh, tools/vps/README.md: change: install the sysctl file; the reference table.
  • host/obc-data/src/cli/deploy_cli/checks.rs: change: the probe, the check of the map limit, a test.
  • specs/obc-data.md: change: Serve.

Done when

Out of scope

Built differently

Pull request #3398 built this design, into routing-overlay, with these changes:

  • The installer runs as a child process, planner-service <install>/route-cells --install (and --missing), not inside the follower. host/obc-data is MIT/Apache and tools/check_obc_data_licence.py stops it from linking the GPL crate planner-router; the follower on the server is built from obc-data alone. The child runs at nice 10 and stops when the follower stops or dies.
  • The worker count has one source. planner-service --workers prints the count by the rule of the service, and the follower sets the data limit of the routing service to that count plus 1 GiB.
  • A damaged object is fetched again. The installer exits with 3 and names the object; the follower deletes that one object from its store. It deletes one object for each failed install, and each retry waits longer.
  • The prune also deletes routing/directory.bin and routing/package.json from the store, because the installer keeps a copy of each in route-cells/.
  • When a service or the installer fails, the reason ends with its last line of standard error, by the rule of engine::ended: no line after a signal and no empty line.
  • The deploy installs the sysctl file and the unit before the switch of current, and applies the sysctl value only when the kernel has a lower one.
  • The test of "Done when" is in two parts: a follower test with a script in place of the routing program, and a planner-service test with the real installer on a small package.
  • Open, for the owner on the server after the deploy: obc data deploy shows the check vm.max_map_count as passed; the routing service has its data limit and search and downloads have oom_score_adj 1000; the fio run on the candidate server. The data limit takes effect when Route engine: the overlay search with the cost tables, and unpacking of the chosen legs only #3305 adds planner/routing-cells to --formats.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions