Skip to content

Routing service: one worker for each CPU, up to a measured memory limit #3053

Closed

Description

@timohueser

Status: done
Size: S

Part of workstream 7 of #3000.

TL;DR

Goal: Each routing service runs one worker for each CPU that it may use, up to a worker limit whose busy memory is measured to fit the routing share of the follower unit. ROUTE_WORKERS can set a lower count.
Problem: Production runs 2 routing workers in each release, because nothing sets ROUTE_WORKERS and the service default is 2. The follower unit allows 18 CPUs. Requests get 503 busy from about 10 req/s. The present limit of 8 workers has no source: nothing shows that 8 busy workers fit in memory.
Decision:

  • When ROUTE_WORKERS is unset, main.rs uses the smaller of std::thread::available_parallelism() and MAX_WORKERS. When available_parallelism() fails, it uses 1, as host/obc-data/src/engine/runs.rs does. On Linux with cgroup v2, available_parallelism() obeys the CPUQuota of the unit and the CPU affinity of the process.
  • MAX_WORKERS in http.rs replaces the limit of 8. It is the largest worker count whose peak RSS stays below the 2048 MiB that the follower unit gives a routing service, when every worker is busy with the heaviest allowed requests. A measurement sets it (see Plan step 1).
  • app refuses 0 workers and more than MAX_WORKERS workers, also when ROUTE_WORKERS sets the count.
  • The service writes its worker count in its "listening" log line.

Rejected:

  • No upper limit. Each busy worker fills its own label blocks, search queues (up to 256 MiB of its budget) and page caches. With 18 CPUs, 18 busy workers per release share one MemoryMax=9216M with all other services. Above that limit the kernel kills a process, systemd's default OOMPolicy=stop stops the whole follower, and every release installs again. Two clients that loop the heaviest allowed request make every worker busy, so this case is not rare.
  • Keep the limit of 8. No measurement shows that 8 busy workers fit.
  • Environment=ROUTE_WORKERS=N in the unit. N must be chosen again for each server, and that is why 2 stayed. The core count of the VPS is not known, and the Contabo server is not chosen yet.
  • A count for each release from the follower: many workers for the serving release, 1 for the next release and the rollback. The follower must then restart the routing service when it promotes a release. An idle worker costs little memory, so the saving is small.

Systems: routing
Contracts: none
Size: S (1 system, about 20 changed lines, no contract change)

Facts

  • planner/service/src/main.rs reads ROUTE_WORKERS and uses 2 when it is unset. app in planner/service/src/http.rs refuses a count outside 1 to 8.
  • The follower does not set the variable (host/obc-data/src/serve/services.rs), and the unit does not set it (tools/vps/obc-data-serve.service). The child processes inherit the environment of the unit.
  • The 2 comes from the legacy routing unit, which had CPUQuota=200%. The follower unit runs the services of three releases with CPUQuota=1800% and MemoryMax=9216M. Its comment gives 2048 MiB to each routing service.
  • A worker shares the graph and the prepared profiles with the other workers. It has its own label blocks, queues and page caches, which it fills only when it routes (engine bounds). The next release and the rollback get little traffic, so their workers stay small. The serving release can have every worker busy.

Plan

  1. Measure, before the code change. Use the routing package of the production release and a release build of planner-service. For each worker count from 2 up, send the heaviest allowed requests for 60 s, at low priority, with at least twice as many concurrent clients as workers. The heaviest requests are the longest route in the package with alternatives: true, and a shape request with the longest allowed line. Record the peak RSS of the service, the rate of 503 busy, and the median and P95 latency. MAX_WORKERS is the largest count whose peak RSS stays below 2048 MiB. Stop at the first count that goes over, or at the CPU count of the machine.
  2. planner/service/src/http.rs: add MAX_WORKERS with the value from step 1. Its comment says that it is the largest count whose busy peak RSS fits the 2048 MiB that tools/vps/obc-data-serve.service gives a routing service, measured on the routing package of the production release. The figures go in the pull request, not in the comment. app refuses 0 and more than MAX_WORKERS, with the message "Use 1 to {MAX_WORKERS} workers".
  3. planner/service/src/main.rs: when ROUTE_WORKERS is unset, use the smaller of available_parallelism() (or 1 when it fails) and MAX_WORKERS. Add the worker count to the line "Routing service listening on …".
  4. planner/service/README.md: the ROUTE_WORKERS row gives the default as the number of CPUs that the process may use, which includes a systemd CPUQuota, up to the limit of the service. It says that the limit comes from a memory measurement, and that a server with more memory needs a new measurement and a higher MemoryMax together.

Files

  • planner/service/src/http.rs: change: MAX_WORKERS and the range check of the worker count.
  • planner/service/src/main.rs: change: the default worker count and the log line.
  • planner/service/README.md: change: the reference row of ROUTE_WORKERS.

Done when

  • obc test -p planner-service passes.
  • cargo clippy -p planner-service --all-targets -- -D warnings passes, also with --no-default-features.
  • The pull request records the measurement of Plan step 1 for each worker count, with the package and the commit of the build.
  • If fewer than 8 workers fit, the pull request says so, because the present limit of 8 is then also unsafe.
  • After obc data deploy services, journalctl -u obc-data-serve on the VPS shows the worker count.

Out of scope

Prototype

None.

Activity

  1. changed the title [-]Routing service: one worker for each CPU that it may use[/-] [+]Routing service: one worker for each CPU, up to a measured memory limit[/+] on Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions