Repository navigation
Closed
Description
Status: done
Size: S
Part of workstream 7 of #3000.
TL;DR
Goal: Each routing service runs one worker for each CPU that it may use, up to a worker limit whose busy memory is measured to fit the routing share of the follower unit. ROUTE_WORKERS can set a lower count.
Problem: Production runs 2 routing workers in each release, because nothing sets ROUTE_WORKERS and the service default is 2. The follower unit allows 18 CPUs. Requests get 503 busy from about 10 req/s. The present limit of 8 workers has no source: nothing shows that 8 busy workers fit in memory.
Decision:
- When
ROUTE_WORKERSis unset,main.rsuses the smaller ofstd::thread::available_parallelism()andMAX_WORKERS. Whenavailable_parallelism()fails, it uses 1, ashost/obc-data/src/engine/runs.rsdoes. On Linux with cgroup v2,available_parallelism()obeys theCPUQuotaof the unit and the CPU affinity of the process. MAX_WORKERSinhttp.rsreplaces the limit of 8. It is the largest worker count whose peak RSS stays below the 2048 MiB that the follower unit gives a routing service, when every worker is busy with the heaviest allowed requests. A measurement sets it (see Plan step 1).apprefuses 0 workers and more thanMAX_WORKERSworkers, also whenROUTE_WORKERSsets the count.- The service writes its worker count in its "listening" log line.
Rejected:
- No upper limit. Each busy worker fills its own label blocks, search queues (up to 256 MiB of its budget) and page caches. With 18 CPUs, 18 busy workers per release share one
MemoryMax=9216Mwith all other services. Above that limit the kernel kills a process, systemd's defaultOOMPolicy=stopstops the whole follower, and every release installs again. Two clients that loop the heaviest allowed request make every worker busy, so this case is not rare. - Keep the limit of 8. No measurement shows that 8 busy workers fit.
Environment=ROUTE_WORKERS=Nin the unit. N must be chosen again for each server, and that is why 2 stayed. The core count of the VPS is not known, and the Contabo server is not chosen yet.- A count for each release from the follower: many workers for the serving release, 1 for the next release and the rollback. The follower must then restart the routing service when it promotes a release. An idle worker costs little memory, so the saving is small.
Systems: routing
Contracts: none
Size: S (1 system, about 20 changed lines, no contract change)
Facts
planner/service/src/main.rsreadsROUTE_WORKERSand uses 2 when it is unset.appinplanner/service/src/http.rsrefuses a count outside 1 to 8.- The follower does not set the variable (
host/obc-data/src/serve/services.rs), and the unit does not set it (tools/vps/obc-data-serve.service). The child processes inherit the environment of the unit. - The 2 comes from the legacy routing unit, which had
CPUQuota=200%. The follower unit runs the services of three releases withCPUQuota=1800%andMemoryMax=9216M. Its comment gives 2048 MiB to each routing service. - A worker shares the graph and the prepared profiles with the other workers. It has its own label blocks, queues and page caches, which it fills only when it routes (engine bounds). The next release and the rollback get little traffic, so their workers stay small. The serving release can have every worker busy.
Plan
- Measure, before the code change. Use the routing package of the production release and a release build of
planner-service. For each worker count from 2 up, send the heaviest allowed requests for 60 s, at low priority, with at least twice as many concurrent clients as workers. The heaviest requests are the longest route in the package withalternatives: true, and a shape request with the longest allowed line. Record the peak RSS of the service, the rate of503 busy, and the median and P95 latency.MAX_WORKERSis the largest count whose peak RSS stays below 2048 MiB. Stop at the first count that goes over, or at the CPU count of the machine. planner/service/src/http.rs: addMAX_WORKERSwith the value from step 1. Its comment says that it is the largest count whose busy peak RSS fits the 2048 MiB thattools/vps/obc-data-serve.servicegives a routing service, measured on the routing package of the production release. The figures go in the pull request, not in the comment.apprefuses 0 and more thanMAX_WORKERS, with the message "Use 1 to {MAX_WORKERS} workers".planner/service/src/main.rs: whenROUTE_WORKERSis unset, use the smaller ofavailable_parallelism()(or 1 when it fails) andMAX_WORKERS. Add the worker count to the line "Routing service listening on …".planner/service/README.md: theROUTE_WORKERSrow gives the default as the number of CPUs that the process may use, which includes a systemdCPUQuota, up to the limit of the service. It says that the limit comes from a memory measurement, and that a server with more memory needs a new measurement and a higherMemoryMaxtogether.
Files
planner/service/src/http.rs: change:MAX_WORKERSand the range check of the worker count.planner/service/src/main.rs: change: the default worker count and the log line.planner/service/README.md: change: the reference row ofROUTE_WORKERS.
Done when
obc test -p planner-servicepasses.cargo clippy -p planner-service --all-targets -- -D warningspasses, also with--no-default-features.- The pull request records the measurement of Plan step 1 for each worker count, with the package and the commit of the build.
- If fewer than 8 workers fit, the pull request says so, because the present limit of 8 is then also unsafe.
- After
obc data deploy services,journalctl -u obc-data-serveon the VPS shows the worker count.
Out of scope
- The 1 s queue wait before
503 busy, and a separate pool for alternatives. They are other items of EPIC — Scale readiness: 100 concurrent users, long routes, from one region to the world #3000. - A change to the unit limits. A server with more memory changes
MemoryMaxandMAX_WORKERStogether, after a new measurement.
Prototype
None.
Metadata
Metadata
Assignees
Labels
No labels
Activity
[-]Routing service: one worker for each CPU that it may use[/-][+]Routing service: one worker for each CPU, up to a measured memory limit[/+]