Skip to content

EPIC — Scale readiness: 100 concurrent users, long routes, from one region to the world #3000

Open
5 of 6 issues completed

Description

@timohueser

Goal

The route planner and TrailView must serve 100 concurrent users on Europe, North America, Australia and New Zealand, and later the world. Riders must be able to plan long routes. A single leg of 2,000 km must answer in a few seconds.

This epic holds the audit, the owner decisions and the workstreams. Follow-up sessions investigate each workstream and create its sub-issues. Add each sub-issue to the list below.

Current state

Production serves Central Switzerland only: about 9,000 km², 1.2M directed roads, 7,215 files, 1.6 GB.

System Today At the target coverage, as built
Tile Worker and R2 Fits Fits. Abuse has no spending ceiling
TrailView client Fits Fits. Slow links are weak
Routing Busy errors from about 10 req/s Does not start
Search Fits Fails for all users
Data updates and manifests Fit Fail

How to read the numbers. "Measured" means a run on the production packages on an Apple M1 Pro, or a read of the live Cloudflare account. "Estimated" means an extrapolation from Central Switzerland and an older Baden-Württemberg package, ±40% or worse. No bake or test at continental size exists. No test ran on the VPS.

Cost (estimated)

Item Monthly
R2 storage, 1.4 TB mid case (0.8–2.3 TB) about $21 ($12–35)
Workers and R2 reads, 100 users 8 h/day, 80% planner, 20% light TrailView about $24, with the $5 plan
The same, everyone in heavy TrailView about $149
Server about 40–50 € (see Decisions)

Usage per hour of one user (estimated): the planner makes about 970 Worker requests and 1,440 R2 reads. TrailView makes 3,000–9,000 Worker requests and 3,300–9,900 R2 reads.

Owner decisions

  • Long routes are a requirement. The planner must make long routes possible. A few seconds for 2,000 km is acceptable. A route-length limit is not the answer.
  • Hierarchical search is in scope (workstream 4).
  • Server. The candidate is a Contabo server with 64 GB of RAM at about 40 €/month, with storage upgraded to 1 TB of SSD for about 8 € more. Test its disk before it is chosen (workstream 4).
  • Alternatives need a major rework or removal (workstream 5).
  • Search needs a major rework (workstream 6).
  • Order. Quick protection comes first. The owner and an agent do it item by item.

Workstreams

0. Quick protection

Small Cloudflare, Worker and VPS changes. No format change.

1. Larger map packs

Problem. R2 charges for each upload. One upload costs the same as one month of storage for 300 KB. The mean map pack is 200 KB, so the object count drives publish cost, and it also drives publish time, manifest size, update downloads, delete time and the file count of an offline cell.

Today one pack holds one z11 cell and its deeper tiles (MAP_ZOOM, tools/planner_grid_maps.py). Each tile at z10 or lower has its own pack (packName, planner/tiles/src/grid.mjs).

Estimated cost at the target coverage, today's layout.

Europe snow only Snow z12 everywhere
Objects 9.5M 36M
Full publish, Class A $38 $158
Time at an assumed 200 uploads/s 13 h 50 h

An OSM refresh is about 2M uploads, $9. A weekly refresh costs more than the storage.

Direction. Deep packs per z9 cell, low zooms in shared packs, and one snow pack per cell (workstream 2 decides the snow shape). Readers already use byte ranges, so a tile read does not change in kind.

Simulation on the real file sizes of the production release.

Layout Map objects Median pack Largest pack
Today 5,942 22.7 MB
One snow pack per cell 1,017 0.78 MB 29.8 MB
z11, low zooms under z7, snow merged 1,056 0.68 MB 14.8 MB
z10, low zooms under z7, snow merged 375 1.23 MB 14.9 MB
z9, low zooms under z7, snow merged 171 1.47 MB 56.9 MB

At the target coverage, z9 gives about 0.4–0.5M map objects. Routing, search and the other cell files add about 0.45M (estimated), and that part can also shrink. The total stays near R2's free allowance of 1M Class A operations per month, so a publish costs about $0–2. With z10, a publish costs about $4–5.

Read first.

  • tools/planner_grid_maps.py, tools/planner_grid_index.py, tools/planner_runtime.py, tools/planner_downloads.py, tools/planner_verify.py
  • host/obc-data-steps/src/planner/catalog.rs, which pins map_zoom: 11
  • planner/tiles/src/grid.mjs, whose check already allows map_zoom 9
  • builder/web/src/lib/trailview/stream/terrain.ts, which reads z13 terrain packs in its own layout
  • specs/planner-release.md, specs/planner-offline.md, specs/obc-data.md

Measure before the zoom is chosen.

  • The time of a cold range read of a 50 MB pack through maps.openbikecomputer.com. Cloudflare caches the whole object on a range request (a different range of one pack measured HIT).
  • The Worker R2 reads per tile miss, when a larger pack needs a leaf directory.
  • The bytes that an OSM refresh uploads. One changed tile uploads its whole pack.
  • The storage that two releases share. Larger packs share less.
  • The publish time. Upload bandwidth becomes the limit: 1.4 TB takes about 31 h at 100 Mbit/s.

Done when a Central Switzerland re-bake shows the objects, the requests and the first-tile and TrailView open times against today, with no visual change.

Coordinate with #2921. Its architecture A1 bakes prepared layers once per continent into block-packed archives. Its phase 5 opens a design issue for the continental layers. This pack layout must feed that design, so that the two do not define two layouts.

2. Snow

Problem. The agent that takes this finds the solution.

Measured on the production release Value
TrailView snow archives 46, one per band of 8 days
Band packs 4,600 of 7,215 files (64%), 77% of map packs
Band bytes 110 MB, about 16% of the map bytes without z13 terrain
Mean band pack 24 KB
Planner snow archive 114 MB, the same information in another shape
Date change, new band 35 requests
Date change, new band and quarter 77 requests
Date slider, one year in 12 s 303 requests, 16.5 MB

Constraints: a view shows the snow of its date. A change to the look needs owner approval with a before and after gallery. #2921 S1 makes the planner season blocks the one artifact from which TrailView bands derive.

3. Manifests

Problem. release.json lists each file of a release with its path, size, hash, and a transport size and hash: 267 bytes per file, 1.9 MB today. For 6,867 of 7,215 files the second hash repeats the first.

Document Today Target coverage (estimated) Reader
release.json, today's layout 1.9 MB 2.5–10 GB Phone, follower, tools
release.json, larger packs about 230 MB The same
offline/catalog.json 2.2 MB 3.6 GB Downloads service
routing/blocks.json 3.4 MB 5.7 GB Routing service
  • The phone reads only profiles and routing_package from it (PlannerService.swift).
  • The follower parses the whole file and needs only the runtime files (host/obc-data/src/serve/services.rs).
  • The downloads service parses its whole catalog (tools/planner_downloads.py).
  • routing/blocks.json has a limit of 128 MB (MAX_MANIFEST_BYTES). A package over about 47M roads answers Limit at open, and that is less than Germany.

Direction. Put the two phone fields in the catalog. Keep a small release.json that holds the hash of a file list per layer, so the release ID still proves all the content. Give each cell its own offline catalog. Workstream 4 replaces routing/blocks.json. Publish tools stream lists and delete in batches (host/obc-data/src/collection.rs, host/obc-data/src/r2.rs).

Land workstreams 1, 2 and 3 together. They all change the data format, so they need only one re-bake.

4. Routing: hierarchical search and disk storage

Problem 1: effort grows with area. The search has no hierarchy. It settles labels on all roads in an area that grows about with distance squared.

Primary route, touring Settled labels Time, M1
10 km 14k 6 ms
75 km 80k 18 ms
171 km, the longest in the package 646k 168 ms
2,000 km on Europe (estimated) 80–250M+ 80–250 s on a VPS

A 2,000 km query also needs 1.3–4 GiB of labels per worker (estimated), and it passes the 15 s deadline.

Problem 2: landmarks stop working at scale (measured by rebuilding the columns at a larger scale).

Scale 10 km 75 km 171 km
Today 14k 80k 646k
×8 46k 342k 665k
×80, about the world 85k 681k 947k
No landmarks 85k 682k 947k
  • Values are 16 bits with one scale per package. The scale rises from 18 to about 1,400 at world size (estimated). The median road cost is 123, so most roads round down to zero.
  • There are 6 landmarks, all in the largest connected component, so other continents get no guidance (from reading the builder).
  • Landmark columns take 4.9 B/road per profile: 83 of the 174 B/road of all profiles.
  • Find out whether the hierarchy replaces landmarks. If it does, do not fix them separately.

Problem 3: the graph is in RAM. Union::open and Selection::load decode the graph and the cost columns.

Coverage Roads (estimated) RAM, touring only RAM, 17 profiles
Europe + North America + Australia + NZ 650M 41–49 GB about 146 GB
World about 1B about 80–90 GB about 240 GB

Disk reads are viable (measured). Road IDs follow a Morton curve, so a query reads near pages. A Hilbert order gains less than 4%.

Route Search 64 KiB reads Added at 100 µs Added at 300 µs Near query reuses
10 km 6 ms 70 7 ms 21 ms 88%
75 km 18 ms 176 18 ms 53 ms 73%
171 km 168 ms 537 54 ms 161 ms 98%
  • Geometry adds 320–550 object reads per route when cold (about 40–70 ms on NVMe).
  • Storage per 1B roads: 37 B/road of topology plus 174 B/road of profiles, about 250 GB without geometry. Sparse turn columns and u16 costs with exception lists save about 26% of the 174 (estimated).
  • The label table costs about 94 MB per worker and 4.5 ms per search at 1B roads (computed). Remove the walk over the whole table.
  • World projection for a 300 km leg, 64 GB, primary route: with working guidance 0.36 s warm and 0.7–1.0 s cold on a VPS disk. Without guidance 2.5 s warm and 8.3 s cold.

Investigate and decide.

  • The hierarchical method. A multi-level overlay is the first candidate, because it does not depend on a road class hierarchy, and bicycle profiles have none. Check it against exact route costs, 17 profiles, turn costs, closures, direction continuity, via, pins and turnarounds, bake time and memory, storage per profile, and the phone and WebAssembly engines.
  • The disk layout: raw columns with memory mapping, or compressed pages with a bounded cache.
  • Which profiles need their own stored data.
  • The search queue, 30% of a search with 48-byte entries, and the other costs of the search profile.

Server test. Run fio random reads at 64 KiB at queue depth 1 on the candidate server, several times at different hours. Record the median and P99 latency. Record whether 64 GB of page cache holds the hot regions. Public benchmarks of Contabo NVMe plans show about 2,800–2,900 reads/s at 64 KiB, which looks like a rate limit. No figure for the SSD plan was found.

Prove it on a bake of Germany's size or larger, on the real server, with routes up to the largest the bake allows.

5. Alternatives

Problem. One alternatives call is expensive for what it returns (measured).

Part, 171 km route (1,145 ms, 2 routes returned) Searches Labels ms
Primary 1 646k 91
Shorter 1 667k 95
Less climbing 1 705k 108
Corridor probes 12 4.69M 616
Probe snap, line and checks 222
  • All four probes were rejected on this route. They took 70% of the labels.
  • On a 10 km route the call runs 15 searches, 17× the work of the primary, and returns only the primary.
  • At an assumed 10% share of requests, alternatives take over 80% of routing CPU: about 5 requests/s per M1 core with them today, 12 without the wasted probes, 29 with them outside the main pool. The client asks only while the routes list is open (PlannerApp.svelte), so the real share can be lower.

Decide: rework or remove, after the hierarchical search is known, because it changes the cost of each search. Read the product decisions in #2711 first. It records the compare flow.

6. Search

Problem. A plain text search asks every cell, in series, on the single Node thread (planner/search/web/engine.mjs, planner/search/federation.mjs). This system needs a major rework. It gets its own investigation and sub-issues.

Cells (measured, real files linked into larger grids) Miss query Prefix query Memory
9 28 ms 26 ms 84 MB
500 571 ms 1.2 s 2.3–2.6 GB after 50–60 queries
2,000 3.5 s 8.0 s 4.2 GB after 13 queries, growing
  • At 15,000 cells a miss takes about 26 s (estimated), and the thread blocks all users.
  • The view centre is written into the SQL, so each view prepares new statements. Every cell stays attached (planner/search/cells.mjs). Memory reaches about 78 GB as built (estimated).
  • Smart queries reach about 15 req/s (measured). The resolver uses most of that time, not the model (4–10 ms).
  • One release has one time zone for opening hours (planner/search/hours.mjs). Mixed zones stop the service at start.
  • All cells need one OSM source (planner/search/federation.mjs), so the continents need one merged PBF.
  • The model takes 420 MB per process, and three releases can run.
  • Search data at the target coverage is about 51–56 GB, plus 20–22 GB of gzip copies (estimated).

7. Follower and operations

  • A hung model stops search. The follower then fails the whole release, stops routing and downloads, and installs again. Recovery takes about 30 minutes at the target coverage (estimated).
  • Disk during an update is about 465–525 GB for three releases (estimated). Delete gzip copies after decoding, and remove the rollback before an update.
  • An update is about 210,000 R2 reads and about 100 GB, run in series, 3–6 h (estimated).
  • The VPS has no Cloudflare proxy and no per-address limit. Two clients that loop the heaviest allowed request give a normal user 79% errors (measured).
  • Production runs 2 routing workers, because nothing sets ROUTE_WORKERS.
  • A request gets 503 busy after 1 s in the queue.
  • The release bounding box: one box per release turns three continents into most of the world's land, about 1.6 TB more (host/obc-data-steps/src/planner.rs). Clip to area outlines. Check it against the scope rules of Prepared sources: pin the formats, then build climate, snow and colour #2921.

8. Full-scale proof

Bake Europe (#2921 A6). Run a load test of 100 simulated users on the chosen server, with owner approval. Read the real Cloudflare usage. Compare it with this epic, and record the result. The owner then decides on the launch and on more coverage.

Later, only if the numbers ask for it

Read all tiles by byte range from maps.openbikecomputer.com and skip the Worker. It removes the Worker fee and about 85% of Class B reads (estimated): the heavy case goes from $144 to about $2–13. It needs the cold read test of workstream 1.

Sub-issues

Order

  1. Workstream 0.
  2. Workstreams 1, 2 and 3, with one re-bake. Coordinate with Prepared sources: pin the formats, then build climate, snow and colour #2921.
  3. Workstreams 4 and 6 can run in parallel. Workstream 5 follows the first results of 4. Workstream 7 runs alongside.
  4. Workstream 8.

Left out of the audit

No load test on production. No SSH to the VPS, so its specs are unknown. No continental bake. No test on a slow disk. No iPhone memory test. Cache hit rates (35–64%) are from a model. The Workers plan could not be read through the API.

Related: #2921, #2648, #2711, #2747, #2291

Activity

  1. added
    epicTracking issue spanning multiple sub-issues
    area:routeAffects the obc-route area
    infraTooling, scaffold, CI, dev harness
    on Nov 1, 2026
  2. timohueser commented on Nov 4, 2026

    @timohueser
    OwnerAuthor

    These issues are merged here and closed:


    Generated by Claude Code

  3. timohueser commented on Nov 5, 2026

    @timohueser
    OwnerAuthor

    Workstream 3 (Manifests): code merged, verification still open

    The epic #3141 is closed. Its code is in develop:

    These steps are not done yet. They run in the one re-bake of workstreams 1, 2 and 3, so the snow sub-issues of workstream 2 (#3129, #3127, #3128, after #2937) must merge first.

    Before a release with the new formats is published

    1. Deploy the follower of Manifests B: engine manifest file lists as objects, collection in batches #3147 on the VPS (obc data deploy). The review of an update of prod names a follower that cannot read manifest format 2.
    2. Deploy the tile Worker and the web build of Larger map packs: one pack rule with a 15 MiB limit #3069 B, and ship an iOS app build with the offline reader of Larger map packs: one pack rule with a 15 MiB limit #3069 C. See "Still to do" in Larger map packs: one pack rule with a 15 MiB limit #3069.

    Re-bake of Central Switzerland to the local target
    3. Plan the bake with the owner first, because it runs on the shared PC. planner/routing is at revision 3, so the bake builds routing and the overlays again.
    4. Record in this issue, against the production release:

    1. On the local target:

    Owner checks
    6. Open the planner and TrailView on the re-bake.
    7. Download an offline area of the re-bake on the phone and plan a route in it.

    Publish to prod
    8. Publish to prod only with an approved review ID.
    9. Read the output of the first collection on prod. It is the first real run of the DeleteObjects request (Writer::delete_many in host/obc-data/src/r2/s3.rs). Only offline tests cover it so far.
    10. Close #3069 and #3125 when their steps are done, and tick them here.


    Generated by Claude Code

  4. timohueser commented on Nov 10, 2026

    @timohueser
    OwnerAuthor

    Review of 2026-11-10: routing at Europe and world scale

    Scope: the search, the bake, the stored format and the serving path of routing on develop 9ec892f. Four agents read the code. Nothing was built or run, so each scale figure is an estimate from the code and from the figures in this epic, #3227 and #3251. The three full reports are comments on #3227.

    State

    What stops Europe as built

    Area Problem Effect
    Format Each page of 4,096 entries has a hash in one JSON manifest with a limit of 128 MiB A package above about 47–63M roads does not open. A Europe bake fails after 9 h or more
    Bake The work files are memory-mapped and read at random (node lookups, way tags for each road and profile, the shape gather) Germany passed because its 17.8 GB of work files fit the RAM. Europe has about 125–150 GB
    Bake One thread, the profiles in sequence, no restart after a crash 9.7 h at best for Europe; a cost change repeats the import
    Engine The graph and the cost columns are in heap About 16 GB for one profile at 300M roads, for each release
    Serving The follower deletes and installs each release again at every start. The unit has MemoryMax=9216M Long recovery; memory maps would get only 9 GB of page cache
    Phone Pages span grid cells One cell downloads 1.03–3.3 times its own routing bytes

    Europe can have 290–415M directed roads, not 250–300M. Nobody has counted them.

    Proposal

    The partition cell of the overlay becomes the unit of the road numbers, the files on disk, the published objects, the phone downloads and the cost tables.

    • Bake: join nodes to ways by sort; do all tag work once in the way scan; put the roads into cell order once; calculate all profiles in one parallel pass; split the step into four cached layers (source, roads, costs, package). Estimate: Europe in 3–4 h on the owner PC, the world in 10–13 h with 32 GB.
    • Format 9: one compressed object for each cell and part, which is a slice of the installed file. blocks.json, the packs, the page hashes and the Python selection merge go.
    • Engine: columns as byte slices (memory maps on the server and the phone), the overlay search, and the unpacking of the chosen legs only, with read-ahead. The landmarks, the heap graph and the memory budget code go.
    • Serving: installs that stay on disk, two installed releases, and unit limits that fit memory maps.

    Owner decisions of 2026-11-10

    • Profiles: 10, not 17. Each activity has two profiles: touring, road and gravel each with /less-climbing; mtb with mtb/technical; hiking with hiking/alpine. The shorter profiles go. Profile tuning and a new alternatives view are later work.
    • Order of work: measure (road count of Europe, a Germany bake with a capped page cache, fio on the candidate server, border roads for each level), revise the design of Routing: hierarchical search and disk storage #3227, then build: the bake first, then format 9 and the cost tables, then the engine and the follower. Proof on Germany, then Europe.
    • Central Switzerland gets a new bake and publish first, on swissALTI3D heights, so that the online planner and TrailView work with the current clients.

    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:routeAffects the obc-route areaepicTracking issue spanning multiple sub-issuesinfraTooling, scaffold, CI, dev harness

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions