Repository navigation
Description
Goal
The route planner and TrailView must serve 100 concurrent users on Europe, North America, Australia and New Zealand, and later the world. Riders must be able to plan long routes. A single leg of 2,000 km must answer in a few seconds.
This epic holds the audit, the owner decisions and the workstreams. Follow-up sessions investigate each workstream and create its sub-issues. Add each sub-issue to the list below.
Current state
Production serves Central Switzerland only: about 9,000 km², 1.2M directed roads, 7,215 files, 1.6 GB.
| System | Today | At the target coverage, as built |
|---|---|---|
| Tile Worker and R2 | Fits | Fits. Abuse has no spending ceiling |
| TrailView client | Fits | Fits. Slow links are weak |
| Routing | Busy errors from about 10 req/s | Does not start |
| Search | Fits | Fails for all users |
| Data updates and manifests | Fit | Fail |
How to read the numbers. "Measured" means a run on the production packages on an Apple M1 Pro, or a read of the live Cloudflare account. "Estimated" means an extrapolation from Central Switzerland and an older Baden-Württemberg package, ±40% or worse. No bake or test at continental size exists. No test ran on the VPS.
Cost (estimated)
| Item | Monthly |
|---|---|
| R2 storage, 1.4 TB mid case (0.8–2.3 TB) | about $21 ($12–35) |
| Workers and R2 reads, 100 users 8 h/day, 80% planner, 20% light TrailView | about $24, with the $5 plan |
| The same, everyone in heavy TrailView | about $149 |
| Server | about 40–50 € (see Decisions) |
Usage per hour of one user (estimated): the planner makes about 970 Worker requests and 1,440 R2 reads. TrailView makes 3,000–9,000 Worker requests and 3,300–9,900 R2 reads.
Owner decisions
- Long routes are a requirement. The planner must make long routes possible. A few seconds for 2,000 km is acceptable. A route-length limit is not the answer.
- Hierarchical search is in scope (workstream 4).
- Server. The candidate is a Contabo server with 64 GB of RAM at about 40 €/month, with storage upgraded to 1 TB of SSD for about 8 € more. Test its disk before it is chosen (workstream 4).
- Alternatives need a major rework or removal (workstream 5).
- Search needs a major rework (workstream 6).
- Order. Quick protection comes first. The owner and an agent do it item by item.
Workstreams
0. Quick protection
Small Cloudflare, Worker and VPS changes. No format change.
1. Larger map packs
Problem. R2 charges for each upload. One upload costs the same as one month of storage for 300 KB. The mean map pack is 200 KB, so the object count drives publish cost, and it also drives publish time, manifest size, update downloads, delete time and the file count of an offline cell.
Today one pack holds one z11 cell and its deeper tiles (MAP_ZOOM, tools/planner_grid_maps.py). Each tile at z10 or lower has its own pack (packName, planner/tiles/src/grid.mjs).
Estimated cost at the target coverage, today's layout.
| Europe snow only | Snow z12 everywhere | |
|---|---|---|
| Objects | 9.5M | 36M |
| Full publish, Class A | $38 | $158 |
| Time at an assumed 200 uploads/s | 13 h | 50 h |
An OSM refresh is about 2M uploads, $9. A weekly refresh costs more than the storage.
Direction. Deep packs per z9 cell, low zooms in shared packs, and one snow pack per cell (workstream 2 decides the snow shape). Readers already use byte ranges, so a tile read does not change in kind.
Simulation on the real file sizes of the production release.
| Layout | Map objects | Median pack | Largest pack |
|---|---|---|---|
| Today | 5,942 | 22.7 MB | |
| One snow pack per cell | 1,017 | 0.78 MB | 29.8 MB |
| z11, low zooms under z7, snow merged | 1,056 | 0.68 MB | 14.8 MB |
| z10, low zooms under z7, snow merged | 375 | 1.23 MB | 14.9 MB |
| z9, low zooms under z7, snow merged | 171 | 1.47 MB | 56.9 MB |
At the target coverage, z9 gives about 0.4–0.5M map objects. Routing, search and the other cell files add about 0.45M (estimated), and that part can also shrink. The total stays near R2's free allowance of 1M Class A operations per month, so a publish costs about $0–2. With z10, a publish costs about $4–5.
Read first.
tools/planner_grid_maps.py,tools/planner_grid_index.py,tools/planner_runtime.py,tools/planner_downloads.py,tools/planner_verify.pyhost/obc-data-steps/src/planner/catalog.rs, which pinsmap_zoom: 11planner/tiles/src/grid.mjs, whose check already allowsmap_zoom9builder/web/src/lib/trailview/stream/terrain.ts, which reads z13 terrain packs in its own layoutspecs/planner-release.md,specs/planner-offline.md,specs/obc-data.md
Measure before the zoom is chosen.
- The time of a cold range read of a 50 MB pack through
maps.openbikecomputer.com. Cloudflare caches the whole object on a range request (a different range of one pack measured HIT). - The Worker R2 reads per tile miss, when a larger pack needs a leaf directory.
- The bytes that an OSM refresh uploads. One changed tile uploads its whole pack.
- The storage that two releases share. Larger packs share less.
- The publish time. Upload bandwidth becomes the limit: 1.4 TB takes about 31 h at 100 Mbit/s.
Done when a Central Switzerland re-bake shows the objects, the requests and the first-tile and TrailView open times against today, with no visual change.
Coordinate with #2921. Its architecture A1 bakes prepared layers once per continent into block-packed archives. Its phase 5 opens a design issue for the continental layers. This pack layout must feed that design, so that the two do not define two layouts.
2. Snow
Problem. The agent that takes this finds the solution.
| Measured on the production release | Value |
|---|---|
| TrailView snow archives | 46, one per band of 8 days |
| Band packs | 4,600 of 7,215 files (64%), 77% of map packs |
| Band bytes | 110 MB, about 16% of the map bytes without z13 terrain |
| Mean band pack | 24 KB |
Planner snow archive |
114 MB, the same information in another shape |
| Date change, new band | 35 requests |
| Date change, new band and quarter | 77 requests |
| Date slider, one year in 12 s | 303 requests, 16.5 MB |
- Worst case in code: 46 bands × 35 tiles = 1,610 requests per view.
tools/planner_snow.pybakes one maximum zoom per archive, so 500 m MODIS data would be stored at the zoom of 20 m HR-WSI data. That is most of the 408 GB high case. Prepared sources: pin the formats, then build climate, snow and colour #2921 S3 fixes the zoom (HR-WSI 12, MODIS 8) in PR obc data: the prepared source snow (format 1), and planner/snow bakes from it #2937, which is merged. Check this before more work.- At the target coverage, snow is the difference between 9.5M and 36M objects (estimated).
Constraints: a view shows the snow of its date. A change to the look needs owner approval with a before and after gallery. #2921 S1 makes the planner season blocks the one artifact from which TrailView bands derive.
3. Manifests
Problem. release.json lists each file of a release with its path, size, hash, and a transport size and hash: 267 bytes per file, 1.9 MB today. For 6,867 of 7,215 files the second hash repeats the first.
| Document | Today | Target coverage (estimated) | Reader |
|---|---|---|---|
release.json, today's layout |
1.9 MB | 2.5–10 GB | Phone, follower, tools |
release.json, larger packs |
about 230 MB | The same | |
offline/catalog.json |
2.2 MB | 3.6 GB | Downloads service |
routing/blocks.json |
3.4 MB | 5.7 GB | Routing service |
- The phone reads only
profilesandrouting_packagefrom it (PlannerService.swift). - The follower parses the whole file and needs only the runtime files (
host/obc-data/src/serve/services.rs). - The downloads service parses its whole catalog (
tools/planner_downloads.py). routing/blocks.jsonhas a limit of 128 MB (MAX_MANIFEST_BYTES). A package over about 47M roads answersLimitat open, and that is less than Germany.
Direction. Put the two phone fields in the catalog. Keep a small release.json that holds the hash of a file list per layer, so the release ID still proves all the content. Give each cell its own offline catalog. Workstream 4 replaces routing/blocks.json. Publish tools stream lists and delete in batches (host/obc-data/src/collection.rs, host/obc-data/src/r2.rs).
Land workstreams 1, 2 and 3 together. They all change the data format, so they need only one re-bake.
4. Routing: hierarchical search and disk storage
Problem 1: effort grows with area. The search has no hierarchy. It settles labels on all roads in an area that grows about with distance squared.
| Primary route, touring | Settled labels | Time, M1 |
|---|---|---|
| 10 km | 14k | 6 ms |
| 75 km | 80k | 18 ms |
| 171 km, the longest in the package | 646k | 168 ms |
| 2,000 km on Europe (estimated) | 80–250M+ | 80–250 s on a VPS |
A 2,000 km query also needs 1.3–4 GiB of labels per worker (estimated), and it passes the 15 s deadline.
Problem 2: landmarks stop working at scale (measured by rebuilding the columns at a larger scale).
| Scale | 10 km | 75 km | 171 km |
|---|---|---|---|
| Today | 14k | 80k | 646k |
| ×8 | 46k | 342k | 665k |
| ×80, about the world | 85k | 681k | 947k |
| No landmarks | 85k | 682k | 947k |
- Values are 16 bits with one scale per package. The scale rises from 18 to about 1,400 at world size (estimated). The median road cost is 123, so most roads round down to zero.
- There are 6 landmarks, all in the largest connected component, so other continents get no guidance (from reading the builder).
- Landmark columns take 4.9 B/road per profile: 83 of the 174 B/road of all profiles.
- Find out whether the hierarchy replaces landmarks. If it does, do not fix them separately.
Problem 3: the graph is in RAM. Union::open and Selection::load decode the graph and the cost columns.
| Coverage | Roads (estimated) | RAM, touring only | RAM, 17 profiles |
|---|---|---|---|
| Europe + North America + Australia + NZ | 650M | 41–49 GB | about 146 GB |
| World | about 1B | about 80–90 GB | about 240 GB |
Disk reads are viable (measured). Road IDs follow a Morton curve, so a query reads near pages. A Hilbert order gains less than 4%.
| Route | Search | 64 KiB reads | Added at 100 µs | Added at 300 µs | Near query reuses |
|---|---|---|---|---|---|
| 10 km | 6 ms | 70 | 7 ms | 21 ms | 88% |
| 75 km | 18 ms | 176 | 18 ms | 53 ms | 73% |
| 171 km | 168 ms | 537 | 54 ms | 161 ms | 98% |
- Geometry adds 320–550 object reads per route when cold (about 40–70 ms on NVMe).
- Storage per 1B roads: 37 B/road of topology plus 174 B/road of profiles, about 250 GB without geometry. Sparse turn columns and u16 costs with exception lists save about 26% of the 174 (estimated).
- The label table costs about 94 MB per worker and 4.5 ms per search at 1B roads (computed). Remove the walk over the whole table.
- World projection for a 300 km leg, 64 GB, primary route: with working guidance 0.36 s warm and 0.7–1.0 s cold on a VPS disk. Without guidance 2.5 s warm and 8.3 s cold.
Investigate and decide.
- The hierarchical method. A multi-level overlay is the first candidate, because it does not depend on a road class hierarchy, and bicycle profiles have none. Check it against exact route costs, 17 profiles, turn costs, closures, direction continuity,
via, pins and turnarounds, bake time and memory, storage per profile, and the phone and WebAssembly engines. - The disk layout: raw columns with memory mapping, or compressed pages with a bounded cache.
- Which profiles need their own stored data.
- The search queue, 30% of a search with 48-byte entries, and the other costs of the search profile.
Server test. Run fio random reads at 64 KiB at queue depth 1 on the candidate server, several times at different hours. Record the median and P99 latency. Record whether 64 GB of page cache holds the hot regions. Public benchmarks of Contabo NVMe plans show about 2,800–2,900 reads/s at 64 KiB, which looks like a rate limit. No figure for the SSD plan was found.
Prove it on a bake of Germany's size or larger, on the real server, with routes up to the largest the bake allows.
5. Alternatives
Problem. One alternatives call is expensive for what it returns (measured).
| Part, 171 km route (1,145 ms, 2 routes returned) | Searches | Labels | ms |
|---|---|---|---|
| Primary | 1 | 646k | 91 |
| Shorter | 1 | 667k | 95 |
| Less climbing | 1 | 705k | 108 |
| Corridor probes | 12 | 4.69M | 616 |
| Probe snap, line and checks | 222 |
- All four probes were rejected on this route. They took 70% of the labels.
- On a 10 km route the call runs 15 searches, 17× the work of the primary, and returns only the primary.
- At an assumed 10% share of requests, alternatives take over 80% of routing CPU: about 5 requests/s per M1 core with them today, 12 without the wasted probes, 29 with them outside the main pool. The client asks only while the routes list is open (
PlannerApp.svelte), so the real share can be lower.
Decide: rework or remove, after the hierarchical search is known, because it changes the cost of each search. Read the product decisions in #2711 first. It records the compare flow.
6. Search
Problem. A plain text search asks every cell, in series, on the single Node thread (planner/search/web/engine.mjs, planner/search/federation.mjs). This system needs a major rework. It gets its own investigation and sub-issues.
| Cells (measured, real files linked into larger grids) | Miss query | Prefix query | Memory |
|---|---|---|---|
| 9 | 28 ms | 26 ms | 84 MB |
| 500 | 571 ms | 1.2 s | 2.3–2.6 GB after 50–60 queries |
| 2,000 | 3.5 s | 8.0 s | 4.2 GB after 13 queries, growing |
- At 15,000 cells a miss takes about 26 s (estimated), and the thread blocks all users.
- The view centre is written into the SQL, so each view prepares new statements. Every cell stays attached (
planner/search/cells.mjs). Memory reaches about 78 GB as built (estimated). - Smart queries reach about 15 req/s (measured). The resolver uses most of that time, not the model (4–10 ms).
- One release has one time zone for opening hours (
planner/search/hours.mjs). Mixed zones stop the service at start. - All cells need one OSM source (
planner/search/federation.mjs), so the continents need one merged PBF. - The model takes 420 MB per process, and three releases can run.
- Search data at the target coverage is about 51–56 GB, plus 20–22 GB of gzip copies (estimated).
7. Follower and operations
- A hung model stops search. The follower then fails the whole release, stops routing and downloads, and installs again. Recovery takes about 30 minutes at the target coverage (estimated).
- Disk during an update is about 465–525 GB for three releases (estimated). Delete gzip copies after decoding, and remove the rollback before an update.
- An update is about 210,000 R2 reads and about 100 GB, run in series, 3–6 h (estimated).
- The VPS has no Cloudflare proxy and no per-address limit. Two clients that loop the heaviest allowed request give a normal user 79% errors (measured).
- Production runs 2 routing workers, because nothing sets
ROUTE_WORKERS. - A request gets
503 busyafter 1 s in the queue. - The release bounding box: one box per release turns three continents into most of the world's land, about 1.6 TB more (
host/obc-data-steps/src/planner.rs). Clip to area outlines. Check it against the scope rules of Prepared sources: pin the formats, then build climate, snow and colour #2921.
8. Full-scale proof
Bake Europe (#2921 A6). Run a load test of 100 simulated users on the chosen server, with owner approval. Read the real Cloudflare usage. Compare it with this epic, and record the result. The owner then decides on the launch and on more coverage.
Later, only if the numbers ask for it
Read all tiles by byte range from maps.openbikecomputer.com and skip the Worker. It removes the Worker fee and about 85% of Class B reads (estimated): the heavy case goes from $144 to about $2–13. It needs the cold read test of workstream 1.
Sub-issues
- 0. Quick protection (done with the owner; no sub-issue needed)
- 1. Larger map packs
- Larger map packs: one pack rule with a 15 MiB limit #3069 Larger map packs: one pack rule with a 15 MiB limit
- 2. Snow
- Snow: one date-free snow layer for TrailView and the planner, and small per-year tiles for routes #3125 Snow: one date-free snow layer for TrailView and the planner, and small per-year tiles for routes
- 3. Manifests
- Manifests: small release manifests, per-cell offline catalogs and file lists as objects #3141 Manifests: small release manifests, per-cell offline catalogs and file lists as objects
- 4. Routing: hierarchical search and disk storage
- Routing: 10 profiles, two for each activity #3275 Routing: 10 profiles, two for each activity
- Routing: hierarchical search and disk storage #3227 Routing: hierarchical search and disk storage
- Routing bake: bake a continent in bounded memory #3228 Routing bake: bake a continent in bounded memory
- Routing bake 1/4: one compact copy of the OSM source data #3238 Routing bake 1/4: one compact copy of the OSM source data
- Routing bake 2/4: roads as flat columns #3239 Routing bake 2/4: roads as flat columns
- Routing bake 3/4: flat working columns in the preparation, and terrain on demand #3240 Routing bake 3/4: flat working columns in the preparation, and terrain on demand
- Routing bake 4/4: work tables in files, and sorts on disk #3241 Routing bake 4/4: work tables in files, and sorts on disk
- Routing bake: bake a continent in bounded memory #3228 Routing bake: bake a continent in bounded memory
- 5. Alternatives
- 6. Search
- 7. Follower and operations
- Routing service: one worker for each CPU, up to a measured memory limit #3053 Routing service: one worker for each CPU, up to a measured memory limit
- 8. Full-scale proof
Order
- Workstream 0.
- Workstreams 1, 2 and 3, with one re-bake. Coordinate with Prepared sources: pin the formats, then build climate, snow and colour #2921.
- Workstreams 4 and 6 can run in parallel. Workstream 5 follows the first results of 4. Workstream 7 runs alongside.
- Workstream 8.
Left out of the audit
No load test on production. No SSH to the VPS, so its specs are unknown. No continental bake. No test on a slow disk. No iPhone memory test. Cache hit rates (35–64%) are from a model. The Workers plan could not be read through the API.
Activity
timohueser commentedon Nov 4, 2026
These issues are merged here and closed:
Planet router: design how the routing package scales to the whole planet #2657 Planet router. Workstream 4 covers this question: hierarchical search, disk storage, planet RAM and bake memory. The partitions hook that it depends on is merged (obc data v2 · P7-A — named partitions of a whole layer #2693).
obc data v2 · P7 — Planet coverage #2655 obc data v2 P7: planet coverage. Its acceptance tests landed (obc data v2 · P7-B — leaf extracts in a binary split with deterministic bytes #2688, obc data v2 · P7-C — a step whose key is its served key keeps its output; the run counts unchanged leaves #2690, obc data v2 · P7-A — named partitions of a whole layer #2693), and coverage is now Europe (Prepared sources: pin the formats, then build climate, snow and colour #2921 A6). What remains:
Q6 (delete
obc bake --all) goes to obc data v2 · P8 — Cutover, deletion and documentation #2656.Generated by Claude Code
timohueser commentedon Nov 5, 2026
Workstream 3 (Manifests): code merged, verification still open
The epic #3141 is closed. Its code is in
develop:release.jsonnames the step indexes and no longer lists every file. The offline catalog has one file per cell. The follower reads only the indexes that it serves.These steps are not done yet. They run in the one re-bake of workstreams 1, 2 and 3, so the snow sub-issues of workstream 2 (#3129, #3127, #3128, after #2937) must merge first.
Before a release with the new formats is published
obc data deploy). The review of an update of prod names a follower that cannot read manifest format 2.Re-bake of Central Switzerland to the local target
3. Plan the bake with the owner first, because it runs on the shared PC.
planner/routingis at revision 3, so the bake builds routing and the overlays again.4. Record in this issue, against the production release:
release.json, of the engine manifest and of its largest list, the number of outputs, the largest step index,offline/catalog.jsonand the largest cell file;Owner checks
6. Open the planner and TrailView on the re-bake.
7. Download an offline area of the re-bake on the phone and plan a route in it.
Publish to prod
8. Publish to prod only with an approved review ID.
9. Read the output of the first collection on prod. It is the first real run of the
DeleteObjectsrequest (Writer::delete_manyinhost/obc-data/src/r2/s3.rs). Only offline tests cover it so far.10. Close #3069 and #3125 when their steps are done, and tick them here.
Generated by Claude Code
timohueser commentedon Nov 10, 2026
Review of 2026-11-10: routing at Europe and world scale
Scope: the search, the bake, the stored format and the serving path of routing on
develop9ec892f. Four agents read the code. Nothing was built or run, so each scale figure is an estimate from the code and from the figures in this epic, #3227 and #3251. The three full reports are comments on #3227.State
developruns the two-sided search with landmarks and decodes the whole graph into RAM. Routing: hierarchical search and disk storage #3227 has the statusexplored.What stops Europe as built
MemoryMax=9216MEurope can have 290–415M directed roads, not 250–300M. Nobody has counted them.
Proposal
The partition cell of the overlay becomes the unit of the road numbers, the files on disk, the published objects, the phone downloads and the cost tables.
blocks.json, the packs, the page hashes and the Python selection merge go.Owner decisions of 2026-11-10
touring,roadandgraveleach with/less-climbing;mtbwithmtb/technical;hikingwithhiking/alpine. Theshorterprofiles go. Profile tuning and a new alternatives view are later work.fioon the candidate server, border roads for each level), revise the design of Routing: hierarchical search and disk storage #3227, then build: the bake first, then format 9 and the cost tables, then the engine and the follower. Proof on Germany, then Europe.Generated by Claude Code