Repository navigation
Description
Goal
One system bakes, updates and publishes all map data, for the device and for the planner. It runs the rare jobs too: the Wikimedia snapshot and the national lidar terrain archive. It works the same on a laptop and on a bake VPS, for areas from one town up to the whole planet. A new person learns it in one hour. The rule for "what is stale" fits in three lines.
This epic replaces the design of #2410 and the plan of #2612. The sub-issues below are the work order. Each one is self-contained, so a fresh session can start with any phase whose dependencies are done.
Why a second design
obc data works, but it is hard to operate and hard to teach. It has never published to production (#2612), so breaking changes cost nothing now.
- Four entry points write the same bucket:
obc data,obc bake(host/obc-bake),obc planner(tools/planner.py) andobc r2(tools/r2.py). There are two VPS deploy models (tools/planner_publish.py,tools/planner_deploy.py) and three identity schemes. - Staleness is a code hash that nobody can predict. A layer key hashes git-listed files, the Cargo closure, the compiler,
RUSTFLAGS, the uv lock and the Python ABI (host/obc-data/src/engine/code.rs). A refactor rebuilds layers whose bytes do not change. Python steps need hand-written import lists that a regex test polices (host/obc-data-steps/src/planner.rs). - About 20 nouns: environment, live, local, pending, applied, snapshot, request, move, plan, prepare, stage, pointer, publication, portable provenance… Several have two meanings.
- A plan-file ritual: plan → prepare → plan again →
apply --plan FILE --yes. - Live settings live on one machine (
STORE/settings/pending.json), so a laptop and the VPS disagree. - Publishing data deploys code: SSH,
git archive,cargo buildandnpm cion the VPS, new systemd units and Caddy routes, a short outage. - Rollback does not work: old objects are deleted about 10 minutes after the switch.
- Launch indirection: bash → just →
cargo run→ launcher →cargo build→ temp worker → hiddenperform, plus exit codes 75–79 to reload the TUI. - The TUI is a second interface (5 tabs, 11 overlays, about 50 keys) on top of the CLI. Every feature is built twice, and the TUI hides the command it runs.
- Rare jobs are bolted on: the lidar archive is its own Python CLI; the Wikimedia snapshot (PR Add manual Wikimedia snapshot preparation for landmark and peak bakes #2647) adds a sixth tab and a parallel command tree.
Size today: host/obc-data ≈ 30k Rust lines (CLI 11.5k, TUI 4.5k), host/obc-data-steps ≈ 7.3k, specs/obc-data.md 5 307 lines, host/obc-bake ≈ 18k, plus the legacy Python planner. Aim: the new orchestration is much smaller. Each PR reports the lines it adds and removes.
Decisions from earlier issues: what stays and what changes
| Earlier decision | Now |
|---|---|
| #2410: content keys from code identity; "no person bumps a revision number" | Replaced. Each layer has format and revision numbers. Code changes are not a cause of staleness. |
| #2410: TUI, "no web dashboard" | Replaced. A local web page is the main interface; the CLI stays complete. The TUI is deleted. |
| #2410: one environment file per environment; Live settings in the store | Replaced. A target stores its own area, layers and refresh policies in its release. |
| #2612 decision 2: laptop installs planner services with a short outage | Replaced. A follower on the VPS reads the bucket; no SSH from the publisher, no outage. |
| #2612 decision 4: services build on the VPS from the published commit | Replaced. Service code deploys separately from data, rarely. |
| #2612 decision 5: code identity uses version strings | Replaced by format + revision. |
#2612 decision 1: dtm-* streams one pooled tile at a time; a leaf fetches only its tiles |
Kept, inside the terrain-reference prepared source. |
#2612 decision 3: never remove reference/v1/, old cell-catalog/ or old planner/ keys until equivalent data replaces them |
Kept. |
| #2410: laptop and VPS can both publish; one machine at a time | Kept, through a lock in R2. |
#2410: licence and attribution come only from data/sources.toml |
Kept. |
| Open PR #2647 (Wikimedia snapshot) | Hold. Its dump-processing logic is ported into the wiki prepared source (#2654). |
The model: six nouns
| Noun | Meaning | Example |
|---|---|---|
| Source | Input bytes with a version. A version is a fixed set of files. | Geofabrik europe/germany 2026-10-08; Copernicus GLO-30; wiki 2026-09-01 |
| Layer | What we bake from sources and other layers. It has a version. | device/map, planner/routing |
| Region | An area: a preset, Geofabrik paths, a list of them, or a box | europe/germany/baden-wuerttemberg, --box 8.15,46.48,8.46,46.72, planet |
| Target | A place that serves data | prod (R2), local (a folder on this machine) |
| Run | One background job; it is exactly one CLI command | obc data update prod on the bake VPS |
| Release | What a target serves: one manifest that names every layer output | prod release 7f3a |
Layers the operator sees
Internal steps stay as fine-grained as producers need. The operator sees these:
| Layer | Shape | Reads | Readers |
|---|---|---|---|
device/terrain |
tiled | Copernicus GLO-30, terrain-reference/* |
device (inside each map), map builder |
device/map |
tiled | OSM, land polygons, device/terrain |
device (.obcm bands and network), map builder |
device/landmarks |
tiled | OSM, wiki |
device |
device/catalog |
whole | every device/* layer |
map builder, desktop app |
planner/basemap |
whole | OSM, basemap sources | web planner, iOS |
planner/terrain |
whole | Copernicus GLO-30 | web planner, planner/sun |
planner/routing |
whole | OSM, Copernicus GLO-30 | router service, iOS offline router |
planner/search |
whole | OSM, wiki |
search service |
planner/places |
whole | planner/search |
web planner |
planner/climate |
whole, optional | ERA5-Land | web planner |
planner/snow |
whole, optional | MODIS, HR-WSI, Hansen | web planner |
planner/sun |
whole, optional | planner/terrain |
web planner |
planner/index |
whole | every planner/* layer |
every planner client |
- A tiled layer bakes per grid leaf (2^23 µdeg). A leaf whose inputs did not change is not baked again.
- A whole layer bakes once over the full coverage of the target, because a route can cross any border. A whole layer may declare named partitions (the hook for a planet router, see obc data v2 · P7 — Planet coverage #2655 and Planet router: design how the routing package scales to the whole planet #2657).
Staleness: exactly three causes
A layer in a target is stale when:
- Its version changed. The version in code differs from the version in the target's release.
- A source has a newer version to use. Either a newer version is already known (you fetched it, or you prepared a new
wiki), or the target's copy is older than the target's refresh policy and upstream has a newer one.manualonly switches off the automatic upstream check. - An input changed. A layer or source that it reads now has other bytes.
Code changes, commits and compiler versions are not causes.
Two numbers per layer, declared next to its producer:
Layer { name: "device/map", format: obcm::VERSION /* readers check it */, revision: 3, … }formatis the number that the reader checks. Where a reader constant exists, the layer uses it directly:obcm::VERSION(firmware/obc-formats/src/obcm.rs), the routerFORMAT(planner/router/src/package.rs),CATALOG_SCHEMA_VERSION(host/obc-pack/src/catalog.rs). A firmware format change therefore makes the layer stale with no other edit.revisionis changed by hand in the PR that improves the output, for example a bug fix in contours.- The UI says which:
format 19 → 20: readers need new dataorrevision 3 → 4: improved output. - Naming a layer rebuilds it.
obc data update local --only device/terrainalways bakes it again, even when nothing is stale. This is how a developer sees a code change locally without a version change. - Accepted consequence: tiles of one tiled layer can come from different producer builds. A revision change makes them uniform again.
Targets and machines are two different things
A machine is where work runs (the laptop, the bake VPS). A target is where the result is served. prod is one shared R2 bucket; every machine sees the same prod. local is a folder on the machine you are on; each machine has its own, and no other machine sees it.
update local |
update prod |
|
|---|---|---|
| on the laptop | daily development tests | allowed; the machine check guards the size |
| on the bake VPS | rarely useful | the normal prod refresh |
- Where a target is: committed in
data/targets.toml([prod] bucket, public/[local] dir). - What a target holds: its region, its optional layers and its refresh policy per source. They are stored in the target, inside its release. The UI or flags change them, and the next update remembers them. Every machine reads prod's settings from prod.
data/sources.tomlgives the default refresh policy.
Sources do not belong to a machine
A release records the exact version of every source it used. The bytes of that version live in R2, so every machine reads the same bytes. A machine's store is only a cache.
- Every redistributable source gets an R2 copy by default. Today ERA5-Land, MODIS, HR-WSI and Hansen do not, so a machine that bakes climate or snow needs its own accounts.
- The four sources that may not be redistributed (Copernicus GLO-30, Planetiler, protomaps-basemaps, nominatim-country-data) are fetched upstream by each machine, pinned by version and digest.
- Staleness is judged per target: one version can be up to date for
local(manual) and stale forprod(7 days).
Prepared sources: the rare jobs
The Wikimedia snapshot and the national terrain archive are prepared sources: long jobs that publish an immutable, versioned dataset to R2 under sources/<id>/<version>/. Bakes read them like any other source, pinned by version.
| Prepared source | Built from | Typical cadence |
|---|---|---|
wiki |
Wikidata, Wikipedia Enterprise and SQL dumps, Commons photos (logic of PR #2647) | a few times a year |
terrain-reference/<country> |
the national dtm-* sources (logic of host/obc-dem/reference/ingest.py, streamed per tile as in #2612) |
once per country |
The 24 dtm-* sources and the three Wikimedia sources are inputs of these two; the Sources screen shows them inside them. A prepared version that no kept release uses is deleted from R2.
Planet scale
Production coverage reaches the whole planet within about a year. Nothing in this design assumes a small region.
- OSM by size. A small region reads Geofabrik extracts. The planet reads
osm-planet, kept current withosm-replicationdaily diffs. Both sources exist indata/sources.tomland are unused today. - Unchanged leaves are skipped. A leaf extract must not carry the planet timestamp in its header, so a weekly refresh bakes only the leaves whose OSM content changed.
- The planet router is a separate design (Planet router: design how the routing package scales to the whole planet #2657).
planner/router-buildkeeps all source OSM objects in memory. A planet router needs a very large machine or a partitioned router. This epic only adds the "named partitions" hook for whole layers. - Bake machine. Regional prod bakes run on a bake VPS. Planet bakes need a dedicated large machine, sized after Planet router: design how the routing package scales to the whole planet #2657.
Publishing
obc data update TARGET = bake what is stale → verify → upload what is missing → write the release → switch. One review, one confirmation.
- Review instead of a plan file. It lists what is fetched, downloaded, baked and kept by reference, why each layer is stale, time, peak RAM, upload size, and what each reader sees. It has a short ID. An agent runs
obc data update prod --yes --expect REVIEW_ID; the run starts only when the review is still identical. The owner approves a review ID, not a file.localupdates need no review. - Publish from any machine. Layers that are not stale stay in the release by reference and are not downloaded. A machine downloads only what its stale layers read, at the source versions that prod uses. If the review moves a source to a newer version, the run uploads that version's copy with the release.
- Machine check. The review compares the estimated peak RAM and disk of each step (from earlier receipts) with this machine. Work that does not fit is refused before it starts, and the bake VPS is named as the place to run it.
- Lock in R2.
lock.jsonis taken and released with conditional writes. It holds the machine, run, phase, progress and a heartbeat. Before the switch, a run checks that it still owns the lock; if not, it stops. A failed or stopped run releases it. A lock with an old heartbeat can be taken over after a confirmation (obc data unlock prod --from MACHINE). - One rollback. A target keeps the release it serves and one rollback. An update makes the serving release the rollback and deletes the files of the old rollback that neither kept release names. Unchanged layers share files, so a rollback costs only what changed.
obc data rollback prodswitches; the release it leaves becomes the new rollback. - Format change. When a layer's
formatchanges, the review lists every reader, what it reads now and what happens to old readers. Start needs an acknowledgement (--accept-format-change). The cutover is complete: old devices and old iOS apps keep what they have and get no new data until they update. No old format is served beside the new one. - Resume is the same update again: finished layers are reused; if anything changed since the review, a new review comes first.
How data reaches clients and the router
Three hosts serve today, tied together by small JSON pointers:
| Host | What it is | Serves |
|---|---|---|
maps.openbikecomputer.com |
the public R2 bucket obc-maps |
every published file and the pointers |
tiles.openbikecomputer.com |
a Cloudflare Worker over the same bucket (planner/tiles) |
map tiles |
releases.openbikecomputer.com |
the VPS: Caddy → routing, search and downloads services | routing, search, offline bundles |
The web planner (builder/web/src/lib/planner/release.ts) and iOS read planner/catalog.json first and follow its URLs, which carry the release ID (…/planner-api/releases/<id>/routing). planner-service reads a folder at start (planner/service/src/main.rs) and knows nothing of R2.
New: the VPS follows the bucket. The same program that serves local on a laptop (obc data serve) runs on the VPS as obc data serve prod:
publisher (any machine) R2 obc-maps VPS: obc data serve prod
1 upload new files ────────────────► objects/…
2 write planner/next.json ─────────► next.json ── polled ────► 3 download <id>, verify,
start it beside the current one
4 wait for releases…/planner-api/releases/<id>/ready ◄────────── ready
5 write planner/catalog.json ──────► catalog.json (clients switch now)
6 keep current + 1 previous
- The follower reads public URLs only: it needs no R2 credentials. The publisher needs R2 credentials and one HTTPS request: it needs no SSH.
- Old and new releases run side by side, so there is no outage. A rollback is a pointer switch and is instant.
- If the installed service cannot read the new routing format, the follower reports
not ready: needs planner-service with routing format 9. The run waits at Router ready (up to 24 h) and does not change prod.
Interface
A web page is the main interface; the CLI has the same verbs and stays complete for agents, scripts and SSH. The page never does what the CLI cannot, and each action shows its CLI command.
obc data uiserves the page and a JSON API onlocalhostof the machine it runs on. The page acts only on that machine. The header names the machine and has its own colour.- The bake VPS runs
obc data uias a systemd service. You reach its page through one SSH tunnel (obc data open MACHINE). Nothing is public, so there is no login code. - Target state lives in R2, so every page shows prod's release, what is stale, and a publish that runs on another machine (from
lock.json), read only. - A run is a detached process that writes
runs/<id>/status.jsonandlog. Closing a browser, a terminal or an SSH tunnel never stops it.
Screens (described in full, with screenshots, in #2653): Overview (sources → layers → target as one flow diagram, coloured for the chosen target), Target prod/local (coverage map, area picker, layers, releases with rollback, Update → Review, Serve for local), Run (phases, progress per layer, cell map, log, Stop), Sources (grouped sources, policy per target, prepared sources), Machine (what this machine can do, tools, credentials, store).
obc data state of every target and the next step
obc data ui serve the page on this machine
obc data update TARGET [--region R | --box W,S,E,N] [--add L] [--remove L]
[--only L…] [--accept-format-change] [--dry-run] [--yes --expect ID]
obc data rollback TARGET
obc data serve [TARGET] serve a target: local here, prod on the VPS
obc data sources | source refresh ID | source prepare ID
obc data runs [RUN] [--follow | --stop]
obc data unlock TARGET --from MACHINE
obc data open MACHINE
obc data setup
obc data clean [--apply]
Every command has --json.
What we keep
- The content-addressed store and immutable R2 objects; upload only what is missing; the pointer is written last.
data/sources.tomlwith licences and attribution;data/regions/presets; box → Geofabrik area resolution (host/obc-data-steps/src/region_sources.rs).- Verify before publish (the catalog verify runs the real assembler and reader).
- Memory-aware step scheduling, receipts with time and peak RAM, the "Fix:" line on every error.
- Every producer crate and Python producer. Only the orchestration around them changes.
- The licence split:
host/obc-datastays MIT/Apache (tools/check_obc_data_licence.py); GPL product code stays outside it.
What gets deleted
obc bake and host/obc-bake (after its planet sharding moves into the OSM input of tiled layers), obc planner and tools/planner.py, planner_bake.py, planner_components.py, planner_deploy.py, planner_cleanup.py, planner_publish.py, tools/planner-regions/, obc r2 and tools/r2.py, the TUI, the launcher/worker split, engine/code.rs, store-local settings, saved region definitions and the region editor, plan files, obc data dev and prepare, the sources osm-trails and mapzen-terrain-tiles, and .github/workflows/bake.yml (or a rewrite that calls obc data update).
Work order
- obc data v2 · P1 — Layers, versions and staleness #2649 — P1 Layers, versions and staleness
- obc data v2 · P2 — Targets, releases, update, lock and rollback #2650 — P2 Targets, releases, update, lock and rollback
- obc data v2 · P3 — The local target and obc data serve #2651 — P3 The local target and
obc data serve - obc data v2 · P4 — The VPS follows the bucket; service code deploys separately #2652 — P4 The VPS follows the bucket; service code deploys separately
- obc data v2 · P5 — The operator page (obc data ui) #2653 — P5 The operator page (
obc data ui) - obc data v2 · P6 — Prepared sources: wiki and terrain-reference #2654 — P6 Prepared sources:
wikiandterrain-reference - obc data v2 · P7 — Planet coverage #2655 — P7 Planet coverage
- obc data v2 · P8 — Cutover, deletion and documentation #2656 — P8 Cutover, deletion and documentation
P1 → P2 → P3 → P4
P2 → P5 (the page grows with P3, P4 and P6)
P2 → P6
P1 → P7 (a planet router needs its own design: #2657)
P1–P6 → P8
Rules for every PR in this epic
- Breaking changes to the store, the release format and the CLI are fine: nothing published by
obc datais in production. The settings blob in device RRAM is not touched. - Never remove
reference/v1/, oldcell-catalog/or oldplanner/keys until equivalent new data is live (obc data: fix the first-bake blockers and take the simple path #2612 decision 3). - Each phase updates the part of
specs/obc-data.mdthat it changes. The spec states bytes and layouts only: store, release manifest, pointers, lock, target layout, prepared-source layout, follower protocol. - Use the
local(dir) target as the test double for R2. Test the crates you change, asAGENTS.mdsays. Focused tests only: one per staleness cause, lock races, rollback, review mismatch, follower protocol. - Report added and removed lines.
- Each PR carries
Requirements: …(checkobc reqbefore the first PR; map coverage and attribution requirements are likely).
Out of scope
- The planet router design: Planet router: design how the routing package scales to the whole planet #2657.
- A hosted control page with logins.
- A remote-control channel between machines. The VPS is operated through its own page or SSH.
- Serving an old format beside a new one.
- New data sources or layers.
Wireframes
The owner approved two rounds of wireframes. Every implementer can use them:
- In the repository:
wireframes/obc-data/at commit 239bb3b, on branchclaude/beautiful-sagan-5tcij7(a reference, not for merging). It holds the screen sources, a clickable viewer (python3 -m http.server --directory wireframes/obc-data) and 23 screenshots. Keep the branch until obc data v2 · P5 — The operator page (obc data ui) #2653 is done. - In obc data v2 · P5 — The operator page (obc data ui) #2653: every screen described in words, with the screenshots embedded.
- The original canvas (owner's claude.ai account): https://claude.ai/artifact/9VAx3V1k9BS78BqRFrf4ui
Activity
timohueser commentedon Nov 4, 2026
This epic is closed.
docs/content/software/data.mdandspecs/obc-data.md.Generated by Claude Code