Skip to content

EPIC — obc data, second design: versioned layers, targets and one operator page #2648

Closed
7 of 8 issues completed

Description

@timohueser

Goal

One system bakes, updates and publishes all map data, for the device and for the planner. It runs the rare jobs too: the Wikimedia snapshot and the national lidar terrain archive. It works the same on a laptop and on a bake VPS, for areas from one town up to the whole planet. A new person learns it in one hour. The rule for "what is stale" fits in three lines.

This epic replaces the design of #2410 and the plan of #2612. The sub-issues below are the work order. Each one is self-contained, so a fresh session can start with any phase whose dependencies are done.

Why a second design

obc data works, but it is hard to operate and hard to teach. It has never published to production (#2612), so breaking changes cost nothing now.

  1. Four entry points write the same bucket: obc data, obc bake (host/obc-bake), obc planner (tools/planner.py) and obc r2 (tools/r2.py). There are two VPS deploy models (tools/planner_publish.py, tools/planner_deploy.py) and three identity schemes.
  2. Staleness is a code hash that nobody can predict. A layer key hashes git-listed files, the Cargo closure, the compiler, RUSTFLAGS, the uv lock and the Python ABI (host/obc-data/src/engine/code.rs). A refactor rebuilds layers whose bytes do not change. Python steps need hand-written import lists that a regex test polices (host/obc-data-steps/src/planner.rs).
  3. About 20 nouns: environment, live, local, pending, applied, snapshot, request, move, plan, prepare, stage, pointer, publication, portable provenance… Several have two meanings.
  4. A plan-file ritual: plan → prepare → plan again → apply --plan FILE --yes.
  5. Live settings live on one machine (STORE/settings/pending.json), so a laptop and the VPS disagree.
  6. Publishing data deploys code: SSH, git archive, cargo build and npm ci on the VPS, new systemd units and Caddy routes, a short outage.
  7. Rollback does not work: old objects are deleted about 10 minutes after the switch.
  8. Launch indirection: bash → just → cargo run → launcher → cargo build → temp worker → hidden perform, plus exit codes 75–79 to reload the TUI.
  9. The TUI is a second interface (5 tabs, 11 overlays, about 50 keys) on top of the CLI. Every feature is built twice, and the TUI hides the command it runs.
  10. Rare jobs are bolted on: the lidar archive is its own Python CLI; the Wikimedia snapshot (PR Add manual Wikimedia snapshot preparation for landmark and peak bakes #2647) adds a sixth tab and a parallel command tree.

Size today: host/obc-data ≈ 30k Rust lines (CLI 11.5k, TUI 4.5k), host/obc-data-steps ≈ 7.3k, specs/obc-data.md 5 307 lines, host/obc-bake ≈ 18k, plus the legacy Python planner. Aim: the new orchestration is much smaller. Each PR reports the lines it adds and removes.

Decisions from earlier issues: what stays and what changes

Earlier decision Now
#2410: content keys from code identity; "no person bumps a revision number" Replaced. Each layer has format and revision numbers. Code changes are not a cause of staleness.
#2410: TUI, "no web dashboard" Replaced. A local web page is the main interface; the CLI stays complete. The TUI is deleted.
#2410: one environment file per environment; Live settings in the store Replaced. A target stores its own area, layers and refresh policies in its release.
#2612 decision 2: laptop installs planner services with a short outage Replaced. A follower on the VPS reads the bucket; no SSH from the publisher, no outage.
#2612 decision 4: services build on the VPS from the published commit Replaced. Service code deploys separately from data, rarely.
#2612 decision 5: code identity uses version strings Replaced by format + revision.
#2612 decision 1: dtm-* streams one pooled tile at a time; a leaf fetches only its tiles Kept, inside the terrain-reference prepared source.
#2612 decision 3: never remove reference/v1/, old cell-catalog/ or old planner/ keys until equivalent data replaces them Kept.
#2410: laptop and VPS can both publish; one machine at a time Kept, through a lock in R2.
#2410: licence and attribution come only from data/sources.toml Kept.
Open PR #2647 (Wikimedia snapshot) Hold. Its dump-processing logic is ported into the wiki prepared source (#2654).

The model: six nouns

Noun Meaning Example
Source Input bytes with a version. A version is a fixed set of files. Geofabrik europe/germany 2026-10-08; Copernicus GLO-30; wiki 2026-09-01
Layer What we bake from sources and other layers. It has a version. device/map, planner/routing
Region An area: a preset, Geofabrik paths, a list of them, or a box europe/germany/baden-wuerttemberg, --box 8.15,46.48,8.46,46.72, planet
Target A place that serves data prod (R2), local (a folder on this machine)
Run One background job; it is exactly one CLI command obc data update prod on the bake VPS
Release What a target serves: one manifest that names every layer output prod release 7f3a

Layers the operator sees

Internal steps stay as fine-grained as producers need. The operator sees these:

Layer Shape Reads Readers
device/terrain tiled Copernicus GLO-30, terrain-reference/* device (inside each map), map builder
device/map tiled OSM, land polygons, device/terrain device (.obcm bands and network), map builder
device/landmarks tiled OSM, wiki device
device/catalog whole every device/* layer map builder, desktop app
planner/basemap whole OSM, basemap sources web planner, iOS
planner/terrain whole Copernicus GLO-30 web planner, planner/sun
planner/routing whole OSM, Copernicus GLO-30 router service, iOS offline router
planner/search whole OSM, wiki search service
planner/places whole planner/search web planner
planner/climate whole, optional ERA5-Land web planner
planner/snow whole, optional MODIS, HR-WSI, Hansen web planner
planner/sun whole, optional planner/terrain web planner
planner/index whole every planner/* layer every planner client

Staleness: exactly three causes

A layer in a target is stale when:

  1. Its version changed. The version in code differs from the version in the target's release.
  2. A source has a newer version to use. Either a newer version is already known (you fetched it, or you prepared a new wiki), or the target's copy is older than the target's refresh policy and upstream has a newer one. manual only switches off the automatic upstream check.
  3. An input changed. A layer or source that it reads now has other bytes.

Code changes, commits and compiler versions are not causes.

Two numbers per layer, declared next to its producer:

Layer { name: "device/map", format: obcm::VERSION /* readers check it */, revision: 3, … }
  • format is the number that the reader checks. Where a reader constant exists, the layer uses it directly: obcm::VERSION (firmware/obc-formats/src/obcm.rs), the router FORMAT (planner/router/src/package.rs), CATALOG_SCHEMA_VERSION (host/obc-pack/src/catalog.rs). A firmware format change therefore makes the layer stale with no other edit.
  • revision is changed by hand in the PR that improves the output, for example a bug fix in contours.
  • The UI says which: format 19 → 20: readers need new data or revision 3 → 4: improved output.
  • Naming a layer rebuilds it. obc data update local --only device/terrain always bakes it again, even when nothing is stale. This is how a developer sees a code change locally without a version change.
  • Accepted consequence: tiles of one tiled layer can come from different producer builds. A revision change makes them uniform again.

Targets and machines are two different things

A machine is where work runs (the laptop, the bake VPS). A target is where the result is served. prod is one shared R2 bucket; every machine sees the same prod. local is a folder on the machine you are on; each machine has its own, and no other machine sees it.

update local update prod
on the laptop daily development tests allowed; the machine check guards the size
on the bake VPS rarely useful the normal prod refresh
  • Where a target is: committed in data/targets.toml ([prod] bucket, public / [local] dir).
  • What a target holds: its region, its optional layers and its refresh policy per source. They are stored in the target, inside its release. The UI or flags change them, and the next update remembers them. Every machine reads prod's settings from prod. data/sources.toml gives the default refresh policy.

Sources do not belong to a machine

A release records the exact version of every source it used. The bytes of that version live in R2, so every machine reads the same bytes. A machine's store is only a cache.

  • Every redistributable source gets an R2 copy by default. Today ERA5-Land, MODIS, HR-WSI and Hansen do not, so a machine that bakes climate or snow needs its own accounts.
  • The four sources that may not be redistributed (Copernicus GLO-30, Planetiler, protomaps-basemaps, nominatim-country-data) are fetched upstream by each machine, pinned by version and digest.
  • Staleness is judged per target: one version can be up to date for local (manual) and stale for prod (7 days).

Prepared sources: the rare jobs

The Wikimedia snapshot and the national terrain archive are prepared sources: long jobs that publish an immutable, versioned dataset to R2 under sources/<id>/<version>/. Bakes read them like any other source, pinned by version.

Prepared source Built from Typical cadence
wiki Wikidata, Wikipedia Enterprise and SQL dumps, Commons photos (logic of PR #2647) a few times a year
terrain-reference/<country> the national dtm-* sources (logic of host/obc-dem/reference/ingest.py, streamed per tile as in #2612) once per country

The 24 dtm-* sources and the three Wikimedia sources are inputs of these two; the Sources screen shows them inside them. A prepared version that no kept release uses is deleted from R2.

Planet scale

Production coverage reaches the whole planet within about a year. Nothing in this design assumes a small region.

  • OSM by size. A small region reads Geofabrik extracts. The planet reads osm-planet, kept current with osm-replication daily diffs. Both sources exist in data/sources.toml and are unused today.
  • Unchanged leaves are skipped. A leaf extract must not carry the planet timestamp in its header, so a weekly refresh bakes only the leaves whose OSM content changed.
  • The planet router is a separate design (Planet router: design how the routing package scales to the whole planet #2657). planner/router-build keeps all source OSM objects in memory. A planet router needs a very large machine or a partitioned router. This epic only adds the "named partitions" hook for whole layers.
  • Bake machine. Regional prod bakes run on a bake VPS. Planet bakes need a dedicated large machine, sized after Planet router: design how the routing package scales to the whole planet #2657.

Publishing

obc data update TARGET = bake what is stale → verify → upload what is missing → write the release → switch. One review, one confirmation.

  • Review instead of a plan file. It lists what is fetched, downloaded, baked and kept by reference, why each layer is stale, time, peak RAM, upload size, and what each reader sees. It has a short ID. An agent runs obc data update prod --yes --expect REVIEW_ID; the run starts only when the review is still identical. The owner approves a review ID, not a file. local updates need no review.
  • Publish from any machine. Layers that are not stale stay in the release by reference and are not downloaded. A machine downloads only what its stale layers read, at the source versions that prod uses. If the review moves a source to a newer version, the run uploads that version's copy with the release.
  • Machine check. The review compares the estimated peak RAM and disk of each step (from earlier receipts) with this machine. Work that does not fit is refused before it starts, and the bake VPS is named as the place to run it.
  • Lock in R2. lock.json is taken and released with conditional writes. It holds the machine, run, phase, progress and a heartbeat. Before the switch, a run checks that it still owns the lock; if not, it stops. A failed or stopped run releases it. A lock with an old heartbeat can be taken over after a confirmation (obc data unlock prod --from MACHINE).
  • One rollback. A target keeps the release it serves and one rollback. An update makes the serving release the rollback and deletes the files of the old rollback that neither kept release names. Unchanged layers share files, so a rollback costs only what changed. obc data rollback prod switches; the release it leaves becomes the new rollback.
  • Format change. When a layer's format changes, the review lists every reader, what it reads now and what happens to old readers. Start needs an acknowledgement (--accept-format-change). The cutover is complete: old devices and old iOS apps keep what they have and get no new data until they update. No old format is served beside the new one.
  • Resume is the same update again: finished layers are reused; if anything changed since the review, a new review comes first.

How data reaches clients and the router

Three hosts serve today, tied together by small JSON pointers:

Host What it is Serves
maps.openbikecomputer.com the public R2 bucket obc-maps every published file and the pointers
tiles.openbikecomputer.com a Cloudflare Worker over the same bucket (planner/tiles) map tiles
releases.openbikecomputer.com the VPS: Caddy → routing, search and downloads services routing, search, offline bundles

The web planner (builder/web/src/lib/planner/release.ts) and iOS read planner/catalog.json first and follow its URLs, which carry the release ID (…/planner-api/releases/<id>/routing). planner-service reads a folder at start (planner/service/src/main.rs) and knows nothing of R2.

New: the VPS follows the bucket. The same program that serves local on a laptop (obc data serve) runs on the VPS as obc data serve prod:

publisher (any machine)               R2 obc-maps                 VPS: obc data serve prod
1 upload new files ────────────────►  objects/…
2 write planner/next.json ─────────►  next.json ── polled ────►  3 download <id>, verify,
                                                                    start it beside the current one
4 wait for releases…/planner-api/releases/<id>/ready  ◄────────── ready
5 write planner/catalog.json ──────►  catalog.json (clients switch now)
                                                                  6 keep current + 1 previous
  • The follower reads public URLs only: it needs no R2 credentials. The publisher needs R2 credentials and one HTTPS request: it needs no SSH.
  • Old and new releases run side by side, so there is no outage. A rollback is a pointer switch and is instant.
  • If the installed service cannot read the new routing format, the follower reports not ready: needs planner-service with routing format 9. The run waits at Router ready (up to 24 h) and does not change prod.

Interface

A web page is the main interface; the CLI has the same verbs and stays complete for agents, scripts and SSH. The page never does what the CLI cannot, and each action shows its CLI command.

  • obc data ui serves the page and a JSON API on localhost of the machine it runs on. The page acts only on that machine. The header names the machine and has its own colour.
  • The bake VPS runs obc data ui as a systemd service. You reach its page through one SSH tunnel (obc data open MACHINE). Nothing is public, so there is no login code.
  • Target state lives in R2, so every page shows prod's release, what is stale, and a publish that runs on another machine (from lock.json), read only.
  • A run is a detached process that writes runs/<id>/status.json and log. Closing a browser, a terminal or an SSH tunnel never stops it.

Screens (described in full, with screenshots, in #2653): Overview (sources → layers → target as one flow diagram, coloured for the chosen target), Target prod/local (coverage map, area picker, layers, releases with rollback, Update → Review, Serve for local), Run (phases, progress per layer, cell map, log, Stop), Sources (grouped sources, policy per target, prepared sources), Machine (what this machine can do, tools, credentials, store).

obc data                                     state of every target and the next step
obc data ui                                  serve the page on this machine
obc data update TARGET [--region R | --box W,S,E,N] [--add L] [--remove L]
                       [--only L…] [--accept-format-change] [--dry-run] [--yes --expect ID]
obc data rollback TARGET
obc data serve [TARGET]                      serve a target: local here, prod on the VPS
obc data sources | source refresh ID | source prepare ID
obc data runs [RUN] [--follow | --stop]
obc data unlock TARGET --from MACHINE
obc data open MACHINE
obc data setup
obc data clean [--apply]

Every command has --json.

What we keep

  • The content-addressed store and immutable R2 objects; upload only what is missing; the pointer is written last.
  • data/sources.toml with licences and attribution; data/regions/ presets; box → Geofabrik area resolution (host/obc-data-steps/src/region_sources.rs).
  • Verify before publish (the catalog verify runs the real assembler and reader).
  • Memory-aware step scheduling, receipts with time and peak RAM, the "Fix:" line on every error.
  • Every producer crate and Python producer. Only the orchestration around them changes.
  • The licence split: host/obc-data stays MIT/Apache (tools/check_obc_data_licence.py); GPL product code stays outside it.

What gets deleted

obc bake and host/obc-bake (after its planet sharding moves into the OSM input of tiled layers), obc planner and tools/planner.py, planner_bake.py, planner_components.py, planner_deploy.py, planner_cleanup.py, planner_publish.py, tools/planner-regions/, obc r2 and tools/r2.py, the TUI, the launcher/worker split, engine/code.rs, store-local settings, saved region definitions and the region editor, plan files, obc data dev and prepare, the sources osm-trails and mapzen-terrain-tiles, and .github/workflows/bake.yml (or a rewrite that calls obc data update).

Work order

P1 → P2 → P3 → P4
     P2 → P5   (the page grows with P3, P4 and P6)
     P2 → P6
P1 → P7        (a planet router needs its own design: #2657)
P1–P6 → P8

Rules for every PR in this epic

  • Breaking changes to the store, the release format and the CLI are fine: nothing published by obc data is in production. The settings blob in device RRAM is not touched.
  • Never remove reference/v1/, old cell-catalog/ or old planner/ keys until equivalent new data is live (obc data: fix the first-bake blockers and take the simple path #2612 decision 3).
  • Each phase updates the part of specs/obc-data.md that it changes. The spec states bytes and layouts only: store, release manifest, pointers, lock, target layout, prepared-source layout, follower protocol.
  • Use the local (dir) target as the test double for R2. Test the crates you change, as AGENTS.md says. Focused tests only: one per staleness cause, lock races, rollback, review mismatch, follower protocol.
  • Report added and removed lines.
  • Each PR carries Requirements: … (check obc req before the first PR; map coverage and attribution requirements are likely).

Out of scope

Wireframes

The owner approved two rounds of wireframes. Every implementer can use them:

Activity

  1. added
    enhancementNew feature or request
    maintainabilityReadability, structure, duplication
    epicTracking issue spanning multiple sub-issues
    infraTooling, scaffold, CI, dev harness
    on Oct 12, 2026
  2. timohueser commented on Nov 4, 2026

    @timohueser
    OwnerAuthor

    This epic is closed.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestepicTracking issue spanning multiple sub-issuesinfraTooling, scaffold, CI, dev harnessmaintainabilityReadability, structure, duplication

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions