Repository navigation
Description
Part of #2648. Read the epic first: it holds the model and the decisions.
Goal
obc data update TARGET makes a target match its settings in one reviewed step, from any machine. A target keeps one rollback. Two machines never publish at once.
Scope
-
Targets.
data/targets.tomlsays where each target is:[prod] bucket, publicand[local] dir. Two kinds:r2anddir. Adirtarget is also the test double for R2. -
Release manifest. Its ID is the SHA-256 of its bytes. It holds:
- the target settings: a copy of the region definition, the optional layers, a refresh policy per source;
- every layer with format, revision, key and files;
- every source version with its digest;
- the machine and time.
Proposed bucket layout (define it in
specs/obc-data.md):release.json {serving, rollback}; written last releases/<id>.json objects/<sha256> sources/<id>/<version>/… source copies and prepared sources lock.json cell-catalog/catalog.json client pointer from device/catalog planner/catalog.json client pointer from planner/indexLegacy keys (
reference/v1/, oldcell-catalog/andplanner/objects) are never touched. -
Target settings live in the target.
--region,--box,--add Land--remove Lchange them; the next update remembers them. Every machine reads prod's settings from prod. A region is a preset indata/regions/, one or more Geofabrik paths, or a box. Delete the saved definitions inSTORE/regions/and the region editor (cli/regions_cli.rs,regions_cli/). -
Refresh policy per target (days or
manual), stored with the target settings.data/sources.tomlgives the default. -
Source copies. Every redistributable source gets an R2 copy by default (today ERA5-Land, MODIS, HR-WSI and Hansen do not). Non-redistributable sources are fetched upstream, pinned by version and digest.
-
Review. It lists fetch, download, bake (with each cause), keep by reference, cost (time, peak RAM, upload), readers, and after (rollback and deletions). Its ID is the hash of its content and the base release ID. Use
--dry-run,--json, and--yes --expect ID; refuse when the review changed.localupdates need no review. -
Publish from any machine. Layers that are not stale stay by reference and are not downloaded. Download only what stale layers read, at the source versions the target uses. A source moved to a newer version uploads its copy with the release.
-
Machine check. Compare the estimated peak RAM and disk of each step (from earlier receipts) with this machine. Refuse work that does not fit before it starts. With no estimate, say so.
-
Lock.
lock.jsonis taken and released with conditional writes (ETag). It holds machine, run, phase, progress and a heartbeat (each minute). Before the switch, a run checks that it still owns the lock. A failed or stopped run releases it.obc data unlock TARGET --from MACHINEtakes over a lock with an old heartbeat after a confirmation. -
Rollback. Keep the serving release and one rollback. An update makes the serving release the rollback and deletes the files that neither kept release names.
obc data rollback TARGETswaps the two. -
Format change. When a layer's
formatchanges, the review lists its readers and what happens to old ones. Starting needs--accept-format-change. -
Runs. A run is one CLI command, started as a detached process. It writes
runs/<id>/status.jsonandlog. Supportobc data runs [RUN] [--follow | --stop]. A resume is the same update again. -
obc datawith no arguments prints every target's state and the next command to run. -
Rule in
AGENTS.md. Replace "Runobc data apply liveonly with a plan that the owner approved" with "Publish to prod only with a review ID that the owner approved (obc data update prod --yes --expect ID)." -
Delete:
settings.rs(pending and applied settings),env.rs,data/env/;- plan files,
cli/apply_cli.rs, the plan logic incli/build_cli.rs,cli/edit_cli.rs; - the
prepareoperation and the old run journals (operation.rs,cli/operation_cli*); - the old pointer handling in
live.rs.
Carry over these pitfalls from #2612, slice L:
- a small box must not resolve to a Geofabrik meta-area such as
europe/dach; - regions without
countriesortime_zonemust work; - Stop interrupts a fetch;
- the disk check works on macOS;
partial/is collected.
Do not
Do not update the real prod bucket in this phase: the router follower (P4) does not exist yet. Test against dir targets and a throwaway bucket.
Acceptance
Focused tests on a dir target:
- two concurrent updates: one wins, one stops;
- a takeover of an old lock: the old run refuses the switch;
- a rollback swap;
- review mismatch refusal;
- no downloads for unchanged layers;
- a machine-check refusal;
- a format change without acknowledgement is refused.
specs/obc-data.md defines the release manifest, the bucket layout, the lock and the review ID. Run obc test -p obc-data, obc test -p obc-data-steps and clippy. Report added and removed lines.
Depends on
P1.
Activity
timohueser commentedon Oct 12, 2026
Rules that P2 takes over from the P1 reviews (#2659, #2660) and the P2-1a review (#2661):
--onlycan give a key other bytes in the store.--only Lon a target. It also bakes the stale inputs of L. A release never holds a named layer that was built from inputs the release does not hold.manualsource, or when a due policy finds it upstream.--moveis removed.--accept-format-changeapplies only to a version cause withfield: formatandfrom != null. A new layer (from: null) needs no acknowledgement.cell-catalog/catalog.jsonandplanner/catalog.json. Each pointer must be a layer output.Generated by Claude Code
timohueser commentedon Oct 12, 2026
P2-1a is merged in #2661. It adds library code only; no command uses it yet. The diff is +851 / −2 lines.
data/targets.tomllists the targets, andtarget.rsaddsTarget::DirandTarget::R2. A dir target does conditional writes under a lock, with a SHA-256 tag. An R2 target reads only, through its public URL, until P2-4.release.rsadds the target release manifest.release.jsonfirst, then the two client pointers. A switch is refused only when a pointer holds bytes that neither the serving release nor the new release names, so a rollback after a stopped switch works.releases/<folder>/, where the folder is the digest of its files.cell-catalog/catalog.jsonandplanner/catalog.json. Every release names both.Next is P2-1b:
obc data updateon dir targets, which replaces the plan, apply and live flow.Generated by Claude Code
timohueser commentedon Oct 12, 2026
This item is added to P2-3, from the #2665 review. It must land before P2-4 switches on prod writes.
obc data source refresh IDis in the epic's command list. It fetches the newest upstream version of a source, so that the next update of a target can use it.natural-earth,daylight-landcoverandpgf-encoding.startinsources.tomlchanges; feat(obc-data): update a dir target and keep unchanged outputs by reference (P2-1b) #2665 does this.obc data source refresh ID. The review lists the move with its cause.Generated by Claude Code
timohueser commentedon Oct 12, 2026
P2-1b is merged in two pull requests.
#2662: renames only.
Input::Layeris nowInput::Step, the input kind in the key is nowstep, andmapsis nowdevice. It also fixesplanner/search/build.py, which now selects content by file. Before, a renamed step would have built every POI database with no content.#2665:
obc data update TARGET. It writesdirtargets only and refuses R2 until P2-4. The diff is +3681 / −13867 lines.Settings in the target. The settings live in the release manifest.
--region,--box,--add,--removeand--refresh SOURCE=DAYS|manualchange them, and the next update remembers them.Versions. Three rules, stated in the spec:
start. Astartthat differs from serving moves unordered sources too.Release and date versions are compared by their numeric parts.
Kept layers. The update judges every layer against the serving release. Kept layers are read from the target release's files.
--only Lalso bakes the stale inputs of L, at any depth.Pointers. The device pointer uses
/objects/<sha>URLs; this isdevice/catalogrevision 2. A newplanner/catalogstep writes the planner pointer. The planner release id is the digest of the named files.Overview and store.
obc datajudges each target and prints amachineblock. The store layout is now 2, with a Fix line that keeps the snapshots.Tiles Worker. The Worker and
local.mjsread the new layout. Do not deploy the Worker before P2-4.Deleted.
data/env, plan, prepare, build and apply,--move, input copies, detached operations,live.rs, the store-local settings and the per-area Wikimedia captures.Next: P2-2 rollback and collection, then P2-3 review, machine check, source copies and
source refresh, then P2-4 lock, runs and R2 writes. P2-4 needs the owner's scratch bucket.Generated by Claude Code
timohueser commentedon Oct 12, 2026
P2 is complete. It merged in 12 pull requests: +11,783 / −16,135 lines.
--dry-run/--expect, machine check, format-change gateobc data source refreshsources/with licence and attributionlock.json, heartbeat, takeover withunlock --fromruns/<id>/{status.json, log},runs --follow/--stopThe acceptance tests run on
dirtargets. They cover:The spec defines the release manifest, the bucket layout, the lock, the review ID and the runs.
Known gaps:
tests/r2_live.rsis opt-in (--ignored) and has not run yet. It needs the R2 credentials, and it writes only under atest/prefix.release.json.Generated by Claude Code