Skip to content

obc data v2 · P2 — Targets, releases, update, lock and rollback #2650

Closed

Description

@timohueser

Part of #2648. Read the epic first: it holds the model and the decisions.

Goal

obc data update TARGET makes a target match its settings in one reviewed step, from any machine. A target keeps one rollback. Two machines never publish at once.

Scope

  1. Targets. data/targets.toml says where each target is: [prod] bucket, public and [local] dir. Two kinds: r2 and dir. A dir target is also the test double for R2.

  2. Release manifest. Its ID is the SHA-256 of its bytes. It holds:

    • the target settings: a copy of the region definition, the optional layers, a refresh policy per source;
    • every layer with format, revision, key and files;
    • every source version with its digest;
    • the machine and time.

    Proposed bucket layout (define it in specs/obc-data.md):

    release.json                {serving, rollback}; written last
    releases/<id>.json
    objects/<sha256>
    sources/<id>/<version>/…    source copies and prepared sources
    lock.json
    cell-catalog/catalog.json   client pointer from device/catalog
    planner/catalog.json        client pointer from planner/index
    

    Legacy keys (reference/v1/, old cell-catalog/ and planner/ objects) are never touched.

  3. Target settings live in the target. --region, --box, --add L and --remove L change them; the next update remembers them. Every machine reads prod's settings from prod. A region is a preset in data/regions/, one or more Geofabrik paths, or a box. Delete the saved definitions in STORE/regions/ and the region editor (cli/regions_cli.rs, regions_cli/).

  4. Refresh policy per target (days or manual), stored with the target settings. data/sources.toml gives the default.

  5. Source copies. Every redistributable source gets an R2 copy by default (today ERA5-Land, MODIS, HR-WSI and Hansen do not). Non-redistributable sources are fetched upstream, pinned by version and digest.

  6. Review. It lists fetch, download, bake (with each cause), keep by reference, cost (time, peak RAM, upload), readers, and after (rollback and deletions). Its ID is the hash of its content and the base release ID. Use --dry-run, --json, and --yes --expect ID; refuse when the review changed. local updates need no review.

  7. Publish from any machine. Layers that are not stale stay by reference and are not downloaded. Download only what stale layers read, at the source versions the target uses. A source moved to a newer version uploads its copy with the release.

  8. Machine check. Compare the estimated peak RAM and disk of each step (from earlier receipts) with this machine. Refuse work that does not fit before it starts. With no estimate, say so.

  9. Lock. lock.json is taken and released with conditional writes (ETag). It holds machine, run, phase, progress and a heartbeat (each minute). Before the switch, a run checks that it still owns the lock. A failed or stopped run releases it. obc data unlock TARGET --from MACHINE takes over a lock with an old heartbeat after a confirmation.

  10. Rollback. Keep the serving release and one rollback. An update makes the serving release the rollback and deletes the files that neither kept release names. obc data rollback TARGET swaps the two.

  11. Format change. When a layer's format changes, the review lists its readers and what happens to old ones. Starting needs --accept-format-change.

  12. Runs. A run is one CLI command, started as a detached process. It writes runs/<id>/status.json and log. Support obc data runs [RUN] [--follow | --stop]. A resume is the same update again.

  13. obc data with no arguments prints every target's state and the next command to run.

  14. Rule in AGENTS.md. Replace "Run obc data apply live only with a plan that the owner approved" with "Publish to prod only with a review ID that the owner approved (obc data update prod --yes --expect ID)."

  15. Delete:

    • settings.rs (pending and applied settings), env.rs, data/env/;
    • plan files, cli/apply_cli.rs, the plan logic in cli/build_cli.rs, cli/edit_cli.rs;
    • the prepare operation and the old run journals (operation.rs, cli/operation_cli*);
    • the old pointer handling in live.rs.

Carry over these pitfalls from #2612, slice L:

  • a small box must not resolve to a Geofabrik meta-area such as europe/dach;
  • regions without countries or time_zone must work;
  • Stop interrupts a fetch;
  • the disk check works on macOS;
  • partial/ is collected.

Do not

Do not update the real prod bucket in this phase: the router follower (P4) does not exist yet. Test against dir targets and a throwaway bucket.

Acceptance

Focused tests on a dir target:

  • two concurrent updates: one wins, one stops;
  • a takeover of an old lock: the old run refuses the switch;
  • a rollback swap;
  • review mismatch refusal;
  • no downloads for unchanged layers;
  • a machine-check refusal;
  • a format change without acknowledgement is refused.

specs/obc-data.md defines the release manifest, the bucket layout, the lock and the review ID. Run obc test -p obc-data, obc test -p obc-data-steps and clippy. Report added and removed lines.

Depends on

P1.

Activity

  1. timohueser commented on Oct 12, 2026

    @timohueser
    OwnerAuthor

    Rules that P2 takes over from the P1 reviews (#2659, #2660) and the P2-1a review (#2661):

    • Inputs kept by reference. An update keeps a layer by reference. A step that reads that layer reads the target release's files of it. It never reads the store's entry for the same key: --only can give a key other bytes in the store.
    • --only L on a target. It also bakes the stale inputs of L. A release never holds a named layer that was built from inputs the release does not hold.
    • Cause 2, rest of the work. A newer version is known when this machine's store has it, also for a manual source, or when a due policy finds it upstream. --move is removed.
    • Format gate. --accept-format-change applies only to a version cause with field: format and from != null. A new layer (from: null) needs no acknowledgement.
    • Client files only. Only client files go to a target, because the bucket is public and some outputs come from sources that are not redistributable. When a stale step reads a non-client output of a kept layer, this machine bakes that output at the release's source versions. The review lists it.
    • Pointer keys. They are an allowlist of exactly cell-catalog/catalog.json and planner/catalog.json. Each pointer must be a layer output.

    Generated by Claude Code

  2. timohueser commented on Oct 12, 2026

    @timohueser
    OwnerAuthor

    P2-1a is merged in #2661. It adds library code only; no command uses it yet. The diff is +851 / −2 lines.

    • data/targets.toml lists the targets, and target.rs adds Target::Dir and Target::R2. A dir target does conditional writes under a lock, with a SHA-256 tag. An R2 target reads only, through its public URL, until P2-4.
    • release.rs adds the target release manifest.
    • Publish uploads only missing keys, in one batch with a size check. It writes the manifest last.
    • Switch writes release.json first, then the two client pointers. A switch is refused only when a pointer holds bytes that neither the serving release nor the new release names, so a rollback after a stopped switch works.
    • Named files go under releases/<folder>/, where the folder is the digest of its files.
    • Pointers are an allowlist of exactly cell-catalog/catalog.json and planner/catalog.json. Every release names both.
    • Client files only go to a target.

    Next is P2-1b: obc data update on dir targets, which replaces the plan, apply and live flow.


    Generated by Claude Code

  3. timohueser commented on Oct 12, 2026

    @timohueser
    OwnerAuthor

    This item is added to P2-3, from the #2665 review. It must land before P2-4 switches on prod writes.

    obc data source refresh ID is in the epic's command list. It fetches the newest upstream version of a source, so that the next update of a target can use it.


    Generated by Claude Code

  4. timohueser commented on Oct 12, 2026

    @timohueser
    OwnerAuthor

    P2-1b is merged in two pull requests.

    #2662: renames only. Input::Layer is now Input::Step, the input kind in the key is now step, and maps is now device. It also fixes planner/search/build.py, which now selects content by file. Before, a renamed step would have built every POI database with no content.

    #2665: obc data update TARGET. It writes dir targets only and refuses R2 until P2-4. The diff is +3681 / −13867 lines.

    • Settings in the target. The settings live in the release manifest. --region, --box, --add, --remove and --refresh SOURCE=DAYS|manual change them, and the next update remembers them.

    • Versions. Three rules, stated in the spec:

      1. A due fetch reads the newest upstream version.
      2. Otherwise the update takes the newest of serving, store and start. A start that differs from serving moves unordered sources too.
      3. Otherwise it takes the newest upstream version.

      Release and date versions are compared by their numeric parts.

    • Kept layers. The update judges every layer against the serving release. Kept layers are read from the target release's files. --only L also bakes the stale inputs of L, at any depth.

    • Pointers. The device pointer uses /objects/<sha> URLs; this is device/catalog revision 2. A new planner/catalog step writes the planner pointer. The planner release id is the digest of the named files.

    • Overview and store. obc data judges each target and prints a machine block. The store layout is now 2, with a Fix line that keeps the snapshots.

    • Tiles Worker. The Worker and local.mjs read the new layout. Do not deploy the Worker before P2-4.

    • Deleted. data/env, plan, prepare, build and apply, --move, input copies, detached operations, live.rs, the store-local settings and the per-area Wikimedia captures.

    Next: P2-2 rollback and collection, then P2-3 review, machine check, source copies and source refresh, then P2-4 lock, runs and R2 writes. P2-4 needs the owner's scratch bucket.


    Generated by Claude Code

  5. timohueser commented on Oct 12, 2026

    @timohueser
    OwnerAuthor

    P2 is complete. It merged in 12 pull requests: +11,783 / −16,135 lines.

    PR Part
    #2661, #2662, #2665 P2-1: targets, release manifest, update, target settings and refresh policy; deletion of the plan, apply, edit, operation, env and region editor code
    #2666 P2-2: rollback; collection with a 10-minute grace; legacy keys never touched
    #2668 P2-3a: review and review ID, --dry-run/--expect, machine check, format-change gate
    #2670 P2-3b: obc data source refresh
    #2672 P2-3c: source copies under sources/ with licence and attribution
    #2673 P2-3d: publish from any machine
    #2680 P2-3e: planner reads only published files of other layers
    #2677 P2-4a: lock.json, heartbeat, takeover with unlock --from
    #2678 P2-4a2: detached runs, runs/<id>/{status.json, log}, runs --follow/--stop
    #2682 P2-4b: R2 writes (SigV4, conditional writes, key prefix for tests, bucket check), prod confirmation, and the AGENTS.md rule

    The acceptance tests run on dir targets. They cover:

    • two concurrent updates;
    • the takeover of an old lock;
    • a rollback swap;
    • a review mismatch;
    • no downloads for unchanged layers;
    • a machine-check refusal;
    • a format change without acknowledgement.

    The spec defines the release manifest, the bucket layout, the lock, the review ID and the runs.

    Known gaps:

    • Live R2 test: tests/r2_live.rs is opt-in (--ignored) and has not run yet. It needs the R2 credentials, and it writes only under a test/ prefix.
    • Old capture versions: a fresh machine cannot fetch an old capture version (for example a national terrain model) that the serving release read. It refuses with a Fix line that names the machine that has it. P6-2 removes this gap.
    • Landmark and peak content: this content needs the Wikimedia captures, and no target copies them. It is a named exemption until P6.
    • Dry run of an R2 target: it needs credentials, because the review ID holds the S3 ETag of release.json.

    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestinfraTooling, scaffold, CI, dev harness

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions