Skip to content

obc data v2 · P1 — Layers, versions and staleness #2649

Closed

Description

@timohueser

Part of #2648. Read the epic first: it holds the model and the decisions.

Goal

A layer is stale for exactly three reasons: its version changed, a source has a newer version to use, or an input changed. Code changes are not a reason. One binary runs everything.

Scope

  1. Layer declaration in the engine (host/obc-data/src/engine.rs): name, shape (tiled, whole, or whole with named partitions), format, revision, inputs (sources and layers), options, outputs, readers (for the format-change review in P2), steps. One operator layer can have several internal steps; the layer version applies to all of them.

  2. Layer key = SHA-256 of (layer, step, format, revision, input digests, options). Inputs enter by the digest of their bytes, as today, so a rebuild with the same bytes stops there.

  3. Delete code identity: host/obc-data/src/engine/code.rs, store/code/, producers/, the worker binding (host/obc-data/src/worker.rs), the launcher (host/obc-data-steps/src/launcher.rs), the obc-data-worker binary, the exit-code reload, and the Python import-list test in host/obc-data-steps/src/planner.rs. tools/justfile builds and runs one release binary.

  4. Format numbers from the readers: device/map uses obcm::VERSION (firmware/obc-formats/src/obcm.rs), planner/routing uses the router FORMAT (planner/router/src/package.rs), device/catalog uses CATALOG_SCHEMA_VERSION (host/obc-pack/src/catalog.rs). Other layers start at the format their spec states. Every revision starts at 1.

  5. Operator layers as in the epic table. Map today's steps onto them:

    • maps/source/*, maps/region-osm, maps/osm → internal OSM steps shared by the layers that read OSM;
    • maps/reference/*, maps/terrain/* → device/terrain;
    • maps/{coarse,mid,fine,network}/* → device/map;
    • maps/landmark-content, maps/peak-content, maps/landmarks/*, maps/peaks/* → device/landmarks;
    • maps/catalog → device/catalog;
    • planner/* steps → the planner layers; */grid steps belong to their layer.

    This also removes today's fault where the planner product emits maps/… steps that cli/build_cli.rs rejects.

  6. Staleness judge (rewrite engine/state.rs, engine/changes.rs): for a release and the current code, each layer is ok, off or stale with one cause and one short text:

    • format 19 → 20: readers need new data
    • revision 2 → 3: improved output
    • wiki 2026-06-01 → 2026-09-01 (cause 2)
    • follows device/map (cause 3)
  7. Naming a layer rebuilds it: --only L bakes L again even when it is not stale. Its followers bake again only when its bytes change.

  8. The TUI is deleted (cli/tui.rs, cli/tui/). It would need a rewrite against the new states. The CLI is the interface until P5.

Acceptance

  • Focused tests, one per cause: a version change, a newer source version, a changed input layer; plus "naming a layer rebuilds it" and "a producer refactor with no version change leaves every layer ok".
  • Changing obcm::VERSION makes device/map stale with a format cause and no other edit.
  • obc test -p obc-data, obc test -p obc-data-steps, clippy with -D warnings on both.
  • specs/obc-data.md: the key and staleness sections state the new rules; the code-identity sections are removed.
  • The PR reports added and removed lines.

Out of scope

Targets, releases and publishing (P2). Serving (P3).

Depends on

Nothing.

Activity

  1. timohueser commented on Oct 12, 2026

    @timohueser
    OwnerAuthor

    Scope item 8 is done in #2658 (merged). The obc data TUI is deleted, and a bare obc data prints the status. The diff is +67 / −5072 lines, including 37 crates that leave Cargo.lock with ratatui.

    Items 1–7 follow in the next P1 pull requests: layer versions, the new key, the deletion of code identity, and the staleness judge.


    Generated by Claude Code

  2. timohueser commented on Oct 12, 2026

    @timohueser
    OwnerAuthor

    #2659 is merged. It covers scope items 1–5 and deletes obc data dev, the portable adoption and tools/planner_local.py early; P3 replaces them with obc data serve. The diff is +1181 / −9560 lines.

    • Layers and keys. A static engine::Layer { name, shape, format, revision, readers } replaces code identity. Each step points to its layer. The key is SHA-256 of layer, step, format, revision, input digests and options.
    • Formats. They come from the reader constants: obct::SURFACE_VERSION, obcm::VERSION, the landmark and peak section versions, CATALOG_SCHEMA_VERSION and the router FORMAT. Every revision is 1.
    • Credits. Source credits from data/sources.toml go into step options, so a change of attribution makes the step stale.
    • Store. A store layout marker refuses old stores, with a Fix line that keeps the snapshots.

    The staleness judge and --only (items 6 and 7) follow in the next pull request.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestinfraTooling, scaffold, CI, dev harness

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions