Skip to content

obc data v2 · P6 — Prepared sources: wiki and terrain-reference #2654

Closed

Description

@timohueser

Part of #2648. Read the epic first: it holds the model and the decisions.

Goal

The two rare, long jobs become ordinary sources with a version. A bake reads a pinned version from R2 and makes no requests to Wikimedia or to national terrain portals.

Scope

  1. Prepared source, generic.
    • obc data source prepare ID [options] is a normal run with review, machine check, lock, heartbeat and resume.
    • It publishes to sources/<id>/<version>/ in the prod bucket and writes its manifest last.
    • A new version makes its readers stale in each target that uses an older one (staleness cause 2).
    • A version that no kept prod release uses is deleted from R2; a local target keeps its cached copy.
  2. wiki.
  3. terrain-reference/<country>.
    • Port the adapters of host/obc-dem/reference/ingest.py (ingest/sources/*.py).
    • Stream one pooled tile at a time (obc data: fix the first-bake blockers and take the simple path #2612 decision 1).
    • Publish the 2^16 µdeg pooled tiles and index.json per country. device/terrain fetches only the tiles that its leaves read.
    • Credentials: OBC_REFERENCE_*.
    • The 24 dtm-* entries in data/sources.toml become the inputs of this prepared source.
  4. Replace the per-request Wikimedia capture: host/obc-data/src/fetch/wikimedia.rs, tools/wikimedia_acquire.py, tools/wikimedia_adopt.py, and the plan-time content discovery in host/obc-data-steps/src/maps/shared_content.rs.
  5. device/landmarks and planner/search read wiki. device/terrain reads terrain-reference/*.
  6. R2 reference/v1/ stays until P8 shows that equivalent data is live.

Acceptance

  • Prepare wiki for a small subject set and terrain-reference/ch on the bake VPS. Then bake local from those versions with no network access to Wikimedia or the national portals.
  • A new wiki version makes device/landmarks and planner/search stale in prod and not in a target that already uses it.
  • Run focused tests, plus the suites of each crate and the Python tests you change. Report added and removed lines.

Depends on

P2.

Activity

  1. timohueser commented on Oct 12, 2026

    @timohueser
    OwnerAuthor

    P6-0 is merged in #2664. It ports the Wikimedia dump processing from #2647 into tools/wiki/ as Python, with tests. Nothing calls it yet. The diff is +1534 / −189 lines.

    • Command. python -m tools.wiki subjects|import|articles|commons|shards reads local inputs that inputs.json names. Each input's format comes from its first bytes, so store objects need no suffix. The Rust producer downloads the inputs in P6-3a.
    • Import. A two-pass import reads the Wikidata JSON dump and the page and redirect SQL tables. A bad redirect chain becomes a failure row for its key and does not stop the import.
    • Articles. They come from Enterprise HTML, with the revision proofs from Add manual Wikimedia snapshot preparation for landmark and peak bakes #2647.
    • Commons. Requests go through landmark_capture.Capture with its rate limits unchanged.
    • Shards. Each shard holds content/<kind>/<sha>.json and content/assets/<sha> (500 px thumbnails). A subject goes into every 2^20 µdeg cell its box touches, up to 16 cells; other subjects go to common.

    Next: P6-1, the generic prepared-source mechanism, after P2-2, P2-3 and P2-4. PR #2647 stays open until P6-3a lands.


    Generated by Claude Code

  2. timohueser commented on Oct 12, 2026

    @timohueser
    OwnerAuthor

    P6-1 is merged: #2685 (+2553 / −157), the prepared-source mechanism.

    • obc data source prepare ID [--box|--region] [--to TARGET] [--dry-run] [--json] [--yes --expect ID] reuses the update review, the machine check, the source lock (sources/<id>/lock.json) with heartbeat, and the detached run.
    • A version is named by its UTC start date. Its identity is the SHA-256 of its manifest: the store record, the release pin (<date>/<sha256>) and the fetch all use it. Text output shows date (sha12); JSON keeps the full pin.
    • Each file uploads when it is finished. Before the manifest exists, nobody reads the version, so a file may be replaced. The manifest is written last with put-Absent, after a check that every journaled file is on the target. Then versions.json gets the entry with a conditional write. Each entry has format.
    • A resume keeps the version date. A stopped job whose version another machine completed never runs its producer again. If the manifest on the target is the one this job wrote, the same command only lists it.
    • Readers pick the newest version of their format from the target's own versions.json, and from prod's for a non-prod target. With no version of that format, the layer is blocked with "Run obc data source prepare ID".
    • Collection deletes a complete version only when no kept release reads it and a kept release reads a newer one. A new version deletes older incomplete folders under the source lock. reference/v1/, versions.json and lock.json are never deleted.

    Next: P6-2 (terrain-reference/<key>), then P6-3a (the wiki producer) and P6-3b (readers and deletions).


    Generated by Claude Code

  3. timohueser commented on Oct 12, 2026

    @timohueser
    OwnerAuthor

    P6-2 is merged: #2691 (+849 / −438), the terrain-reference/<key> prepared sources.

    • Derived rows. Each of the 25 dtm-<key> rows gets a derived terrain-reference/<key> row. It copies the licence, attribution, extent and areas, has redistribute = true and a date version, and has no new row in data/sources.toml.
    • Producer. The tile list is the model extent cut by its area polygons. The producer runs ingest.py fetch per tile, which writes the tile and its pixel digest. It uploads in batches of 64, and a resume is exact. It writes index.json, then the manifest. It never writes reference/v1/. Credentials are needed only on the prepare machine.
    • Reader. A leaf reads only the manifest paths in its tile window, per pinned version. A fresh machine plans a region with a national model from the manifest alone; this closes the gap noted on obc data v2 · P2 — Targets, releases, update, lock and rollback #2650.
    • No version yet. For a national model without a prepared version, device/terrain bakes those leaves from GLO-30 only. The review and obc data name the missing source with the prepare command. A later version stales only the leaves that read it.
    • Deleted. capture::dtm, the dtm fetch paths and the per-tile fetches in bakes. ingest.py publish/mirror/--per-tile stay for obc-bake until P8.

    Next: P6-3a, the wiki producer. The first real terrain-reference version waits for the owner's choice of country and machine (CH recommended).


    Generated by Claude Code

  4. timohueser commented on Oct 12, 2026

    @timohueser
    OwnerAuthor

    P6-3a is complete. The wiki producer is merged in two PRs:

    A consequence for the owner's decision on text: the landmark compiler needs article text. A version with publish_text = false therefore gives devices no landmarks, only peaks and their photos. The wiki row ships with publish_text = true.

    Next: P6-3b (the layers read a prepared wiki version) as a draft PR. It merges after the owner makes the first prod wiki version on the machine that holds the dumps.


    Generated by Claude Code

  5. timohueser commented on Oct 13, 2026

    @timohueser
    OwnerAuthor

    P6-3b is ready as draft #2701, at head c056967. The review is done and CI is green.

    • The landmark and peak content layers read the prepared wiki version: common.tar.zst plus the shards of the cells that the region polygon touches.
    • A fact that the version lacks is counted, and the run log names the counts.
    • In the producer, an absent fact is a failed compile, so the stop rule of feat(obc-data): wiki photos from Commons for the subjects of the compilers (P6-3a-2) #2699 still applies.
    • The live Wikimedia fetch path is deleted, and the content layers publish no bundles.

    It stays a draft until these two steps are done:

    1. The owner makes the first prod wiki version on the machine that holds the dumps.
    2. Someone measures the memory, disk and time of one region content step on that version. If they are large, view copies only the facts that the chosen subjects reach.

    Generated by Claude Code

  6. timohueser commented on Nov 4, 2026

    @timohueser
    OwnerAuthor

    This issue is closed because the work is done: #2664, #2685, #2691, #2695, #2699 and #2701 (P6-3b) merged. A wiki version exists. #2921 has since moved the prepared-source home from R2 to the archive (#2930), and terrain-reference is now the terrain prepared source. The landmark re-pin from #2026 goes to #2656.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestinfraTooling, scaffold, CI, dev harness

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions