Repository navigation
Closed
Labels
enhancementNew feature or requestNew feature or requestinfraTooling, scaffold, CI, dev harnessTooling, scaffold, CI, dev harness
Description
Part of #2648. Read the epic first: it holds the model and the decisions.
Goal
The two rare, long jobs become ordinary sources with a version. A bake reads a pinned version from R2 and makes no requests to Wikimedia or to national terrain portals.
Scope
- Prepared source, generic.
obc data source prepare ID [options]is a normal run with review, machine check, lock, heartbeat and resume.- It publishes to
sources/<id>/<version>/in the prod bucket and writes its manifest last. - A new version makes its readers stale in each target that uses an older one (staleness cause 2).
- A version that no kept prod release uses is deleted from R2; a
localtarget keeps its cached copy.
wiki.- Port the dump processing from open PR Add manual Wikimedia snapshot preparation for landmark and peak bakes #2647 (branch
codex/wikimedia-snapshot-content:tools/wikimedia_snapshot.py,tools/wikimedia_snapshot_sql.py,host/obc-data/src/content.rs,content/preparation.rs). - Inputs: the Wikidata JSON dump, Wikipedia Enterprise HTML (
OBC_WIKIMEDIA_ENTERPRISE_TOKEN), the SQL tables, and Commons photos only for selected subjects (keep the rate limits oftools/landmark_capture.py). - Subjects: every OSM object with a wikidata tag on the planet, so one version serves every target.
- Do not port its TUI tab or its
contentcommand tree. - After the port, the owner closes PR Add manual Wikimedia snapshot preparation for landmark and peak bakes #2647.
- Port the dump processing from open PR Add manual Wikimedia snapshot preparation for landmark and peak bakes #2647 (branch
terrain-reference/<country>.- Port the adapters of
host/obc-dem/reference/ingest.py(ingest/sources/*.py). - Stream one pooled tile at a time (obc data: fix the first-bake blockers and take the simple path #2612 decision 1).
- Publish the 2^16 µdeg pooled tiles and
index.jsonper country.device/terrainfetches only the tiles that its leaves read. - Credentials:
OBC_REFERENCE_*. - The 24
dtm-*entries indata/sources.tomlbecome the inputs of this prepared source.
- Port the adapters of
- Replace the per-request Wikimedia capture:
host/obc-data/src/fetch/wikimedia.rs,tools/wikimedia_acquire.py,tools/wikimedia_adopt.py, and the plan-time content discovery inhost/obc-data-steps/src/maps/shared_content.rs. device/landmarksandplanner/searchreadwiki.device/terrainreadsterrain-reference/*.- R2
reference/v1/stays until P8 shows that equivalent data is live.
Acceptance
- Prepare
wikifor a small subject set andterrain-reference/chon the bake VPS. Then bakelocalfrom those versions with no network access to Wikimedia or the national portals. - A new
wikiversion makesdevice/landmarksandplanner/searchstale in prod and not in a target that already uses it. - Run focused tests, plus the suites of each crate and the Python tests you change. Report added and removed lines.
Depends on
P2.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestinfraTooling, scaffold, CI, dev harnessTooling, scaffold, CI, dev harness
Activity
timohueser commentedon Oct 12, 2026
P6-0 is merged in #2664. It ports the Wikimedia dump processing from #2647 into
tools/wiki/as Python, with tests. Nothing calls it yet. The diff is +1534 / −189 lines.python -m tools.wiki subjects|import|articles|commons|shardsreads local inputs thatinputs.jsonnames. Each input's format comes from its first bytes, so store objects need no suffix. The Rust producer downloads the inputs in P6-3a.landmark_capture.Capturewith its rate limits unchanged.content/<kind>/<sha>.jsonandcontent/assets/<sha>(500 px thumbnails). A subject goes into every 2^20 µdeg cell its box touches, up to 16 cells; other subjects go tocommon.Next: P6-1, the generic prepared-source mechanism, after P2-2, P2-3 and P2-4. PR #2647 stays open until P6-3a lands.
Generated by Claude Code
timohueser commentedon Oct 12, 2026
P6-1 is merged: #2685 (+2553 / −157), the prepared-source mechanism.
obc data source prepare ID [--box|--region] [--to TARGET] [--dry-run] [--json] [--yes --expect ID]reuses the update review, the machine check, the source lock (sources/<id>/lock.json) with heartbeat, and the detached run.<date>/<sha256>) and the fetch all use it. Text output showsdate (sha12); JSON keeps the full pin.versions.jsongets the entry with a conditional write. Each entry hasformat.versions.json, and from prod's for a non-prod target. With no version of that format, the layer is blocked with "Runobc data source prepare ID".reference/v1/,versions.jsonandlock.jsonare never deleted.Next: P6-2 (
terrain-reference/<key>), then P6-3a (thewikiproducer) and P6-3b (readers and deletions).Generated by Claude Code
timohueser commentedon Oct 12, 2026
P6-2 is merged: #2691 (+849 / −438), the
terrain-reference/<key>prepared sources.dtm-<key>rows gets a derivedterrain-reference/<key>row. It copies the licence, attribution, extent and areas, hasredistribute = trueand a date version, and has no new row indata/sources.toml.ingest.py fetchper tile, which writes the tile and its pixel digest. It uploads in batches of 64, and a resume is exact. It writesindex.json, then the manifest. It never writesreference/v1/. Credentials are needed only on the prepare machine.device/terrainbakes those leaves from GLO-30 only. The review andobc dataname the missing source with the prepare command. A later version stales only the leaves that read it.capture::dtm, thedtmfetch paths and the per-tile fetches in bakes.ingest.py publish/mirror/--per-tilestay forobc-bakeuntil P8.Next: P6-3a, the
wikiproducer. The first realterrain-referenceversion waits for the owner's choice of country and machine (CH recommended).Generated by Claude Code
timohueser commentedon Oct 12, 2026
P6-3a is complete. The
wikiproducer is merged in two PRs:OBC_WIKI_DUMPS, pins them instate/pin.json, and putspublish_textin the review id.index.jsonasphoto_compile_failures. The run stops on an I/O error, or when more than half of a chunk fails.publish_text = false, the shards carry a "missing" article stub in place of the text.A consequence for the owner's decision on text: the landmark compiler needs article text. A version with
publish_text = falsetherefore gives devices no landmarks, only peaks and their photos. Thewikirow ships withpublish_text = true.Next: P6-3b (the layers read a prepared
wikiversion) as a draft PR. It merges after the owner makes the first prodwikiversion on the machine that holds the dumps.Generated by Claude Code
timohueser commentedon Oct 13, 2026
P6-3b is ready as draft #2701, at head c056967. The review is done and CI is green.
wikiversion:common.tar.zstplus the shards of the cells that the region polygon touches.It stays a draft until these two steps are done:
wikiversion on the machine that holds the dumps.viewcopies only the facts that the chosen subjects reach.Generated by Claude Code
timohueser commentedon Nov 4, 2026
This issue is closed because the work is done: #2664, #2685, #2691, #2695, #2699 and #2701 (P6-3b) merged. A
wikiversion exists. #2921 has since moved the prepared-source home from R2 to the archive (#2930), andterrain-referenceis now theterrainprepared source. The landmark re-pin from #2026 goes to #2656.Generated by Claude Code