Skip to content

obc data v2 · P7 — Planet coverage #2655

Closed as duplicate
Closed as duplicate

Description

@timohueser

Part of #2648. Read the epic first: it holds the model and the decisions.

Goal

A target can cover the whole planet. A weekly refresh bakes only what changed.

Scope

  1. OSM by size. A region reads the smallest source that covers it. Small regions read Geofabrik extracts; planet reads osm-planet, kept current with osm-replication daily diffs (osmium apply-changes). Both sources already exist in data/sources.toml.
  2. Leaf extracts. Port the planet sharding of host/obc-bake/src/planet.rs into the internal OSM steps. A leaf extract must not carry the planet timestamp in its header. A leaf whose OSM content did not change keeps its digest, so its tiled layers are not baked again.
  3. Named partitions for whole layers. A whole layer may declare partitions; each partition is staled, baked and published on its own. This is only the hook. The partition rule for the router comes from the planet router design (Planet router: design how the routing package scales to the whole planet #2657).
  4. Bake machine. Measure one regional prod bake and record the machine requirements in the README of the bake host. Size the planet machine after Planet router: design how the routing package scales to the whole planet #2657.

Acceptance

  • After a one-week OSM refresh of a large region, only the leaves whose OSM content changed are baked again; the run reports how many.
  • A whole layer with two declared partitions stales and bakes one partition alone.
  • Focused tests; the suites of each crate you change. Report added and removed lines.

Depends on

P1 (#2649). A planet-scale planner/routing depends on #2657.

Activity

  1. timohueser commented on Oct 12, 2026

    @timohueser
    OwnerAuthor

    Progress on P7. These PRs are merged:

    PR Part Lines
    #2688 P7-B: binary leaf split in obc-osm (at most two extracts per Osmium run), --generator obc-data, Osmium in the Rust CI test job +284 / −81
    #2690 P7-C: a step whose key equals its output in the serving release is unchanged and is not baked, also on another machine; the run reports unchanged leaves; each child program's CPU and peak memory are measured on their own (wait4) +382 / −62

    Acceptance (a) now holds for a large Geofabrik region. After an OSM refresh, only the leaves whose OSM content changed bake again, and the run logs, for example, device/map: 3 of 40 leaves baked, 37 unchanged.

    Next:

    • P7-A: the named partitions hook for whole layers (acceptance b).
    • P7-D: the planet source (osm-planet plus daily osm-replication diffs). It starts after P6-3b.
    • P7-E: delete obc bake --all. It waits for the owner.
    • P7-F: the bake machine requirements. They come after the owner's measurement run.

    Generated by Claude Code

  2. timohueser commented on Oct 12, 2026

    @timohueser
    OwnerAuthor

    P7-A is merged: #2693 (+119 / −20), the named partitions hook.

    • A step of a Shape::Partitioned layer is named by its partition: the last part of its name. A step whose last part is not a declared partition is refused.
    • Each partition stales, bakes, is kept and publishes on its own, because the engine already works per step.
    • obc data --json gives partitions and stale_partitions per layer, and the review gives the partitions that bake.
    • a_partitioned_layer_stales_and_bakes_one_partition_alone covers acceptance (b): a new version of one partition's input stales only that partition, and only it bakes.

    Both acceptance items now have their tests (a: #2688 and #2690, b: #2693). Two scope items are still open:

    • the planet source (P7-D, after P6-3b);
    • the bake machine requirements (P7-F, after the owner's measurement run).

    So this issue stays open. obc bake --all (P7-E) waits for the owner's answer.


    Generated by Claude Code

  3. timohueser commented on Oct 13, 2026

    @timohueser
    OwnerAuthor

    P7-D (the planet source) is reviewed as draft #2702, head ee3c188. It is stacked on #2701 and merges after it.

    • Region kind planet (data/regions/planet.toml). It resolves to B = osm-planet and E = osm-replication with from=B.
    • One shared step, device/source/planet, runs osmium apply-changes … --generator obc-data. When E has no diff after B, the step links the planet. The step has no client file. Another machine builds it again from the R2 copies of the two fetches.
    • osm-planet refreshes every 28 days. The dead "before osm-planet" rule is deleted.
    • The planner refuses a planet region. This is the recommended answer to Q1, which the owner has not decided yet. An update now refuses before it fetches data when a product does not suit the settings, so a planet update does not download 95 GB first. That rule applies to every update.
    • The store-safe hard link is one function, engine::link, used by every step that links a store object.
    • The PR body lists the planet-scale risks to measure: apply-changes RAM, peak disk, step length, manifest and catalog size, GLO-30 downloads, the per-leaf manifest parse, and the diff copies of about 3.7 GB a week.

    P7-E (delete obc bake --all) waits for the owner's answer to Q6. P7-F waits for the owner's measurement run (Q7).


    Generated by Claude Code

  4. timohueser commented on Nov 4, 2026

    @timohueser
    OwnerAuthor

    This issue is merged into #3000 (world scale). The acceptance tests landed (#2688, #2690, #2693), and coverage is now Europe (#2921 A6). Draft PR #2702 and Q7 are noted in #3000. Q6 goes to #2656.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestinfraTooling, scaffold, CI, dev harness

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions