Skip to content

Larger map packs A: bake the pack rule and its block indexes #3075

Closed

Description

@timohueser

Status: done
Part of #3069. The pack rule, its reasons and the formats are in #3069.
Depends on #3073 and #3074: they merge first, so that each bake from develop has readers for its formats.

TL;DR

Goal: The bake cuts every map layer into packs by the pack rule of #3069: base levels 0, 7 and 9, a 15 MiB limit, and the pointers of a base square in one block index.
Systems: planner-map-baking, obc-data
Contracts: specs/planner-release.md, specs/planner-offline.md, specs/obc-data.md, specs/trailview-tiles.md
Size: L (a breaking change to the pack layout)

Plan

  1. tools/planner_grid_maps.py: replace MAP_ZOOM with LEVELS = (0, 7, 9) and PACK_BYTES = 15 * 1024 * 1024. In map_tiles, stage each tile under the pack of its base level. Write the PMTiles file of each staging file. When a file is larger than PACK_BYTES, delete it, move each tile deeper than the pack's zoom to the staging file of its child square one zoom deeper, write the pack again from its one tile at its own zoom (no file when there is none), and repeat for the children. A pack that holds one tile and is larger than PACK_BYTES fails the step. The z13 terrain packs stay under 13/, with base level 9. index.json loses map_zoom.
  2. tools/planner_grid_index.py: write release grid format 3, {format: 3, zoom: 9}, in release.json and in the catalog.
  3. tools/planner_runtime.py public_metadata: write public grid format 4, {"format": 4}, and set GRID_FORMAT = 4. Put the pointer of a map pack under maps/tiles/ into the block index of its base square: B = max(base - 2, 0), where base is the base level of the pack's zoom. Files under routes/tiles/ keep B = max(Z - 4, 0).
  4. tools/planner_verify.py: accept release grid format 3, {format: 3, zoom: 9}, and refuse a map pack whose bytes is larger than PACK_BYTES.
  5. host/obc-data-steps/src/planner/catalog.rs verify: accept {"format": 3, "zoom": 9}.
  6. tools/planner_downloads.py: write offline block format 3, {"format": 3, "zoom": 9}. Selection keeps its rule.
  7. host/obc-data/src/cli/deploy_cli.rs: GRID = 4.
  8. Specs: in specs/planner-release.md, specs/planner-offline.md, specs/obc-data.md and specs/trailview-tiles.md, replace "the ancestor at min(tile_zoom, 11)" and "per zoom 11 parent" with the levels, the 15 MiB limit and its reason (a tile at the end of a cold 15 MiB pack waits about 1 s), the block index of the base square, the reader rule, release grid format 3 and public grid format 4.
  9. Tests: tools/tests/test_planner_grid_maps.py: a square over the limit gives a parent pack with the tiles of its zoom and child packs with the deeper tiles; a z13 terrain square over the limit gives no parent file; a square under the limit gives one pack. tools/tests/test_planner_runtime.py: the pointers of a split square are in the block index of its base square. tools/tests/test_planner_downloads.py: the new offline block.

Done when

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions