Skip to content

Manifests A: planner release.json without file list, offline catalog per cell #3142

Closed

Description

@timohueser

Status: done
Part of #3141 (workstream 3 of #3000).

TL;DR

Goal: The planner release.json holds no file list, the download service reads one small catalog and the files of the requested cells, and the follower reads only the step indexes of the folders that it serves.
Problem: release.json lists every file of a release (1.9 MB today, about 230 MB at the target coverage, estimated), and offline/catalog.json holds a second copy of that list. The phone, the follower and the download service parse them whole.
Decision: See #3141, decisions 1 to 3. release.json format 2 names the step indexes in indexes. The new step planner/offline/grid writes offline/catalog.json format 4 and one offline/cells/<cell>.json per cell. planner/index goes to format 4. After a rollback to a release from before this change, the follower serves its maps but not its planner services.
Systems: planner-map-baking, data-engine, search, planner-tile-server
Contracts: specs/planner-release.md, specs/planner-offline.md, specs/obc-data.md
Size: L (a breaking change of the planner manifest, the offline catalog and the format of planner/index)

Plan

The formats.

  • Planner release.json format 2 has the header of format 1 (region, name, bounds, osm_sha256, routing_package, profiles, the attributions, terrain_bounds, grid) and:
    • indexes: for each grid step, planner/offline/grid included, sorted by step name, {"<step>": {"bytes", "sha256", "folders"}}. The file is indexes/<step>/index.json of the release. folders is the sorted list of the first path parts of the files of the index, for example ["offline", "routes", "routing"].
    • files: the entries of the client files of planner/index that no step index lists: search/<region>.grid.json.
  • offline/catalog.json format 4: format, zoom, release (the header, as today), routing_source, cells (the sorted IDs of the cells), shared (as today) and assets: a map from each file that shared names to its entry.
  • offline/cells/9-X-Y.json format 4, one for each cell: id, bounds, files, routing, packs and routing_packs. files, packs and routing_packs are maps from a path to its entry, as in a step index. files holds the search databases and the route catalog file of the cell. routing is the descriptor of today with its archives, or absent. packs holds the map packs of the kinds of today's map_blocks that selected takes for the box of the bounds and the geometry_bounds of the cell (terrain with its ring of tiles). routing_packs holds pages.bin and pages.idx of each archive of the cell.
  • A selection takes the union of the packs of its cells. Today it takes the packs of one box around the selected cells. The new rule covers the area and the retained roads of each cell, as today, and can take fewer packs at the corners of the box. The TileJSON bounds and terrain_bounds of a bundle stay as today: the box of the selected cells and of their geometry_bounds.

Steps.

  1. tools/planner_grid_pack.py routing: add archives of the routing cell manifest to each cell of the index.
  2. New tools/planner_grid_offline.py, the step planner/offline/grid. It reads only the index.json of each other grid step and writes offline/catalog.json, offline/cells/<cell>.json and its own index.json, packed with pack_file. Move here the part of compose that makes cells, map_blocks, shared and the offline release header.
  3. host/obc-data-steps/src/planner.rs: declare planner/offline/grid in the layer planner/index as planner/fonts/grid is declared, with client of objects and index.json, and add it to GATHERERS. Its inputs are the index.json of the other grid steps. Because its name ends with /grid, planner/index and planner/catalog read its index, and it is a named file. Raise the format of PLANNER_INDEX to 4.
  4. tools/planner_grid_index.py: step writes release.json format 2 from compose. It no longer writes offline/catalog.json, and its local index.json no longer holds files. tools/planner_runtime.py: INDEX_FORMAT = 4. planner/search/installation.mjs: GRID = 4; the search grid file does not change otherwise.
  5. tools/planner_downloads.py: read offline/catalog.json format 4 at start. prepare takes the IDs of cells that the requested bounds meet and reads their cell files. A missing cell file of a listed cell is an error. quote takes the files, packs and routing packs from the cell files, and the assets from shared and assets. The ID of a selection adds the SHA-256 of each selected cell file; its format goes from 5 to 6. The release.json and bundle.json of a bundle keep their formats.
  6. host/obc-data/src/serve/services.rs: runtime reads format 2. It reads each index of indexes whose folders meet RUNTIME, checks its bytes and SHA-256, and takes the entries under RUNTIME, then the entries of files. Format 1 keeps today's path. objects gets these indexes as objects/<sha256> in a first phase, after release.json and before the runtime files, and returns them with the runtime files, so prune keeps them.
  7. catalog.rs: verify and pointer build the map of files from the indexes of indexes and from files. A previous release of format 1 gives its map from files.
  8. tools/planner_verify.py: compare release.json with the output of compose, check each index of indexes with its bytes and SHA-256, and compose the offline files again.
  9. planner/tiles/README.md: the release ID is the digest of the named files.
  10. Specs: specs/planner-release.md gets manifest format 2. specs/planner-offline.md gets offline/catalog.json format 4, the cell files and the selection by union. specs/obc-data.md gets the step planner/offline/grid.
  11. Tests: tools/tests/test_planner_grid_index.py: release.json has indexes and no file of a step index. tools/tests/test_planner_grid_offline.py (new): a cell file lists the packs of its geometry and the routing packs of its archives. tools/tests/test_planner_downloads.py: a request of two cells gives the union of their packs, a request reads only the cell files of its bounds, and a missing file of a listed cell fails. services.rs: the follower installs the runtime files of format 1 and format 2.

Done when


Generated by Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions