Repository navigation
Description
Status: done
Part of #3141 (workstream 3 of #3000).
TL;DR
Goal: The planner release.json holds no file list, the download service reads one small catalog and the files of the requested cells, and the follower reads only the step indexes of the folders that it serves.
Problem: release.json lists every file of a release (1.9 MB today, about 230 MB at the target coverage, estimated), and offline/catalog.json holds a second copy of that list. The phone, the follower and the download service parse them whole.
Decision: See #3141, decisions 1 to 3. release.json format 2 names the step indexes in indexes. The new step planner/offline/grid writes offline/catalog.json format 4 and one offline/cells/<cell>.json per cell. planner/index goes to format 4. After a rollback to a release from before this change, the follower serves its maps but not its planner services.
Systems: planner-map-baking, data-engine, search, planner-tile-server
Contracts: specs/planner-release.md, specs/planner-offline.md, specs/obc-data.md
Size: L (a breaking change of the planner manifest, the offline catalog and the format of planner/index)
Plan
The formats.
- Planner
release.jsonformat 2 has the header of format 1 (region,name,bounds,osm_sha256,routing_package,profiles, the attributions,terrain_bounds,grid) and:indexes: for each grid step,planner/offline/gridincluded, sorted by step name,{"<step>": {"bytes", "sha256", "folders"}}. The file isindexes/<step>/index.jsonof the release.foldersis the sorted list of the first path parts of the files of the index, for example["offline", "routes", "routing"].files: the entries of the client files ofplanner/indexthat no step index lists:search/<region>.grid.json.
offline/catalog.jsonformat 4:format,zoom,release(the header, as today),routing_source,cells(the sorted IDs of the cells),shared(as today) andassets: a map from each file thatsharednames to its entry.offline/cells/9-X-Y.jsonformat 4, one for each cell:id,bounds,files,routing,packsandrouting_packs.files,packsandrouting_packsare maps from a path to its entry, as in a step index.filesholds the search databases and the route catalog file of the cell.routingis the descriptor of today with itsarchives, or absent.packsholds the map packs of the kinds of today'smap_blocksthatselectedtakes for the box of theboundsand thegeometry_boundsof the cell (terrain with its ring of tiles).routing_packsholdspages.binandpages.idxof each archive of the cell.- A selection takes the union of the packs of its cells. Today it takes the packs of one box around the selected cells. The new rule covers the area and the retained roads of each cell, as today, and can take fewer packs at the corners of the box. The TileJSON bounds and
terrain_boundsof a bundle stay as today: the box of the selected cells and of theirgeometry_bounds.
Steps.
tools/planner_grid_pack.pyrouting: addarchivesof the routing cell manifest to each cell of the index.- New
tools/planner_grid_offline.py, the stepplanner/offline/grid. It reads only theindex.jsonof each other grid step and writesoffline/catalog.json,offline/cells/<cell>.jsonand its ownindex.json, packed withpack_file. Move here the part ofcomposethat makescells,map_blocks,sharedand the offlinereleaseheader. host/obc-data-steps/src/planner.rs: declareplanner/offline/gridin the layerplanner/indexasplanner/fonts/gridis declared, withclientofobjectsandindex.json, and add it toGATHERERS. Its inputs are theindex.jsonof the other grid steps. Because its name ends with/grid,planner/indexandplanner/catalogread its index, and it is a named file. Raise the format ofPLANNER_INDEXto 4.tools/planner_grid_index.py:stepwritesrelease.jsonformat 2 fromcompose. It no longer writesoffline/catalog.json, and its localindex.jsonno longer holdsfiles.tools/planner_runtime.py:INDEX_FORMAT = 4.planner/search/installation.mjs:GRID = 4; the search grid file does not change otherwise.tools/planner_downloads.py: readoffline/catalog.jsonformat 4 at start.preparetakes the IDs ofcellsthat the requested bounds meet and reads their cell files. A missing cell file of a listed cell is an error.quotetakes the files, packs and routing packs from the cell files, and the assets fromsharedandassets. The ID of a selection adds the SHA-256 of each selected cell file; itsformatgoes from 5 to 6. Therelease.jsonandbundle.jsonof a bundle keep their formats.host/obc-data/src/serve/services.rs:runtimereads format 2. It reads each index ofindexeswhosefoldersmeetRUNTIME, checks its bytes and SHA-256, and takes the entries underRUNTIME, then the entries offiles. Format 1 keeps today's path.objectsgets these indexes asobjects/<sha256>in a first phase, afterrelease.jsonand before the runtime files, and returns them with the runtime files, soprunekeeps them.catalog.rs:verifyandpointerbuild the map of files from the indexes ofindexesand fromfiles. A previous release of format 1 gives its map fromfiles.tools/planner_verify.py: comparerelease.jsonwith the output ofcompose, check each index ofindexeswith its bytes and SHA-256, and compose the offline files again.planner/tiles/README.md: the release ID is the digest of the named files.- Specs:
specs/planner-release.mdgets manifest format 2.specs/planner-offline.mdgetsoffline/catalog.jsonformat 4, the cell files and the selection by union.specs/obc-data.mdgets the stepplanner/offline/grid. - Tests:
tools/tests/test_planner_grid_index.py:release.jsonhasindexesand no file of a step index.tools/tests/test_planner_grid_offline.py(new): a cell file lists the packs of its geometry and the routing packs of its archives.tools/tests/test_planner_downloads.py: a request of two cells gives the union of their packs, a request reads only the cell files of its bounds, and a missing file of a listed cell fails.services.rs: the follower installs the runtime files of format 1 and format 2.
Done when
- The pull request passes the suites that
obc test affected --base origin/developselects, andobc exec cargo clippy -p obc-data -p obc-data-steps --all-targets -- -D warnings. - The measurements and checks of "Done when" in Manifests: small release manifests, per-cell offline catalogs and file lists as objects #3141 run on the one re-bake, after A and B merge.
Generated by Claude Code
Activity