Repository navigation
Description
Status: done
Part of workstream 3 of #3000.
TL;DR
Goal: No manifest of a release grows with the whole coverage. Each reader reads only the lists of the parts that it uses. The phone reads a release.json of about 10 KB. The download service reads the cells of a request. The engine manifest of an update grows with the number of steps, not with the number of files.
Problem: Four documents list every file of a release, and each reader parses them whole.
- The planner
release.jsonis 1.9 MB with 7,215 files today, and about 230 MB with larger packs (estimated). The phone downloads it for two fields. offline/catalog.jsonis 2.2 MB. 2.0 MB of it is a second copy of thefilesofrelease.json.- The engine manifest
releases/<id>.jsonofobc datais 6.5 MB today. It lists 12,174 output files (2.0 MB), the source files (3.3 MB, of which 6,390 MODIS files) and 1,804 named files (0.4 MB). The follower, the review, rollback and collection parse it whole. At about 160 B per output file, it grows to 150 MB or more at the target coverage (estimated). - Collection lists all keys in one
rclone lsjsondocument and deletes one key per request.
Decision:
- Planner
release.json, format 2. It losesfilesand getsindexes: the path, bytes, SHA-256 and top folders of theindex.jsonof each grid step. These step indexes already hold the entries of their files, and they are already named files, so the release ID already covers them.fileskeeps only the client files that no step index lists:search/<region>.grid.json. The header (profiles,routing_package, the attributions) stays, so the phone and the web planner do not change. - Offline catalog per cell. A new step
planner/offline/gridreads the indexes of the other grid steps. It writes a smalloffline/catalog.json(the release header, the zoom, the routing source, the cell IDs, and the shared assets with their entries) and oneoffline/cells/9-X-Y.jsonfor each cell: its bounds, its files, its routing descriptor, and the map packs that it selects, each with its entry. The download service reads the catalog at start, and the cell files of a request. - The follower reads
release.json, then only the step indexes whose folders it serves (routing/,search/,offline/). It reads planner manifest format 1 and format 2. The planner services read only the new format ofplanner/index, so after a rollback to a release from before this change, the follower serves its maps but not its routing, search and downloads. This is the rule for every format change of a layer that a service reads. - Engine manifest, format 2. Each list of files goes to an object of its own: the
filesand theinputsof each output, thefilesof each source, andnamed. The manifest keeps the size and the SHA-256 of each list. The release ID is still the SHA-256 of the manifest, so it proves every list. - Collection deletes up to 1,000 keys per request.
routing/blocks.json, and the copy of its graph in the routing grid index (3.5 MB today), stay with workstream 4.
Rejected:
- Move
profilesandrouting_packageintoplanner/catalog.json(the direction of EPIC — Scale readiness: 100 concurrent users, long routes, from one region to the world #3000):release.jsonbecomes about 10 KB, so the phone reads it as today, with no new app release. - Omit the transport entry where it repeats the file (6,867 of 7,215 entries): about 45% less per list, but each reader now reads only a part of the lists, and the change touches each reader of an entry.
- The download service finds the packs in the indexes of the map layers: it parses the whole index of each layer for each job.
- A streaming JSON parser for the large documents: each reader needs a new parser, and the documents still upload and download whole.
- Move only the output files of the engine manifest, or leave it to workstream 7: the source files are the largest part today. One rule for all the lists keeps the manifest small.
- The planner services read
planner/indexformat 3 and format 4, so that a rollback keeps its services: each service then reports a list of formats, and the download service keeps the reader of today. Every other format change of a service layer already stops the services of an older release. - Collection lists the keys line by line (
rclone lsf) instead of onerclone lsjsondocument: the keys and the set of kept keys stay in memory anyway, so it saves little. The re-bake measures the memory of a collection.
Systems: planner-map-baking, data-engine, search (GRID in planner/search/installation.mjs), planner-tile-server (its README says that the release ID is the SHA-256 of release.json, which is wrong)
Contracts: specs/planner-release.md, specs/planner-offline.md, specs/obc-data.md
Size: L (a breaking change of two stored manifest formats and of the format of planner/index)
Words
- Target: the bucket that a release goes to: R2 (
prod) or a local folder. Store: the local folder ofobc dataon a machine, with the objects by SHA-256 and a receipt per step. - Step: one build job of
obc data. A grid step (planner/<kind>/grid) packs the files of one kind for the planner. Its step index (index.json) maps each logical path to its entry:{bytes, sha256, transport: {bytes, sha256, encoding}}. The transport is the stored bytes: the file itself (identity) or its gzip. - Named files: the files that a client finds by path, at
releases/<id>/<path>of the target: the plannerrelease.json,public/and the step indexes. Each other file is atobjects/<sha256>. - Engine manifest:
releases/<id>.json, the manifest ofobc datathat lists the outputs of each step, the sources and the named files of a release. Plannerrelease.json: the manifest that the planner clients read. - Follower:
obc data serveon the VPS. It reads the target every 30 seconds, downloads the files of the serving release that the planner services need, and runs those services: routing, search and downloads. - Collection: after an update,
obc datadeletes the keys of the target that no kept release names (the serving release and the rollback).
Facts
Measured on the production release bf0927b0cd57 (Central Switzerland), read from maps.openbikecomputer.com:
| Document | Size | Content |
|---|---|---|
Planner release.json |
1.9 MB | 7,215 files. 6,867 have a transport that is equal to the file |
offline/catalog.json |
2.2 MB | files (2.0 MB) is equal to files of release.json. Also cells (9 cells, 63 KB, of which about 4 KB per cell is the routing adjacency), map_blocks (487 packs, 82 KB) and shared (1,066 aliases, 105 KB) |
Engine manifest releases/313857d5….json |
6.5 MB | sources 3.3 MB (modis-snow alone has 6,390 files, 2.7 MB). outputs 2.9 MB: files 2.0 MB for 12,174 files, inputs 0.8 MB. named 0.4 MB, 1,804 files |
Step indexes indexes/<step>/index.json |
66 files | Basemap 31 KB, assets 286 KB. The routing grid index is 3.5 MB, because it holds a copy of the graph of routing/blocks.json |
Writers
composeandstepoftools/planner_grid_index.pymerge thefilesof theindex.jsonof each grid step into one map.stepwrites that map three times: intorelease.json(line 132), intooffline/catalog.json(line 128) and into the localindex.jsonofplanner/index(line 138).pack_fileoftools/planner_offline.pymakes each entry{bytes, sha256, transport}. It stores.pmtiles,.bin,.webpand.pngfiles without compression (identity).- The planner release ID is the digest of the named files:
release.json,public/and eachindexes/<step>/index.json(namedinhost/obc-data-steps/src/planner/catalog.rs). So each step index is already a file list that the ID covers. routingoftools/planner_grid_pack.pywrites the routing grid index. Itscellshold the routing descriptor of each cell:id,bounds,geometry_bounds,adjacency,manifestandsha256. It reads each routing cell manifest, and that manifest names the routing packs of the cell (archives).- The engine manifest ID is the SHA-256 of
Release::canonical(host/obc-data/src/release.rs).FORMATis 1. A follower reads its own format and the one before;tests/manifests/format-1.jsonholds a manifest of format 1.
Readers of the planner release.json
- The phone:
PlannerService.manifestinPlannerService.swiftdownloads the whole file and decodes onlyrouting_packageandprofiles. Other keys are ignored. - The offline bundle on the phone (
OfflineMap.swift,OfflinePlanner.swift) reads therelease.jsonthat the download service writes for each selection, not the file of the release. - The web planner uses the manifest URL only as the base of
public/(terrain.ts).release.tsreads onlyactiveof the catalog. - The follower:
runtimeinhost/obc-data/src/serve/services.rsparses the whole document and keeps the entries underRUNTIME(routing/,search/,offline/). verifyofcatalog.rsreads each entry and compares it with the previous release.pointerofcatalog.rsreadsfilesonly to see whether a file exists.verifyoftools/planner_verify.pyrequires the document to be equal to the output ofcompose, and comparesoffline/catalog.json.
Readers of offline/catalog.json
Downloadsintools/planner_downloads.pyparses the whole file at start.preparescans allcells.quotemakes one geometry box from the selected cells, scans allmap_blockswithselectedoftools/planner_map_archive.py(terrain adds a ring of tiles), and takes each entry fromfiles. The routing packs of a selection are thearchivesof the merged routing cell manifests.- No iOS or Rust code reads it.
tools/planner_verify.pycomposes it again.
Readers of the engine manifest
- The follower parses the whole manifest (
servedinhost/obc-data/src/serve.rs) and finds the plannerrelease.jsoninnamed. - The review, rollback and overview use
Release::files(). - Collection (
host/obc-data/src/collection.rs) builds the set of keys to keep fromRelease::files()of the kept releases. It lists the target withrclone lsjson --recursive(lsjsoninhost/obc-data/src/r2.rs) into one document in memory. It deletes one key per request (target.remove) and checks the lock every 100 deletes (BATCH).Bucket::deleteofr2.rsalready deletes many keys in one rclone call.
Other facts
- Each planner service reads one format of each layer, and the follower starts the services of a release only when the formats match (
readableinservices.rs).INDEX_FORMAToftools/planner_runtime.py(3) is the format of the layerplanner/index. The download service reports it, and the search service checks it asGRIDinplanner/search/installation.mjs, becausesearch/<region>.grid.jsonis a file of that layer. planner/catalogcomputes the release ID from the inputs that it reads:planner/indexand theindex.jsonof each step whose name ends with/grid(indexesinhost/obc-data-steps/src/planner.rs). So a new step namedplanner/offline/gridjoins the ID, the named files and the inputs ofplanner/indexwith no other change.pruneinservices.rskeeps in the store only whatobjectsreturns for the served releases.- Collection deletes the files of a dropped release first and its manifest last, so that a collection that stops can resume from the manifests that remain. On R2 it lists only
objects/andreleases/. - No reader uses a streaming JSON parser.
MAX_MANIFEST_BYTES(128 MiB) limits only the routing manifest. planner/tiles/README.mdline 56 says that the release ID is the SHA-256 ofrelease.json. The ID is the digest of the named files.
Plan
This is an epic with two sub-issues. Each sub-issue is one pull request.
- Manifests A: planner release.json without file list, offline catalog per cell #3142 A. Planner manifests:
release.jsonformat 2, the offline catalog per cell, the download service and the follower - Manifests B: engine manifest file lists as objects, collection in batches #3143 B. Engine manifest format 2 and collection in batches
B depends on A: both raise what the follower reads, and B raises the engine manifest FORMAT once for both. Both merge before the one re-bake of workstreams 1, 2 and 3 of #3000. No release is published between the merge of A and the deploy of the follower of B. The follower says its manifest format at /planner-api/version, and the review of an update of prod names a follower that cannot read the release, so a release of the new formats does not reach a follower of today.
Verification
The code of A and B is in develop (#3146, #3147). The verification steps on real data are in a comment on #3000, with those of workstream 1, because they run in the one re-bake of workstreams 1, 2 and 3.
A. Planner manifests (planner-map-baking, data-engine, search, planner-tile-server)
The formats.
- Planner
release.jsonformat 2 has the header of format 1 (region,name,bounds,osm_sha256,routing_package,profiles, the attributions,terrain_bounds,grid) and:indexes: for each grid step,planner/offline/gridincluded, sorted by step name,{"<step>": {"bytes", "sha256", "folders"}}. The file isindexes/<step>/index.jsonof the release.foldersis the sorted list of the first path parts of the files of the index, for example["offline", "routes", "routing"].files: the entries of the client files ofplanner/indexthat no step index lists:search/<region>.grid.json.
offline/catalog.jsonformat 4:format,zoom,release(the header, as today),routing_source,cells(the sorted IDs of the cells),shared(as today) andassets: a map from each file thatsharednames to its entry.offline/cells/9-X-Y.jsonformat 4, one for each cell:id,bounds,files,routing,packsandrouting_packs.files,packsandrouting_packsare maps from a path to its entry, as in a step index.filesholds the search databases and the route catalog file of the cell.routingis the descriptor of today with itsarchives, or absent.packsholds the map packs of the kinds of today'smap_blocksthatselectedtakes for the box of theboundsand thegeometry_boundsof the cell (terrain with its ring of tiles).routing_packsholdspages.binandpages.idxof each archive of the cell.- A selection takes the union of the packs of its cells. Today it takes the packs of one box around the selected cells. The new rule covers the area and the retained roads of each cell, as today, and can take fewer packs at the corners of the box. The TileJSON bounds and
terrain_boundsof a bundle stay as today: the box of the selected cells and of theirgeometry_bounds.
Steps.
tools/planner_grid_pack.pyrouting: addarchivesof the routing cell manifest to each cell of the index.- New
tools/planner_grid_offline.py, the stepplanner/offline/grid. It reads only theindex.jsonof each other grid step and writesoffline/catalog.json,offline/cells/<cell>.jsonand its ownindex.json, packed withpack_file. Move here the part ofcomposethat makescells,map_blocks,sharedand the offlinereleaseheader. host/obc-data-steps/src/planner.rs: declareplanner/offline/gridin the layerplanner/indexasplanner/fonts/gridis declared, withclientofobjectsandindex.json, and add it toGATHERERS. Its inputs are theindex.jsonof the other grid steps. Because its name ends with/grid,planner/indexandplanner/catalogread its index, and it is a named file. Raise the format ofPLANNER_INDEXto 4.tools/planner_grid_index.py:stepwritesrelease.jsonformat 2 fromcompose. It no longer writesoffline/catalog.json, and its localindex.jsonno longer holdsfiles.tools/planner_runtime.py:INDEX_FORMAT = 4.planner/search/installation.mjs:GRID = 4; the search grid file does not change otherwise.tools/planner_downloads.py: readoffline/catalog.jsonformat 4 at start.preparetakes the IDs ofcellsthat the requested bounds meet and reads their cell files. A missing cell file of a listed cell is an error.quotetakes the files, packs and routing packs from the cell files, and the assets fromsharedandassets. The ID of a selection adds the SHA-256 of each selected cell file; itsformatgoes from 5 to 6. Therelease.jsonandbundle.jsonof a bundle keep their formats.host/obc-data/src/serve/services.rs:runtimereads format 2. It reads each index ofindexeswhosefoldersmeetRUNTIME, checks its bytes and SHA-256, and takes the entries underRUNTIME, then the entries offiles. Format 1 keeps today's path.objectsgets these indexes asobjects/<sha256>in a first phase, afterrelease.jsonand before the runtime files, and returns them with the runtime files, soprunekeeps them.catalog.rs:verifyandpointerbuild the map of files from the indexes ofindexesand fromfiles. A previous release of format 1 gives its map fromfiles.tools/planner_verify.py: comparerelease.jsonwith the output ofcompose, check each index ofindexeswith its bytes and SHA-256, and compose the offline files again.planner/tiles/README.md: the release ID is the digest of the named files.- Specs:
specs/planner-release.mdgets manifest format 2.specs/planner-offline.mdgetsoffline/catalog.jsonformat 4, the cell files and the selection by union.specs/obc-data.mdgets the stepplanner/offline/grid. - Tests:
tools/tests/test_planner_grid_index.py:release.jsonhasindexesand no file of a step index.tools/tests/test_planner_grid_offline.py(new): a cell file lists the packs of its geometry and the routing packs of its archives.tools/tests/test_planner_downloads.py: a request of two cells gives the union of their packs, a request reads only the cell files of its bounds, and a missing file of a listed cell fails.services.rs: the follower installs the runtime files of format 1 and format 2.
B. Engine manifest and collection (data-engine)
The format. Engine manifest format 2. A list is an object of its own: compact JSON, the keys of each object in byte order, the same items as today, at objects/<sha256> of the target and in the store. The manifest names each list as {"size", "sha256"}:
- an output has
listinstead offilesandinputs: the list object is{"files", "inputs"}with the items of today; - a source has
listinstead offiles; namedis the reference of the list of the named files.
pointers stay in the manifest. The ID is the SHA-256 of the manifest, as today.
Steps.
host/obc-data/src/release.rs:FORMAT = 2.Output,FetchedandReleasehold aListreference{size, sha256}instead of the inline lists.Release::parsechecks only the manifest. A new typeFilesholds the loaded lists:Release::load(&self, read: impl Fn(&List) -> Result<Vec<u8>, String>) -> Result<Files, String>reads each list, checks its size and SHA-256, and runs the checks ofRelease::checkthat need the lists.files,objectsandclient_filesmove toFiles. A manifest of format 1 parses into the same types: its inline lists are encoded as list objects in memory, andloadreads them from there. Addtests/manifests/format-2.json.- Publish (
release::publish): write the lists to the store when the manifest is made, and upload them with the other objects, before the manifest. host/obc-data/src/serve.rsserved: read only the list of the named files, and find the plannerrelease.jsonin it. The follower does not load the other lists.- The review, rollback, overview and
catalog.rsverify(output.files,release.objects()): useRelease::load. host/obc-data/src/collection.rs: keep the lists of the kept releases and the files that they name. Delete the other files in batches of up to 1,000 keys, then the manifests of the dropped releases, then their lists. So a collection that stops can always load the manifests that remain. AddTarget::remove_many(&[String]): on R2 it callsBucket::delete, with the reasons of today inremoved.jsonl; on a folder it removes each key. Check the lock before each batch.specs/obc-data.md: Release manifest format 2, the lists, and the collection in batches.- Tests:
release.rs: a manifest of format 2 names its lists, the ID changes when a list changes, and a manifest of format 1 still reads.collection.rs: a collection keeps the lists of the kept releases, deletes in batches, and resumes after it stops between the files and the lists.
Files
tools/planner_grid_pack.py: change:archivesin each routing cell.tools/planner_grid_offline.py: create: the stepplanner/offline/grid.tools/planner_grid_index.py: change:release.jsonformat 2, no offline catalog.tools/planner_downloads.py: change: the catalog and the cell files of format 4.tools/planner_verify.py: change:indexesand the offline files.tools/planner_runtime.py: change:INDEX_FORMAT = 4.planner/search/installation.mjs: change:GRID = 4.host/obc-data-steps/src/planner.rs,host/obc-data-steps/src/planner/catalog.rs: change: the stepplanner/offline/grid, the layer format, the map of files fromindexes.host/obc-data/src/serve/services.rs,host/obc-data/src/serve.rs: change: read both formats of both manifests.host/obc-data/src/release.rs,host/obc-data/src/collection.rs,host/obc-data/src/target.rs,host/obc-data/src/cli/(review, rollback, overview): change: the lists and the deletes in batches.host/obc-data/tests/manifests/format-2.json: create.planner/tiles/README.md: change: the release ID.specs/planner-release.md,specs/planner-offline.md,specs/obc-data.md: change: the formats.tools/tests/test_planner_grid_index.py,tools/tests/test_planner_grid_offline.py(create),tools/tests/test_planner_downloads.py, the tests inservices.rs,release.rsandcollection.rs: change.
Done when
- Each pull request passes the suites that
obc test affected --base origin/developselects, andobc exec cargo clippy -p obc-data -p obc-data-steps --all-targets -- -D warnings. - A re-bake of Central Switzerland to the local target, with workstreams 1 and 2, records against the production release:
- the bytes of
release.json, of the engine manifest and of its largest list, the number of outputs, of the largest step index, ofoffline/catalog.jsonand of the largest cell file; - the time and the peak memory of the follower install, of a download job for one cell and for the whole region, and of a collection;
- the files of a bundle of a 30 km × 30 km area against today's selection rule, with each pack that the union rule leaves out.
- the bytes of
- On the local target, the follower serves the re-bake. After a rollback to the release of today, it serves the maps of that release and refuses its planner services with the format message. Collection deletes the files of the release that goes, in batches.
- The owner downloads an offline area of the re-bake on the phone and plans a route in it.
Generated by Claude Code
Activity