Skip to content

Manifests: small release manifests, per-cell offline catalogs and file lists as objects #3141

Closed
2 of 2 issues completed

Description

@timohueser

Status: done
Part of workstream 3 of #3000.

TL;DR

Goal: No manifest of a release grows with the whole coverage. Each reader reads only the lists of the parts that it uses. The phone reads a release.json of about 10 KB. The download service reads the cells of a request. The engine manifest of an update grows with the number of steps, not with the number of files.

Problem: Four documents list every file of a release, and each reader parses them whole.

  • The planner release.json is 1.9 MB with 7,215 files today, and about 230 MB with larger packs (estimated). The phone downloads it for two fields.
  • offline/catalog.json is 2.2 MB. 2.0 MB of it is a second copy of the files of release.json.
  • The engine manifest releases/<id>.json of obc data is 6.5 MB today. It lists 12,174 output files (2.0 MB), the source files (3.3 MB, of which 6,390 MODIS files) and 1,804 named files (0.4 MB). The follower, the review, rollback and collection parse it whole. At about 160 B per output file, it grows to 150 MB or more at the target coverage (estimated).
  • Collection lists all keys in one rclone lsjson document and deletes one key per request.

Decision:

  1. Planner release.json, format 2. It loses files and gets indexes: the path, bytes, SHA-256 and top folders of the index.json of each grid step. These step indexes already hold the entries of their files, and they are already named files, so the release ID already covers them. files keeps only the client files that no step index lists: search/<region>.grid.json. The header (profiles, routing_package, the attributions) stays, so the phone and the web planner do not change.
  2. Offline catalog per cell. A new step planner/offline/grid reads the indexes of the other grid steps. It writes a small offline/catalog.json (the release header, the zoom, the routing source, the cell IDs, and the shared assets with their entries) and one offline/cells/9-X-Y.json for each cell: its bounds, its files, its routing descriptor, and the map packs that it selects, each with its entry. The download service reads the catalog at start, and the cell files of a request.
  3. The follower reads release.json, then only the step indexes whose folders it serves (routing/, search/, offline/). It reads planner manifest format 1 and format 2. The planner services read only the new format of planner/index, so after a rollback to a release from before this change, the follower serves its maps but not its routing, search and downloads. This is the rule for every format change of a layer that a service reads.
  4. Engine manifest, format 2. Each list of files goes to an object of its own: the files and the inputs of each output, the files of each source, and named. The manifest keeps the size and the SHA-256 of each list. The release ID is still the SHA-256 of the manifest, so it proves every list.
  5. Collection deletes up to 1,000 keys per request.
  6. routing/blocks.json, and the copy of its graph in the routing grid index (3.5 MB today), stay with workstream 4.

Rejected:

  • Move profiles and routing_package into planner/catalog.json (the direction of EPIC — Scale readiness: 100 concurrent users, long routes, from one region to the world #3000): release.json becomes about 10 KB, so the phone reads it as today, with no new app release.
  • Omit the transport entry where it repeats the file (6,867 of 7,215 entries): about 45% less per list, but each reader now reads only a part of the lists, and the change touches each reader of an entry.
  • The download service finds the packs in the indexes of the map layers: it parses the whole index of each layer for each job.
  • A streaming JSON parser for the large documents: each reader needs a new parser, and the documents still upload and download whole.
  • Move only the output files of the engine manifest, or leave it to workstream 7: the source files are the largest part today. One rule for all the lists keeps the manifest small.
  • The planner services read planner/index format 3 and format 4, so that a rollback keeps its services: each service then reports a list of formats, and the download service keeps the reader of today. Every other format change of a service layer already stops the services of an older release.
  • Collection lists the keys line by line (rclone lsf) instead of one rclone lsjson document: the keys and the set of kept keys stay in memory anyway, so it saves little. The re-bake measures the memory of a collection.

Systems: planner-map-baking, data-engine, search (GRID in planner/search/installation.mjs), planner-tile-server (its README says that the release ID is the SHA-256 of release.json, which is wrong)
Contracts: specs/planner-release.md, specs/planner-offline.md, specs/obc-data.md
Size: L (a breaking change of two stored manifest formats and of the format of planner/index)

Words

  • Target: the bucket that a release goes to: R2 (prod) or a local folder. Store: the local folder of obc data on a machine, with the objects by SHA-256 and a receipt per step.
  • Step: one build job of obc data. A grid step (planner/<kind>/grid) packs the files of one kind for the planner. Its step index (index.json) maps each logical path to its entry: {bytes, sha256, transport: {bytes, sha256, encoding}}. The transport is the stored bytes: the file itself (identity) or its gzip.
  • Named files: the files that a client finds by path, at releases/<id>/<path> of the target: the planner release.json, public/ and the step indexes. Each other file is at objects/<sha256>.
  • Engine manifest: releases/<id>.json, the manifest of obc data that lists the outputs of each step, the sources and the named files of a release. Planner release.json: the manifest that the planner clients read.
  • Follower: obc data serve on the VPS. It reads the target every 30 seconds, downloads the files of the serving release that the planner services need, and runs those services: routing, search and downloads.
  • Collection: after an update, obc data deletes the keys of the target that no kept release names (the serving release and the rollback).

Facts

Measured on the production release bf0927b0cd57 (Central Switzerland), read from maps.openbikecomputer.com:

Document Size Content
Planner release.json 1.9 MB 7,215 files. 6,867 have a transport that is equal to the file
offline/catalog.json 2.2 MB files (2.0 MB) is equal to files of release.json. Also cells (9 cells, 63 KB, of which about 4 KB per cell is the routing adjacency), map_blocks (487 packs, 82 KB) and shared (1,066 aliases, 105 KB)
Engine manifest releases/313857d5….json 6.5 MB sources 3.3 MB (modis-snow alone has 6,390 files, 2.7 MB). outputs 2.9 MB: files 2.0 MB for 12,174 files, inputs 0.8 MB. named 0.4 MB, 1,804 files
Step indexes indexes/<step>/index.json 66 files Basemap 31 KB, assets 286 KB. The routing grid index is 3.5 MB, because it holds a copy of the graph of routing/blocks.json

Writers

  • compose and step of tools/planner_grid_index.py merge the files of the index.json of each grid step into one map. step writes that map three times: into release.json (line 132), into offline/catalog.json (line 128) and into the local index.json of planner/index (line 138).
  • pack_file of tools/planner_offline.py makes each entry {bytes, sha256, transport}. It stores .pmtiles, .bin, .webp and .png files without compression (identity).
  • The planner release ID is the digest of the named files: release.json, public/ and each indexes/<step>/index.json (named in host/obc-data-steps/src/planner/catalog.rs). So each step index is already a file list that the ID covers.
  • routing of tools/planner_grid_pack.py writes the routing grid index. Its cells hold the routing descriptor of each cell: id, bounds, geometry_bounds, adjacency, manifest and sha256. It reads each routing cell manifest, and that manifest names the routing packs of the cell (archives).
  • The engine manifest ID is the SHA-256 of Release::canonical (host/obc-data/src/release.rs). FORMAT is 1. A follower reads its own format and the one before; tests/manifests/format-1.json holds a manifest of format 1.

Readers of the planner release.json

  • The phone: PlannerService.manifest in PlannerService.swift downloads the whole file and decodes only routing_package and profiles. Other keys are ignored.
  • The offline bundle on the phone (OfflineMap.swift, OfflinePlanner.swift) reads the release.json that the download service writes for each selection, not the file of the release.
  • The web planner uses the manifest URL only as the base of public/ (terrain.ts). release.ts reads only active of the catalog.
  • The follower: runtime in host/obc-data/src/serve/services.rs parses the whole document and keeps the entries under RUNTIME (routing/, search/, offline/).
  • verify of catalog.rs reads each entry and compares it with the previous release. pointer of catalog.rs reads files only to see whether a file exists. verify of tools/planner_verify.py requires the document to be equal to the output of compose, and compares offline/catalog.json.

Readers of offline/catalog.json

  • Downloads in tools/planner_downloads.py parses the whole file at start. prepare scans all cells. quote makes one geometry box from the selected cells, scans all map_blocks with selected of tools/planner_map_archive.py (terrain adds a ring of tiles), and takes each entry from files. The routing packs of a selection are the archives of the merged routing cell manifests.
  • No iOS or Rust code reads it. tools/planner_verify.py composes it again.

Readers of the engine manifest

  • The follower parses the whole manifest (served in host/obc-data/src/serve.rs) and finds the planner release.json in named.
  • The review, rollback and overview use Release::files().
  • Collection (host/obc-data/src/collection.rs) builds the set of keys to keep from Release::files() of the kept releases. It lists the target with rclone lsjson --recursive (lsjson in host/obc-data/src/r2.rs) into one document in memory. It deletes one key per request (target.remove) and checks the lock every 100 deletes (BATCH). Bucket::delete of r2.rs already deletes many keys in one rclone call.

Other facts

  • Each planner service reads one format of each layer, and the follower starts the services of a release only when the formats match (readable in services.rs). INDEX_FORMAT of tools/planner_runtime.py (3) is the format of the layer planner/index. The download service reports it, and the search service checks it as GRID in planner/search/installation.mjs, because search/<region>.grid.json is a file of that layer.
  • planner/catalog computes the release ID from the inputs that it reads: planner/index and the index.json of each step whose name ends with /grid (indexes in host/obc-data-steps/src/planner.rs). So a new step named planner/offline/grid joins the ID, the named files and the inputs of planner/index with no other change.
  • prune in services.rs keeps in the store only what objects returns for the served releases.
  • Collection deletes the files of a dropped release first and its manifest last, so that a collection that stops can resume from the manifests that remain. On R2 it lists only objects/ and releases/.
  • No reader uses a streaming JSON parser. MAX_MANIFEST_BYTES (128 MiB) limits only the routing manifest.
  • planner/tiles/README.md line 56 says that the release ID is the SHA-256 of release.json. The ID is the digest of the named files.

Plan

This is an epic with two sub-issues. Each sub-issue is one pull request.

B depends on A: both raise what the follower reads, and B raises the engine manifest FORMAT once for both. Both merge before the one re-bake of workstreams 1, 2 and 3 of #3000. No release is published between the merge of A and the deploy of the follower of B. The follower says its manifest format at /planner-api/version, and the review of an update of prod names a follower that cannot read the release, so a release of the new formats does not reach a follower of today.

Verification

The code of A and B is in develop (#3146, #3147). The verification steps on real data are in a comment on #3000, with those of workstream 1, because they run in the one re-bake of workstreams 1, 2 and 3.

A. Planner manifests (planner-map-baking, data-engine, search, planner-tile-server)

The formats.

  • Planner release.json format 2 has the header of format 1 (region, name, bounds, osm_sha256, routing_package, profiles, the attributions, terrain_bounds, grid) and:
    • indexes: for each grid step, planner/offline/grid included, sorted by step name, {"<step>": {"bytes", "sha256", "folders"}}. The file is indexes/<step>/index.json of the release. folders is the sorted list of the first path parts of the files of the index, for example ["offline", "routes", "routing"].
    • files: the entries of the client files of planner/index that no step index lists: search/<region>.grid.json.
  • offline/catalog.json format 4: format, zoom, release (the header, as today), routing_source, cells (the sorted IDs of the cells), shared (as today) and assets: a map from each file that shared names to its entry.
  • offline/cells/9-X-Y.json format 4, one for each cell: id, bounds, files, routing, packs and routing_packs. files, packs and routing_packs are maps from a path to its entry, as in a step index. files holds the search databases and the route catalog file of the cell. routing is the descriptor of today with its archives, or absent. packs holds the map packs of the kinds of today's map_blocks that selected takes for the box of the bounds and the geometry_bounds of the cell (terrain with its ring of tiles). routing_packs holds pages.bin and pages.idx of each archive of the cell.
  • A selection takes the union of the packs of its cells. Today it takes the packs of one box around the selected cells. The new rule covers the area and the retained roads of each cell, as today, and can take fewer packs at the corners of the box. The TileJSON bounds and terrain_bounds of a bundle stay as today: the box of the selected cells and of their geometry_bounds.

Steps.

  1. tools/planner_grid_pack.py routing: add archives of the routing cell manifest to each cell of the index.
  2. New tools/planner_grid_offline.py, the step planner/offline/grid. It reads only the index.json of each other grid step and writes offline/catalog.json, offline/cells/<cell>.json and its own index.json, packed with pack_file. Move here the part of compose that makes cells, map_blocks, shared and the offline release header.
  3. host/obc-data-steps/src/planner.rs: declare planner/offline/grid in the layer planner/index as planner/fonts/grid is declared, with client of objects and index.json, and add it to GATHERERS. Its inputs are the index.json of the other grid steps. Because its name ends with /grid, planner/index and planner/catalog read its index, and it is a named file. Raise the format of PLANNER_INDEX to 4.
  4. tools/planner_grid_index.py: step writes release.json format 2 from compose. It no longer writes offline/catalog.json, and its local index.json no longer holds files. tools/planner_runtime.py: INDEX_FORMAT = 4. planner/search/installation.mjs: GRID = 4; the search grid file does not change otherwise.
  5. tools/planner_downloads.py: read offline/catalog.json format 4 at start. prepare takes the IDs of cells that the requested bounds meet and reads their cell files. A missing cell file of a listed cell is an error. quote takes the files, packs and routing packs from the cell files, and the assets from shared and assets. The ID of a selection adds the SHA-256 of each selected cell file; its format goes from 5 to 6. The release.json and bundle.json of a bundle keep their formats.
  6. host/obc-data/src/serve/services.rs: runtime reads format 2. It reads each index of indexes whose folders meet RUNTIME, checks its bytes and SHA-256, and takes the entries under RUNTIME, then the entries of files. Format 1 keeps today's path. objects gets these indexes as objects/<sha256> in a first phase, after release.json and before the runtime files, and returns them with the runtime files, so prune keeps them.
  7. catalog.rs: verify and pointer build the map of files from the indexes of indexes and from files. A previous release of format 1 gives its map from files.
  8. tools/planner_verify.py: compare release.json with the output of compose, check each index of indexes with its bytes and SHA-256, and compose the offline files again.
  9. planner/tiles/README.md: the release ID is the digest of the named files.
  10. Specs: specs/planner-release.md gets manifest format 2. specs/planner-offline.md gets offline/catalog.json format 4, the cell files and the selection by union. specs/obc-data.md gets the step planner/offline/grid.
  11. Tests: tools/tests/test_planner_grid_index.py: release.json has indexes and no file of a step index. tools/tests/test_planner_grid_offline.py (new): a cell file lists the packs of its geometry and the routing packs of its archives. tools/tests/test_planner_downloads.py: a request of two cells gives the union of their packs, a request reads only the cell files of its bounds, and a missing file of a listed cell fails. services.rs: the follower installs the runtime files of format 1 and format 2.

B. Engine manifest and collection (data-engine)

The format. Engine manifest format 2. A list is an object of its own: compact JSON, the keys of each object in byte order, the same items as today, at objects/<sha256> of the target and in the store. The manifest names each list as {"size", "sha256"}:

  • an output has list instead of files and inputs: the list object is {"files", "inputs"} with the items of today;
  • a source has list instead of files;
  • named is the reference of the list of the named files.

pointers stay in the manifest. The ID is the SHA-256 of the manifest, as today.

Steps.

  1. host/obc-data/src/release.rs: FORMAT = 2. Output, Fetched and Release hold a List reference {size, sha256} instead of the inline lists. Release::parse checks only the manifest. A new type Files holds the loaded lists: Release::load(&self, read: impl Fn(&List) -> Result<Vec<u8>, String>) -> Result<Files, String> reads each list, checks its size and SHA-256, and runs the checks of Release::check that need the lists. files, objects and client_files move to Files. A manifest of format 1 parses into the same types: its inline lists are encoded as list objects in memory, and load reads them from there. Add tests/manifests/format-2.json.
  2. Publish (release::publish): write the lists to the store when the manifest is made, and upload them with the other objects, before the manifest.
  3. host/obc-data/src/serve.rs served: read only the list of the named files, and find the planner release.json in it. The follower does not load the other lists.
  4. The review, rollback, overview and catalog.rs verify (output.files, release.objects()): use Release::load.
  5. host/obc-data/src/collection.rs: keep the lists of the kept releases and the files that they name. Delete the other files in batches of up to 1,000 keys, then the manifests of the dropped releases, then their lists. So a collection that stops can always load the manifests that remain. Add Target::remove_many(&[String]): on R2 it calls Bucket::delete, with the reasons of today in removed.jsonl; on a folder it removes each key. Check the lock before each batch.
  6. specs/obc-data.md: Release manifest format 2, the lists, and the collection in batches.
  7. Tests: release.rs: a manifest of format 2 names its lists, the ID changes when a list changes, and a manifest of format 1 still reads. collection.rs: a collection keeps the lists of the kept releases, deletes in batches, and resumes after it stops between the files and the lists.

Files

  • tools/planner_grid_pack.py: change: archives in each routing cell.
  • tools/planner_grid_offline.py: create: the step planner/offline/grid.
  • tools/planner_grid_index.py: change: release.json format 2, no offline catalog.
  • tools/planner_downloads.py: change: the catalog and the cell files of format 4.
  • tools/planner_verify.py: change: indexes and the offline files.
  • tools/planner_runtime.py: change: INDEX_FORMAT = 4.
  • planner/search/installation.mjs: change: GRID = 4.
  • host/obc-data-steps/src/planner.rs, host/obc-data-steps/src/planner/catalog.rs: change: the step planner/offline/grid, the layer format, the map of files from indexes.
  • host/obc-data/src/serve/services.rs, host/obc-data/src/serve.rs: change: read both formats of both manifests.
  • host/obc-data/src/release.rs, host/obc-data/src/collection.rs, host/obc-data/src/target.rs, host/obc-data/src/cli/ (review, rollback, overview): change: the lists and the deletes in batches.
  • host/obc-data/tests/manifests/format-2.json: create.
  • planner/tiles/README.md: change: the release ID.
  • specs/planner-release.md, specs/planner-offline.md, specs/obc-data.md: change: the formats.
  • tools/tests/test_planner_grid_index.py, tools/tests/test_planner_grid_offline.py (create), tools/tests/test_planner_downloads.py, the tests in services.rs, release.rs and collection.rs: change.

Done when

  • Each pull request passes the suites that obc test affected --base origin/develop selects, and obc exec cargo clippy -p obc-data -p obc-data-steps --all-targets -- -D warnings.
  • A re-bake of Central Switzerland to the local target, with workstreams 1 and 2, records against the production release:
    • the bytes of release.json, of the engine manifest and of its largest list, the number of outputs, of the largest step index, of offline/catalog.json and of the largest cell file;
    • the time and the peak memory of the follower install, of a download job for one cell and for the whole region, and of a collection;
    • the files of a bundle of a 30 km × 30 km area against today's selection rule, with each pack that the union rule leaves out.
  • On the local target, the follower serves the re-bake. After a rollback to the release of today, it serves the maps of that release and refuses its planner services with the format message. Collection deletes the files of the release that goes, in batches.
  • The owner downloads an offline area of the re-bake on the phone and plans a route in it.

Generated by Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions