Skip to content

Format 10 becomes the only format: delete the old engine and format 9, merge routing-overlay into develop #3311

Open

Description

@timohueser

Status: approved

Part of #3227.
Depends on: #3306, #3307, #3309, #3310

TL;DR

Goal: Format 10 and the new engine are the only routing data and engine: the old search, the landmarks, format 9, the packs and their tools are gone, the docs describe format 10 only, and routing-overlay is merged into develop.
Problem: On routing-overlay the old engine and the writer of format 9 stay beside the new code for the proof (epic, Plan). Each bake writes both formats, the release carries routing/blocks.json, the packs and offline/routing-cells/, the services read format 9, and the specs say that format 9 code stays. Production keeps format 9 until the branch merges.
Decision:

Facts

What goes

The check before each item: a search of each deleted name over planner/, host/, tools/, builder/ and companion-ios/ finds only deleted code; obc governs PATH of each deleted file names no guard, suite or contract that needs it (else that guard changes in the same commit); after the deletion, cargo check --workspace --all-targets and cargo check -p planner-router --target wasm32-unknown-unknown pass. An item that the code of #3302 to #3309 still uses stays or moves; the column "Stays" names where.

Where Goes Stays
planner/router/src/landmarks.rs, blocks.rs, storage.rs, directory.rs Whole: landmarks, selections, Union, Postcard pages, packs, Files, the format 9 open install.rs and cells.rs of #3301
planner/router/src/package.rs, table.rs Whole after the moves: page manifests, Source, Package, FORMAT 9, page tables and caches, Metric Endpoint and Departure move to region.rs; cell and CELL move into router-build/src/terrain.rs, their one user
planner/router/src/search.rs, queue.rs Whole: #3303 moved Seed, Query and Found to overlay.rs overlay.rs, overlay/store.rs and overlay/unpack.rs of #3303 and #3305
planner/router/src/data.rs, open.rs Prepared, Selection, the profile cache, routing_bytes, shared_bytes, retain, the whole-package path, the trait RoutingData; Opened: open gives Region<Maps> region.rs, host.rs, the search limit of Region
planner/router/src/base.rs, geometry.rs, closures.rs, snap.rs, router.rs, lib.rs Format 9 columns and write_*; page encode and decode; the paged Index and the package-wide Closures; the 0.01° squares (cells); the module lines Width, the line functions, Kind, Closure, Set, candidates, project, REACH_M
planner/router/src/lib.rs Error::Memory and Error::Limit sites that only format 9 had (#3308's last row); Error::Memory when no site returns it
planner/router-build/src/landmarks.rs, blocks.rs Whole
planner/router-build/src/lib.rs, base.rs, step.rs, grid.rs The format 9 writer in prepare (pages, manifest, write_closures, check_limits), the format 9 column writes, the output blocks and blocks::publish, grid::cells when only blocks used it base::Dictionary, group, grid::route_tiles, the columns that #3301 reads
planner/router-build/tests/routing.rs, src/osm.rs, src/catalog/tests.rs, planner/service/tests/http.rs Tests that check format 9 bytes Tests that route or read closures: they build format 10 of the same graph with cells::write and the tables of #3304 (columns_from 16) on a partition of sizes [4, 16] with download level 1, so that the routes of oracle() cross a closed cell, and read it through a Memory host, with the same assertions. oracle() stays as it is: it reads the bake's own columns, not a package
planner/router-build/src/catalog/ Directory::open of the pages, ROUTERS_MEMORY, shared_bytes The catalog, on an install of format 10 (plan step 3)
planner/router/src/compare.rs, examples/compare.rs, examples/query.rs The old engine compare of two installs of format 10, --cold, --areas
planner/service/src/ MAX_WORKERS (the count is available_parallelism(), or ROUTE_WORKERS), --verify, ROUTE_CELLS, the fork with a budget in respond; Engine is Router<Region<Maps>> --scrub of #3301
host/obc-data-steps/src/planner.rs, planner/catalog.rs, tests/planner_build.rs The output blocks, the inputs blocks/… of the routing grid step, the overlay step input planner/routing/grid (it reads planner/routing/cells/grid), the format 9 open().verify() install::check of #3301
host/obc-data/src/serve/services.rs The test for routing/directory.bin of #3307, the link of format 9 routing files, ROUTE_CELLS: the route service gets <planner folder>/route-cells/ as its argument The install of #3307
tools/planner_grid_pack.py graph, cells, blocks.json, the packs, offline/routing-cells/ routes/tiles/<cell>.json
tools/planner_grid_index.py, planner_verify.py, planner_offline.py, their tests graph reads, routing/blocks.json, the rest of L229–L237 of the verifier (#3305 changed its identity check), .bin in COMPRESSED
Cargo.toml of planner-router and planner-router-build postcard, flate2 and each other dependency that no code uses after the deletion

Plan

  1. Delete the items of "What goes", one commit for each crate or tool, each after its check.
  2. The tests of exact costs. planner/router-build/tests/engines.rs (Route engine: the road search on format 10, and the test of the old engine against the new one #3303), the_engine_on_format_10_gives_the_costs_and_candidates_of_format_9 becomes tables_give_the_costs_and_candidates_of_open_cells: the reference is town() written with an empty Tables (each slot 0 bytes, so each cell is open, Route package format 10: the contract, the writer and the installer #3301 rule 3), both on a Memory host; compare::check and the requests stay. tests/routing.rs: prepared_coordinate_routes_match_independent_arrival_road_search and closed_packages_route_across_multiple_checked_blocks route on format 10 of their graph as the table says, and still compare with oracle(); each test fails when no route of it has a settled label of a level above 0 (Route engine: the overlay search with the cost tables, and unpacking of the chosen legs only #3305's Stats). Route package format 10: the contract, the writer and the installer #3301's an_install_gives_back_each_road and Route engine: the overlay search with the cost tables, and unpacking of the chosen legs only #3305's overlay test do not change. If obc req lists a deleted test as evidence of a requirement, the pull request says so to the owner and names the new one.
  3. planner/router-build/src/catalog/mod.rs, write(directory, cells, osm, france): cells is the folder of the objects and directory.bin that cells::write made, so the step calls cells::write before catalog::write. The catalog installs all cells into <work>/catalog-install/ with install::install (4 threads), opens it with open, takes available_parallelism() workers, each with a fork of the one Region, and removes the folder at the end, also after an error. It writes route-catalog.json into directory, the output folder routing/ of the step, as today, so step.rs#L133 and the route tiles do not change. Gate on Switzerland: the same route IDs and drop reasons as the catalog of the branch before this change; the pull request records the routes whose lines differ (equal costs can give other roads) and the largest length difference.
  4. planner/router/src/router.rs, alternatives.rs, shape.rs, lib.rs: Router<H: Host> holds Region<H> in place of P: RoutingData. planner/service/src/main.rs: the argument is the install folder; host/obc-data/src/serve/services.rs passes <planner folder>/route-cells/.
  5. planner/router-build/src/step.rs (Route package format 10: the contract, the writer and the installer #3301's cells_step): index.json of planner/routing/cells/grid gets metadata: region, bounds, source_sha256 and the sorted profile names of package.json.
  6. tools/planner_grid_index.py, compose: the required kinds are the map kinds, routing (the route catalog), routing-cells, search, assets, model and fonts. The region, bounds, OSM hash and profiles come from metadata of routing-cells, not from graph. (routing_package and the overlay check use the hash of routing/directory.bin from Route engine: the overlay search with the cost tables, and unpacking of the chosen legs only #3305 on.)
  7. host/obc-data-steps/src/planner.rs: PLANNER_ROUTING gets planner_router::cells::FORMAT (10) and its revision goes up by one. planner/service/src/main.rs: --formats prints cells::FORMAT.
  8. The docs, in one docs: commit:
  9. The merge, after Follower: installs that stay on disk, one fetch at a time, a failure state for each service #3306, Follower: install format 10, and limits for memory maps #3307, Phone: an offline area of whole cells, installed with the same installer #3309 and Proof on Germany: the bake, the level sizes, the download level, the old engine against the new one #3310 are merged on routing-overlay and each gate of Proof on Germany: the bake, the level sizes, the download level, the old engine against the new one #3310 passed or the owner accepted its result:
    1. This sub-issue's pull request merges into routing-overlay after its review and green CI.
    2. The builder merges develop into routing-overlay, resolves the conflicts on the branch, and opens the pull request routing-overlay → develop. A delta review covers the resolution. CI runs obc test full and obc check full (workspace manifests and shared contracts change).
    3. The owner approves this pull request in the conversation; then the builder merges it with a merge commit.
  10. Not part of Format 10 becomes the only format: delete the old engine and format 9, merge routing-overlay into develop #3311, for the owner after the merge, in one session: a deploy of the services from develop stops routing on prod (the follower refuses the format 9 release), so nobody deploys services between the merge and this publish.
  11. The owner approves the review ID; obc data update prod --yes --expect ID uploads and waits at Router ready.
  12. obc data deploy services from the merge commit. The follower installs the format 10 release, and the run switches when it answers ready. Routing on prod is down from the deploy until the switch: the download and install of the Central Switzerland package, minutes [est]. A rollback to a format 9 release needs a deploy of the old services.
  13. The app build for the owner's phone comes from develop after the switch (its downloads need format 10 bundles). Before the owner runs it, the offline maps of the old format go (Settings, or a new install of the app): the new app does not open them.
    The refusal in step 2 takes down the whole serving release, also its search and downloads, not only routing.

Files

  • Deleted: planner/router/src/landmarks.rs, blocks.rs, package.rs, storage.rs, table.rs, directory.rs, search.rs, queue.rs, open.rs (its open moves to lib.rs); planner/router-build/src/landmarks.rs, blocks.rs.
  • Changed: the other files of "What goes"; planner/router/src/router.rs, alternatives.rs, shape.rs, region.rs; planner/router-build/src/step.rs, terrain.rs, catalog/mod.rs, tests/engines.rs; planner/service/src/main.rs, http.rs, lib.rs; host/obc-data-steps/src/planner.rs, planner/catalog.rs; host/obc-data/src/serve/services.rs; tools/planner_grid_index.py, planner_grid_pack.py, planner_verify.py, planner_offline.py and their tests; the specs and READMEs of step 8; Cargo.lock.

Done when

  • obc test -p planner-router, -p planner-router-build, -p planner-service, -p obc-data, and obc exec cargo clippy -p <crate> --all-targets -- -D warnings for each, for planner-service also with --no-default-features. obc test -p obc-data-steps in CI (it needs GEOS). cargo check -p planner-router --target wasm32-unknown-unknown. cargo fmt --all. obc test affected --base origin/routing-overlay; cargo deny, because Cargo.lock changes.
  • The test of step 2 passes with all profiles. A Switzerland bake on the owner PC under nice gives the same directory hash as before this change, and its release has no file under routing/packs/, no routing/blocks.json and no offline/routing-cells/.
  • obc suites check (test files go), obc prose --check (no README grows), python3 docs/build_docs.py --check-links. firmware/tools/resource_baseline.json does not apply: no firmware changes.
  • The catalog: on the Switzerland bake, the pull request records the time and the own peak of the catalog stage and the disk peak of its install. Only when Proof on Germany: the bake, the level sizes, the download level, the old engine against the new one #3310 handed over its catalog gate (bake A did not finish because of the catalog): one Germany bake with this change in the cloud container of Proof on Germany: the bake, the level sizes, the download level, the old engine against the new one #3310 (4 cores, 15 GB, the input of Proof on Germany: the bake, the level sizes, the download level, the old engine against the new one #3310) finishes with an own peak below 15 GB, with the same records.
  • obc data update local on the branch, obc data serve local, and a route through the local follower answer with package equal to the routing_package of the release and of maps/overlays.json (as from Route engine: the overlay search with the cost tables, and unpacking of the chosen legs only #3305).
  • Both pull requests have Requirements: with each requirement whose plan cites a test that this change deletes or moves (from obc req), and the route requirements; after the merge the builder proposes the plan updates (propose --check, then propose) and tells the owner each deleted evidence test. Both say that public docs changed.
  • The merge pull request has green CI and the owner's approval before the merge.
  • Owner, after the first deploy from develop (step 10): the setup check of obc data deploy on the VPS shows vm.max_map_count as passed. Follower: install format 10, and limits for memory maps #3307 adds the sysctl file and the check, and cannot verify them on the production VPS before the merge.

Out of scope

Progress

Pull request #3498 merged Plan steps 1 to 8 into routing-overlay. The issue stays open: Plan step 9 (the merge into develop) needs the owner's approval.

  • Done: the old engine, format 9, the packs and their tools are gone (about 5,500 deleted lines). The route catalog routes on an install of format 10. open gives Region<Maps>; the route service takes the install folder. The layer planner/routing has format 10. The specs and READMEs describe format 10 only.
  • Built differently:
    • tests/routing.rs uses the cell sizes [4, 8], not [4, 16]: its graph of 12 junctions has one level with [4, 16].
    • shape::shape can take back a shaping point that the prune removed. A leg of a whole route starts on the road where the leg before it ended, so it can break a tie between ways of equal cost in another way than the leg routed alone; the repair of the whole route then found no new point and gave Failure::Check. This is new behavior, which this issue put out of scope: it also changes POST /v1/shape of the route service and the shaping of the phone library, where a line that answered 422 not_reproducible can now get a plan. The owner's answer on it is open.
    • The follower refuses a planner release without routing/directory.bin at its fetch.
    • planner/router/src/base.rs is deleted whole. examples/query.rs stays, on an install. The catalog uses the search limit of the region (64 MiB). The work layers keep the format 12, so a bake reuses them. compare keeps its options; OLD is an install of format 10.
    • New helpers: cells::memory, Tables::none, work::Folder.
  • Measured on Switzerland (download level 3, 4 cores, 15 GB RAM): the hash of cells/directory.bin is the same before and after (1726ad5a…), and overlays.sqlite does not change. The route catalog has 4,589 route IDs: the 4,588 of before and 9683671, which the shaping repair now keeps. 906687 is still dropped, as ShapingStall in place of ShapingCheck. 46 lines differ; the largest length difference is 528 m.
  • The catalog stage is slower: 184 to 290 s against about 105 s, with an own peak of 464 MiB against 1,437 MiB. The snap takes 69 % of its routing time (148,190 calls at 3.3 ms). Route engine: the snap on format 10 decodes each road of its window on each request #3471 has the design of the fix.
  • Tests: 19 tests of format 9 bytes are deleted and 2 are renamed; the pull request lists each for the owner. obc req did not run (no token), so the requirements that cite them are not checked.
  • Next: pull request Merge develop into routing-overlay #3500 merges develop into routing-overlay (no textual conflict). Then the pull request routing-overlay → develop, with the owner's approval.
  • Open, for the owner: obc data update local and obc data serve local with a route through the local follower; the check of the release folder (no routing/packs/, no routing/blocks.json, no offline/routing-cells/). The commands are under "Owner steps" in pull request Format 10 is the only routing format: delete the old engine and format 9 #3498.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions