Skip to content

Phone: an offline area of whole cells, installed with the same installer #3309

Closed

Description

@timohueser

Status: done

Part of #3227.
Depends on: #3305

TL;DR

Goal: A phone installs the routing of an offline area as whole download cells with the installer of #3301, the areas of one routing package share one install, and the phone routes on it with the engine of #3303 and #3305.
Problem: Each offline map is its own routing package: the download service merges the format 9 selections of its zoom-9 tiles in Python into a new routing/blocks.json, and the phone downloads 1.03 to 3.3 times the bytes of its tiles in 11 to 25 packs (measured on Central Switzerland, 2.1 to 54 MB, 133 to 425 B for each road). The phone decodes the selected graph into its own memory at each open (37.5 + 14.5 B for each road and profile; about 25 to 30 million roads is the limit of a 4 GB iPhone [est]). Two areas are two routing packages: they share only the packs whose consumers they share, and the phone decodes each one into its own memory.
Decision:

Facts

The size of an area

An area of side t takes the download cells (side c) that meet it: about (1 + c / t)² times its own roads, plus the overlap of the cell boxes. So the download level decides the factor first, then that overlap; the ratio of installed to published bytes, about 1.8 (95 B published and 170 B installed for each road with the tables of 10 profiles, #3301 and #3304), does not change with it. #3301 gives download cells of 160,000 to 650,000 roads at its default level; with cells of 250,000 roads:

Area Own roads c Roads installed Format 10: download / installed Format 9 today
One zoom-9 tile in the Alps (70 roads/km², 2,800 km²) 0.2 M 60 km 0.9 M (× 4.5) 85 MB / 150 MB 27 to 85 MB
One zoom-9 tile around a city (450 roads/km²) 1.26 M 24 km 2.6 M (× 2.1) 250 MB / 450 MB 170 to 250 MB
3 × 3 tiles in the Alps 1.8 M 60 km 3.4 M (× 1.9) 320 MB / 580 MB 240 to 760 MB

So a one-tile area downloads about 1 to 3 times the routing bytes of today and installs about 1.8 to 5.6 times, because the installed files are not compressed. A lower download level lowers the factor: cells of 80,000 roads give × 2.7 and × 1.5 for the two tiles. #3310 measures the factor on Germany for each level and chooses the level.

Plan

  1. planner/router/src/install.rs (Route package format 10: the contract, the writer and the installer #3301): pub fn area_cells(directory: &Directory, areas: &[[f64; 4]]) -> Vec<u32>, the sorted download cell IDs of all areas, each once. An area [west, south, east, north] in degrees goes to microdegrees outward (floor for west and south, ceil for east and north). Then it grows on each side by the half-sizes of snap::window(corner, REACH_M) (Route engine: the data layer on format 10 and the host interface #3302), where corner is the corner of the area with the larger absolute latitude (clamped to ±85°): the largest window of any point inside. The cells are those of Directory::meeting (Route engine: the data layer on format 10 and the host interface #3302) on the grown box. Long roads do not count, as in Route package format 10: the contract, the writer and the installer #3301 rule 2.
  2. planner/service/src/native.rs:
    • planner_router_open(root, memory_budget_bytes, error): the signature stays. It calls planner_router::open; 0 gives default_budget(), the search limit of one search. A folder without a valid install.bin (Route package format 10: the contract, the writer and the installer #3301 rule 4) gives the error of Route engine: the road search on format 10, and the test of the old engine against the new one #3303, as JSON in error. The comment of the budget says that it is the search limit. Format 10 becomes the only format: delete the old engine and format 9, merge routing-overlay into develop #3311 does not change the signature.

    • planner_routing_plan(request: *const c_char, error) -> *mut c_char and planner_routing_install(request: *const c_char, progress: Option<extern "C" fn(*mut c_void, u64, u64) -> bool>, context: *mut c_void, error) -> bool. The request is UTF-8 JSON:

      Field Type Meaning
      package, directory string The paths of package.json and directory.bin
      target string The install folder
      areas array of [w, s, e, n] Degrees: the bounds of each map of the package
      objects string Only for install: the folder of the objects, each named by its SHA-256
    • The plan answers {"objects": [{"sha256", "bytes", "part_bytes"}], "bytes"}: each object of install::plan(directory, area_cells(areas), all groups).missing(target), with its object_bytes and bytes; the top bytes is installed_bytes of the whole plan plus package.json and directory.bin.

    • The install calls install::install with INSTALL_THREADS = 2 and a fetch that opens objects/SHA256. Comment of the constant: each worker holds a zstd window of up to 8 MiB and its buffers; the install of an area takes seconds. fetch wraps the file in a reader that adds its bytes to a shared count and calls progress(context, done, total) after each read, with total the object_bytes of the missing objects; when progress returns false, the reader fails with "cancelled", the install stops and a later call goes on (Route package format 10: the contract, the writer and the installer #3301). With an empty areas, the call takes install.lock, deletes target and returns true. The store runs one call at a time, so the lock of Route package format 10: the contract, the writer and the installer #3301 only stops a second app process.

    • An error writes {"status", "body": {"code", "message"}} into error, as planner_router_open does: invalid_request for a request that does not parse or a path that is not UTF-8, invalid_data for an error of Route package format 10: the contract, the writer and the installer #3301 on an object or the directory, internal with the path for an I/O error, cancelled for a stop.

  3. companion-ios/Packages/OBCKit/Sources/OBCPlanner/OfflineMap.swift: OfflineMap.routing (the hex SHA-256 of directory.bin). protocol OfflineRouting: Sendable with plan(package:directory:target:areas:) async throws -> OfflineRoutingPlan and install(package:directory:target:areas:objects:progress:) async throws; OfflineRoutingPlan { objects: [OfflineFile] (with part bytes), bytes: Int64 }. OfflineDownloadQuote.routing: [OfflineFile]. The selected release must have offline.format 4 and routing/package.json and routing/directory.bin among its files, and routing_package must be the SHA-256 of routing/directory.bin.
  4. OfflineMapStore.swift:
    • init takes routing: any OfflineRouting. func routing(_ map: OfflineMap) -> URL is root/routing/<map.routing>.
    • prepare: after the bundle, it fetches the two routing files through objects/ of the bundle with OfflineTransfer, the path for large files with resume and the check of each object, into objects/SHA256. A world directory has about 78 bytes for each file of the package: about 3 MB at 35,000 files, about 19.5 MB at the 250,000 files that Proof on Germany: the bake, the level sizes, the download level, the old engine against the new one #3310 allows, above the 16 MiB of get. The quote shows a status while it loads. It plans the union of the areas of the maps of that package with the new bounds: the missing objects go to transferBytes and routing. A second plan of the new bounds alone gives the routing part of installedBytes.
    • needed: also the allocated bytes of each missing routing object and of its part.
    • install: it plans the union again, because the maps can change after prepare, and writes pending.json with that list. The bundle objects and the routing objects download in the one task group of today, as identity entries. Then the release folder is linked as today, then routing.install runs with the progress message "Installing routes…" (the callback reads the flag of step 6), then maps.json gets the map, pending.json goes, and collectObjects runs. The order makes the map visible only after its routing is complete; until then the old install.bin of the folder holds.
    • remove: first routing.install with the areas of the other maps of its package (an empty list deletes the folder), then maps.json without the map, then its release folder goes, then collectObjects. No routing object downloads for a removal. The install comes before maps.json also here: the actor can run a route during the await, and a reader that it opens then must not be cached under the new map list on the old install.bin. A route during the install reads its mapped files, or gets missing_region for a removed one and goes online once; the next route after the new maps.json opens a new reader.
    • collectObjects also keeps the objects of pending.routing. Routing objects are in no release.json, so they go after each install.
    • installedBytes of a map is its bundle bytes plus the installed routing bytes of its own area alone. A cell that two maps share counts in each, so the list can show more than the storage that the app uses; the size in the download screen is the routing that this download adds (requiredBytes).
    • coverage() takes /catalog format == 2.
  5. companion-ios/PlannerNative/PlannerNative.h (new): the C declarations of the five functions of native.rs (planner_router_open, planner_router_call, planner_router_cancel, planner_router_close, planner_response_free) and the two new ones, with an opaque PlannerHandle. companion-ios/PlannerNative/BridgingHeader.h (new) includes it; project.yml sets SWIFT_OBJC_BRIDGING_HEADER of the app target. RouteProvider.swift loses its @_silgen_name lines and calls the functions of the header. The comment of the header: keep it equal to the #[no_mangle] block of planner/service/src/native.rs.
  6. companion-ios/PlannerNative/RoutingInstaller.swift (new; project.yml lists it beside RouteProvider.swift): OfflineRouting over the two calls. Each call runs on one serial DispatchQueue (QoS utility) and returns through a checked continuation inside withTaskCancellationHandler, so no blocking call runs on the cooperative pool. The progress context is an Unmanaged box that holds an atomic flag and the Swift progress closure; the cancellation handler sets the flag. The callback runs on a Rust worker thread, which has no current task, so it never reads Task.isCancelled: it returns the inverse of the flag and passes the counts on.
  7. LocalFirstPlanner.swift: Open becomes (OfflineMap, URL, URL): the release folder and store.routing(map). The cached source goes when the set of map IDs changes, so a route after an install or a removal opens a new reader, as Route package format 10: the contract, the writer and the installer #3301 rule 4 asks; a route that runs during a removal keeps its mapped files (Route engine: the data layer on format 10 and the host interface #3302) or gets missing_region and goes online. The type comment: the maps of one routing package share one install; each route uses the install of the first map that holds its points. OBCCompanion/OfflinePlanner.swift: open(map:directory:routing:), RouteProvider(directory: routing), offline.format 4, no source_routing. OBCCompanionApp.swift: the store gets RoutingInstaller().
  8. tools/planner_grid_offline.py: shared and assets of the catalog also hold routing/package.json and routing/directory.bin of the index of kind routing-cells. The packs of a cell come from its bounds. routing_source, routing and routing_packs go. tools/planner_grid_index.py: the routing cell check and the descriptors in compose go (#L67-L73). tools/planner_grid.py: ranges, table, merge and routing go. tools/planner_runtime.py: INDEX_FORMAT = 5; planner/search/installation.mjs: GRID = 5; host/obc-data-steps/src/planner.rs: PLANNER_INDEX format 5. host/obc-data-steps/tests/planner_build.rs: the bundle asserts routing/package.json and routing/directory.bin and no routing/packs/, and the loop over the cell entries reads files and packs only.
  9. tools/planner_downloads.py, quote: no routing selection. The selected release has region area- plus the first 24 hex digits of the ID, routing_package the SHA-256 of the routing/directory.bin entry, the TileJSON bounds of the requested cells, terrain_bounds of them, and offline with format: 4, id, zoom and cells (no source_routing). The selection format of the ID is 7. GET /catalog answers format: 2, so an app of today refuses before a download.
  10. specs/planner-offline.md:
    • Versions: offline/catalog.json and offline/cells/CELL.json format: 5, service catalog format: 2, selected release offline.format: 4, selection format 7.
    • Grid publication: the routing of an area is the rule of step 1, with a link to the route package; the catalog shared holds the two routing files; the cell file loses routing and routing_packs, the catalog routing_source; the packs of a cell come from its bounds.
    • Selected release: routing_package is the identity of the directory; no routing/blocks.json; the TileJSON and terrain bounds are those of the cells.
    • Download service: objects/SHA256 also gives each cell object of the release, by a 307 to the pool.
    • iOS library: routing/<directory>/ is an install of the route package with the union of the areas of the maps of that package; the store calls the installer after each change of maps.json; routing objects stay in objects/ only until their install; a maps.json entry has routing. "Routing graphs are not combined" goes. The known difference: a long road of a download cell that the area does not install is no candidate, where the server can snap to it (Route package format 10: the contract, the writer and the installer #3301 rule 2).
      planner/service/README.md, phone library: the two calls, and the budget of the open as the search limit. companion-ios/PlannerNative/README.md: one sentence on RoutingInstaller.

Files

  • planner/router/src/install.rs: change: area_cells.
  • planner/service/src/native.rs: change: the open on format 10, plan, install, the test.
  • planner/router-build/tests/cells.rs: change: the area test.
  • companion-ios/Packages/OBCKit/Sources/OBCPlanner/OfflineMap.swift, OfflineMapStore.swift, LocalFirstPlanner.swift: change.
  • companion-ios/Packages/OBCKit/Tests/OBCPlannerTests/OfflineMapsTests.swift: change: the fake installer, the new test.
  • companion-ios/PlannerNative/RoutingInstaller.swift, PlannerNative.h, BridgingHeader.h: create. RouteProvider.swift, README.md, companion-ios/project.yml, OBCCompanion/OfflinePlanner.swift, OBCCompanion/OBCCompanionApp.swift: change.
  • tools/planner_downloads.py, planner_grid.py, planner_grid_offline.py, planner_grid_index.py, planner_runtime.py, tools/tests/test_planner_downloads.py, test_planner_grid_offline.py, test_planner_grid_index.py: change.
  • planner/search/installation.mjs, host/obc-data-steps/src/planner.rs, host/obc-data-steps/tests/planner_build.rs: change: format 5; the bundle assertions.
  • specs/planner-offline.md, planner/service/README.md: change.

Done when

  • New tests:
    • planner/router-build/tests/cells.rs, an_area_holds_the_cells_of_each_snap_window_in_it: on Route package format 10: the contract, the writer and the installer #3301's test graph with three download cells, for a grid of points inside an area that touches two cells, among them points on its edges and corners, each download cell whose box meets snap::window(point, REACH_M) is in area_cells; a cell whose box starts 50 microdegrees beyond the grown edge is not.
    • native.rs, areas_of_one_package_share_an_install_and_the_last_one_takes_it, through the C functions: plan and install area A, then A and B; a route in A answers 200 with package the directory hash; with a callback that stops at once, the install returns false, and the next call completes; after an install of A alone, a route with a point only in B answers 422 missing_region, and B's cell files are gone; an install with no area deletes the folder.
    • OfflineMapsTests.swift, mapsOfOnePackageInstallTheirUnionAndTheLastRemovalDeletesIt: a fake OfflineRouting records the areas of each call; two maps of one package give the union, the removal of one gives the other area, the removal of the last gives an empty list; maps.json lists a map only after its install call; no routing object stays in objects/ after an install.
    • The Python tests change only where the routing selection is gone, and test_a_selection_carries_the_routing_directory_and_no_routing_pack checks the two files, routing_package and the absent routing/blocks.json.
  • obc test -p obc-data-steps in CI (it needs GEOS). obc test -p planner-router, -p planner-router-build, -p planner-service, and obc exec cargo clippy -p <crate> --all-targets -- -D warnings for each, for planner-service also with --no-default-features. cargo fmt --all. obc test affected --base origin/routing-overlay (it runs python.repository-tools). obc suites check, obc prose --check, python3 docs/build_docs.py --check-links.
  • swift.obckit-host and the Debug and Release simulator builds run only in CI, on macOS; the pull request says that the builder did not run them locally, and shows PlannerNative.h beside the #[no_mangle] block of native.rs.
  • The pull request records, on the owner's PC under nice, for the Central Switzerland target after obc data update local on the branch: for each zoom-9 tile of the target and for the whole bounds, the own roads, the download cells and the download level, and the routing bytes downloaded and installed with format 9 (the bundle of the branch base) and with format 10 (a plan with an empty target, from a program that is not committed), beside the table above. When a one-tile area installs more than 3 times the bytes of today, the pull request says so to the owner; Proof on Germany: the bake, the level sizes, the download level, the old engine against the new one #3310 sets the download level.
  • Owner, on the iPhone 13 with a Debug build and -OBCPlannerCatalog of a local follower on the branch:
    1. Remove the offline maps of the earlier build first: maps.json has a new field.
    2. Download two overlapping areas; the sizes match the pull request. In flight mode, route in each area, with alternatives near an edge; remove one area and route in the other again; the storage of the app in Settings shrinks.
    3. Stop the app during "Installing routes…" and start it again: the download goes on, and the other area still routes.
    4. Record the memory footprint (Xcode memory report) before and during the longest offline route of the larger area. The footprint grows by less than 100 MB (a design figure: the labels of a worker take less than 10 MB, the epic, and the search limit is 64 MiB, Route engine: the overlay search with the cost tables, and unpacking of the chosen legs only #3305).
  • The pull request has Requirements: (the offline-map and route requirements that obc req lists) and says that public docs changed, in a docs: commit.

Out of scope

Built differently

Pull request #3472 built this design, into routing-overlay, with these changes:

  • The routing installs before the release folder is linked (Plan step 4 said after). The install reads the objects from objects/. The link of releases/ID/ and the write of maps.json then follow with no suspension between them, so a stopped or killed install leaves no release folder. The map is still visible only after its routing is complete.
  • A discard of a pending download cleans the routing. It installs the areas of the listed maps of that package again, from the copies of package.json and directory.bin in the install folder. With no listed map of the package it deletes the folder.
  • The download of the routing objects is by SHA-256, each once. Two parts with equal bytes are one object. The space estimate counts each part, and for a new package the copies of package.json and directory.bin.
  • A removal completes when the routing of the other maps of the package cannot be installed again because an object is missing (the app was killed in the middle of an earlier removal). The folder stays for those maps. The full repair needs a download in the removal; it is not built.
  • install::area_cells returns a Result. The fetch of the installer returns a reader of the new trait install::Object, so the phone counts the progress; install::remove deletes a folder under its lock. An install that the lock of another process blocks answers 503 busy. The plan has one entry for each missing part file. The stop flag of the Swift installer is an OSAllocatedUnfairLock<Bool>, because the app targets iOS 17. OfflineFile is public and has an optional part_bytes.
  • The preparation of a download fetches package.json and directory.bin on any network, before the user allows mobile data, as the Plan says. The contract states it. For a world package the directory has about 3 to 19.5 MB; the owner decides whether this fetch waits for the consent.
  • Measured on the Switzerland bake of routing-overlay at 67320e4 (download level 4, 18 download cells of about 430,000 roads), a plan with an empty target, download and installed bytes: the Uri tile 166 and 276 MB, the Valais tile 151 and 248 MB, the Zurich tile 297 and 528 MB, a block of 3 × 3 tiles in the Alps 335 and 581 MB, the whole bounds 593 and 1,055 MB. A one-tile area installs more than 3 times the bytes of format 9 (27 to 85 MB in the Alps, 170 to 250 MB around a city, the figures of this issue). A tile takes 4 to 10 download cells at level 4, so the download level of Proof on Germany: the bake, the level sizes, the download level, the old engine against the new one #3310 decides this: on Switzerland, the table of Proof on Germany: the bake, the level sizes, the download level, the old engine against the new one #3310 gives a median overhead of 3.5 at level 3 against 12.8 at level 4.
  • Not run: the Swift suites and the simulator builds on a local machine (CI on macOS ran them). The comparison with format 9 on the Central Switzerland target and the four iPhone steps are with the owner; the commands are under "Owner steps" in the pull request.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions