Skip to content

Larger map packs: one pack rule with a 15 MiB limit #3069

Closed
3 of 3 issues completed
Closed

Description

@timohueser

Status: approved
Part of workstream 1 of #3000.

TL;DR

Goal: The planner stores its map tiles in packs of at most 15 MiB, on one global grid, with far fewer R2 objects. On the production release (Central Switzerland) the map objects go from 6,546 to 1,219 (5×). When workstream 2 merges the 46 TrailView snow layers, they go to 327 (20×).

Problem: A pack (one PMTiles file in R2 that holds many tiles) covers one zoom 11 square today, and each tile at zoom 10 or lower has a pack of its own. At the coverage of #3000 that is 9.5–36M objects (estimated). R2 charges for each upload, so a full publish costs $38–158 and takes 13–50 h.

Decision: One pack rule for every map layer.

  • Levels. Each tile zoom has a base level and a level range: zooms 0 to 6 have base level 0, zooms 7 and 8 have base level 7, zoom 9 and deeper have base level 9. A pack starts at the base level and holds all tiles of its square in its level range.
  • The limit. When the PMTiles file of a pack is larger than 15 MiB, the pack keeps only its one tile at its own zoom. Its four squares one zoom deeper get packs for the deeper tiles of the range, by the same rule. A pack with no tiles gets no file. A pack that holds one tile cannot split; if it is larger than 15 MiB, the bake fails. Packs are stored without compression, because readers read them by byte range, so the file size is the stored size.
  • Accepted cost. The z13 terrain of TrailView is read straight from maps.. A cold z13 tile at the end of a full pack waits about 1 s, against about 0.4 s for today's z11 packs of up to about 4 MB. The owner chose one limit for all layers.
  • The index. The pointers of all packs of one base square are in one block index: the block of its base square, at zoom max(base - 2, 0). A block index holds 4 × 4 base squares.
  • The readers. The pack of a tile is the deepest pack that exists whose square holds the tile, at a zoom from the tile zoom back to the base level of the tile zoom. The tile Worker and the TrailView terrain reader look the names up in the one block index. The iOS offline reader looks for the files. A cold tile costs one block index read, as today. The reads of the pack header and directories are measured at the re-bake. A tile that is absent in a split square falls back to the parent pack, which costs one more header read.

Rejected:

  • Packs per z9 square with no limit: packs up to 56.9 MB, so a cold tile read straight from maps. takes about 3 s (prototype below).
  • Packs per z10 square everywhere: about 2× more objects than z9 with the limit.
  • One fixed pack zoom per layer: one dense square sets the zoom of the whole layer. The z13 terrain packs are 14.9 MB at z10 in the Alps already, so one denser square moves the layer to z11 on every continent, about 770k objects for that layer (estimated).
  • A split mark on each split pack, in the block index of that pack's own zoom: a cold tile reads up to five block indexes one after another, and a split pack with no tiles of its own zoom (all of the z13 terrain) has no entry for the mark.
  • The TrailView snow shape: workstream 2 of EPIC — Scale readiness: 100 concurrent users, long routes, from one region to the world #3000 decides it. This rule applies to the snow layers that exist.
  • Which layers bake once per continent, and how a region release points to them: a later issue of Prepared sources: pin the formats, then build climate, snow and colour #2921 decides it. This issue defines only the pack layout.

Systems: planner-map-baking, planner-tile-server, trailview, companion-ios, obc-data
Contracts: specs/planner-release.md, specs/planner-offline.md, specs/obc-data.md, specs/trailview-tiles.md
Size: L (more than 3 systems, a breaking change to the pack layout)

Facts

  • map_tiles in tools/planner_grid_maps.py cuts every layer into packs. It names a pack level-x-y with level = min(z, MAP_ZOOM) and MAP_ZOOM = 11. It first writes the tiles of each pack to a staging file, then writes one PMTiles file from each staging file. The PMTiles writer needs at least one tile. The z13 terrain tiles go to packs under 13/, because the tile Worker and an offline area stop at zoom 12.
  • pack_file of tools/planner_offline.py stores .pmtiles files without compression (COMPRESSED).
  • There are two grid documents, each with its own format number:
    • The release grid: grid in release.json and in the catalog, today {format: 2, zoom: 9, map_zoom: 11}. tools/planner_grid_index.py writes it. tools/planner_verify.py and verify of host/obc-data-steps/src/planner/catalog.rs accept only that value. The offline block of the catalog (tools/planner_downloads.py) has {format: 2, zoom, map_zoom}.
    • The public grid: public/grid.json, today {format: 3, map_zoom}. public_metadata of tools/planner_runtime.py writes it, with the block indexes. The tile Worker and TrailView read it.
  • A block index public/DIR/B-BX-BY.json maps each pack name Z-X-Y.pmtiles to its pointer (stored object, size, hash). Today B = max(Z - 4, 0). The same rule serves the route catalog files under routes/tiles/. An index holds at most 64 KiB, the limit of the Worker.
  • The tile Worker (planner/tiles/src/grid.mjs, worker.mjs) reads the public grid, makes the pack name with packName(name, tile, map_zoom), reads the pointer from the block index, then reads the tile by byte range through the R2 binding. It reads public grid formats 2 and 3 (GRIDS), and keeps up to 256 block indexes in memory. deploy_cli.rs (GRID = 3) refuses a release while the Worker does not report its public grid format.
  • TrailView reads all layers through the Worker, except the z13 terrain. builder/web/src/lib/trailview/stream/terrain.ts reads the public grid, hard-codes the pack 11-${x>>2}-${y>>2} and the block index 7-${x>>6}-${y>>6}, and reads the pack by byte range straight from maps.openbikecomputer.com.
  • The iOS offline reader (companion-ios/OBCCompanion/OfflineTilesProtocol.swift) makes the pack path from min(z, map_zoom) and the local files. OfflinePlanner.swift accepts only format == 2, zoom == 9, map_zoom == 11. It maps a whole pack into memory and accepts a slice of up to 32 MiB.
  • Offline selection (tools/planner_downloads.py, planner_map_archive.selected) keeps each map pack whose square at its own zoom meets the requested area.
  • The prototype below measured that a cold range read through maps. waits until Cloudflare has read the pack from R2 up to that range, at 17–25 MB/s. So a tile at the end of a 15 MiB pack waits about 1 s.
  • tools/prototype_pack_rule.py applies this rule to the file sizes of the production release (estimated: it uses today's packs as the smallest units, so it cannot split below zoom 11): 878 packs and 341 block indexes, the largest pack 14.2 MiB, the largest block index 2.6 KiB (20 entries). Today: 5,942 packs and 604 block indexes.

Plan

This is an epic with three sub-issues. Each sub-issue is one pull request.

B and C merge first. They read the formats of today's release and the new ones. A merges after B and C, so that each bake from develop has readers for its formats. A release with the new formats is published only when B is deployed and C is in the app, and with workstreams 2 and 3 of #3000 in one re-bake. A rollback activates the previous release: B reads its public grid format 3, and C reads its offline block format 2.

Still to do

The code of A, B and C is in develop (#3088, #3077, #3076, and the spec fix #3093). The epic closes when these items are done:

  1. Deploy the tile Worker of B: obc data deploy tiles --dry-run, then the command that it shows. Deploy also the web build that holds the TrailView reader of B. Do these before a release with the new formats is published.
  2. Ship an iOS app build with the reader of C.
  3. Run the Central Switzerland re-bake of "Done when" to the local target, and record its measurements in this issue: object count, largest pack, Worker R2 reads for a cold tile, planner first-tile time, TrailView open time, offline bundle bytes, OSM refresh upload bytes and the bytes that two releases share. On the shared PC, plan the bake with the owner first.
  4. The owner opens the planner and TrailView on the re-bake.
  5. Publish the new formats only in the one re-bake with workstreams 2 and 3 of EPIC — Scale readiness: 100 concurrent users, long routes, from one region to the world #3000.
  6. Then set Status: done and close this issue.

The branch prototype/cold-range-read holds the prototype scripts that this issue links. Keep it until this issue closes.

A. Bake

  1. tools/planner_grid_maps.py: replace MAP_ZOOM with LEVELS = (0, 7, 9) and PACK_BYTES = 15 * 1024 * 1024. In map_tiles, stage each tile under the pack of its base level. Write the PMTiles file of each staging file. When a file is larger than PACK_BYTES, delete it, move each tile deeper than the pack's zoom to the staging file of its child square one zoom deeper, write the pack again from its one tile at its own zoom (no file when there is none), and repeat for the children. A pack that holds one tile and is larger than PACK_BYTES fails the step. The z13 terrain packs stay under 13/, with base level 9. index.json loses map_zoom.
  2. tools/planner_grid_index.py: write release grid format 3, {format: 3, zoom: 9}, in release.json and in the catalog.
  3. tools/planner_runtime.py public_metadata: write public grid format 4, {"format": 4}, and set GRID_FORMAT = 4. Put the pointer of a map pack under maps/tiles/ into the block index of its base square: B = max(base - 2, 0), where base is the base level of the pack's zoom. Files under routes/tiles/ keep B = max(Z - 4, 0).
  4. tools/planner_verify.py: accept release grid format 3, {format: 3, zoom: 9}, and refuse a map pack whose bytes is larger than PACK_BYTES.
  5. host/obc-data-steps/src/planner/catalog.rs verify: accept {"format": 3, "zoom": 9}.
  6. tools/planner_downloads.py: write offline block format 3, {"format": 3, "zoom": 9}. Selection keeps its rule.
  7. host/obc-data/src/cli/deploy_cli.rs: GRID = 4.
  8. Specs: in specs/planner-release.md, specs/planner-offline.md, specs/obc-data.md and specs/trailview-tiles.md, replace "the ancestor at min(tile_zoom, 11)" and "per zoom 11 parent" with the levels, the 15 MiB limit and its reason (a tile at the end of a cold 15 MiB pack waits about 1 s), the block index of the base square, the reader rule, release grid format 3 and public grid format 4.
  9. Tests: tools/tests/test_planner_grid_maps.py: a square over the limit gives a parent pack with the tiles of its zoom and child packs with the deeper tiles; a z13 terrain square over the limit gives no parent file; a square under the limit gives one pack. tools/tests/test_planner_runtime.py: the pointers of a split square are in the block index of its base square. tools/tests/test_planner_downloads.py: the new offline block.

B. Tile Worker and TrailView

  1. planner/tiles/src/grid.mjs: GRIDS = [3, 4]. gridConfig accepts public grid format 4, {format: 4}, without map_zoom. Add packPointer(bucket, prefix, name, tile): for public grid format 4, read the block index of the tile's base square once, then check the names from the tile zoom back to the base level and return the first pointer, or null. For public grid format 3, keep today's packName path.
  2. planner/tiles/src/worker.mjs: use packPointer.
  3. builder/web/src/lib/trailview/stream/terrain.ts: when the public grid it reads has format 4, find the z13 pack with the same rule in the block index public/maps/tiles/terrain/13/7-…json of the base square. Cache the result per base square. Public grid format 3 keeps today's path.
  4. Tests: planner/tiles/test/worker.test.mjs: a tile in a split square reads its child pack; a tile at the zoom of a split pack reads the parent; a cold tile reads one block index; a warm read is still one R2 read per tile. terrain.test.ts: the same rule for public grid format 4.

C. iOS offline reader

  1. companion-ios/OBCCompanion/OfflinePlanner.swift validate: accept offline block format 3, format == 3, zoom == 9 without map_zoom, and keep today's offline block format 2, format == 2, zoom == 9, map_zoom == 11, so that the bundles of today's release still open.
  2. companion-ios/OBCCompanion/OfflineTilesProtocol.swift: for offline block format 2, keep today's min(z, map_zoom) path. For offline block format 3, for zooms from the tile zoom back to the base level, build maps/tiles/<kind>/<k>-<x >> (z-k)>-<y >> (z-k)>.pmtiles and use the first file that exists.
  3. Tests: a bundle of offline block format 3 with a split square serves tiles from the parent and from the child packs; a bundle of format 2 still serves its tiles.

Files

  • tools/planner_grid_maps.py: change: levels, limit, split.
  • tools/planner_grid_index.py: change: release grid format 3.
  • tools/planner_runtime.py: change: public grid format 4, block index of the base square.
  • tools/planner_verify.py: change: release grid, pack size check.
  • tools/planner_downloads.py: change: offline block format 3.
  • host/obc-data-steps/src/planner/catalog.rs: change: accept the release grid.
  • host/obc-data/src/cli/deploy_cli.rs: change: GRID = 4.
  • planner/tiles/src/grid.mjs, planner/tiles/src/worker.mjs: change: format 4, packPointer.
  • builder/web/src/lib/trailview/stream/terrain.ts: change: format 4 rule.
  • companion-ios/OBCCompanion/OfflinePlanner.swift, companion-ios/OBCCompanion/OfflineTilesProtocol.swift: change: offline block, rule by file.
  • specs/planner-release.md, specs/planner-offline.md, specs/obc-data.md, specs/trailview-tiles.md: change: the pack rule.
  • tools/tests/test_planner_grid_maps.py, tools/tests/test_planner_runtime.py, tools/tests/test_planner_downloads.py, planner/tiles/test/worker.test.mjs, builder/web/src/lib/trailview/stream/terrain.test.ts, the iOS offline tile tests: change.

Done when

  • Each pull request passes the suites that obc test affected --base origin/develop selects.
  • A Central Switzerland re-bake (to the local target) shows, against the production release: the object count, the largest pack (at most 15 MiB), the R2 reads of the Worker for a cold tile (index, pack header, directories and tile), the first-tile time of the planner and the TrailView open time. The same views show no visual change.
  • The same re-bake records the offline bundle bytes of a sample area (one z9 square, and a 30 km × 30 km area) against today, because an area now gets whole base packs.
  • The same re-bake records the bytes that one OSM refresh uploads and the bytes that two releases share. Workstream 1 of EPIC — Scale readiness: 100 concurrent users, long routes, from one region to the world #3000 lists them as measurements before the layout is final.
  • The owner opens the planner and TrailView on the re-bake.

Prototype: How long does a cold range read of a 50 MB pack take through maps.openbikecomputer.com?

Answer: About as long as a download of the pack up to the requested bytes. On a cold cache, Cloudflare reads the whole pack from R2 from the start, and it sends the bytes of a range only when that read reaches them. A tile near the end of a 50 MB pack arrives after 2–3 s. A tile near the start arrives after about 0.25 s. In a 1 MB pack, each tile arrives after about 0.3–0.4 s.

Evidence: tools/prototype_cold_range_read.py uploads one random pack of 1, 15 and 50 MB to test/ in obc-maps. It copies the pack to a new key for each test (a new key is in no cache), reads with curl, and deletes the folder. 3 runs per size. Time to the last byte, median (min–max), with curl on the VPS (France):

Read 1 MB 15 MB 50 MB
First 16 KB, cold 0.30 s (0.23–0.41) 0.36 s (0.23–0.61) 0.24 s (0.23–0.29)
Then 64 KB in the middle, same connection 0.09 s 0.45 s (0.18–0.56) 0.91 s (0.81–2.92)
Last 64 KB, cold 0.36 s (0.25–0.81) 0.70 s (0.60–0.77) 2.18 s (1.91–2.93)
5 parallel ranges over the pack, cold, slowest 0.31 s 0.83 s 2.70 s (2.54–3.42)
Whole pack, cold 0.61 s 1.19 s 2.93 s (2.83–3.36)
  • The first byte arrives after 0.15–0.4 s for each size: the headers come first, and the range bytes wait for the read from R2.
  • Cloudflare reads one object from R2 at about 17–25 MB/s.
  • Parallel ranges of one cold pack all wait for the same read. A second range during that read reports HIT, but it arrives only when the read reaches it.
  • From a home line in Germany (Telekom), the same pattern: last 64 KB of 50 MB cold 2.3 s (1.4–2.6), of 1 MB 0.49 s. Telekom sends each new connection to a different Cloudflare site (Frankfurt, Stockholm, Paris, Newark, Miami, Toronto, San José). Each site reads its own copy, and a far site adds 0.1–0.5 s. The VPS also reached Newark at times.
  • Not measured: a pack that is cold at the edge site but already in the upper tier of Tiered Cache. All test keys were cold in both tiers.

For the design:

  • Limit the size of a pack, not only its zoom. The worst cold tile costs about 0.25 s plus 1 s for each 20 MB in front of it. The z9 layout has packs up to 56.9 MB (about 3 s); the z10 layout has packs up to 14.9 MB (about 0.8 s). The z9 median of 1.47 MB is fine. A size cap that splits only the large z9 cells keeps the low object count.
  • Put the bytes that a client reads first at the start of a pack. PMTiles v3 already puts the header and the directories before the tile data, and it orders tiles by tile ID, so low zooms come first.
  • The tile Worker reads only the requested range through its R2 binding (planner/tiles/src/worker.mjs), so this cost does not apply to it. It applies to each read straight from maps.: the TrailView terrain packs today, and the "Later" item that skips the Worker.
  • One cold read is one R2 read of the whole pack. Egress is free, so the cost in money does not change.

Size cap (owner: z9 packs with a size cap). tools/prototype_pack_cap.py splits each z9 pack of the production release that is over the cap into its 4 z10 children, and a child over the cap into its z11 children:

Cap Map objects Largest pack Worst cold tile at 20 MB/s
none 196 56.9 MB 3.1 s
25 MB 232 24.7 MB 1.5 s
20 MB 253 19.6 MB 1.2 s
15 MB 273 14.9 MB 1.0 s
12 MB 351 11.9 MB 0.85 s
10 MB 408 9.9 MB 0.74 s
6 MB 508 6.0 MB 0.55 s
  • The packs over 15 MB are the TrailView terrain packs (terrain/13, up to 56.9 MB) and the two snow layers (up to 42 MB). The terrain packs are the ones that TrailView reads straight from maps..
  • 15 MB is the knee. A split divides a pack by about 4, and the dense z10 children are 10–15 MB. Under 15 MB they split again into z11 packs, so the object count rises fast: 273 at 15 MB, 408 at 10 MB.
  • 15 MB also keeps the worst cold tile near 1 s (0.85–1.1 s at the measured 17–25 MB/s).
  • This simulation merges the 46 TrailView snow bands into one layer and keeps one pack per layer for zooms under 7. It gives 196 objects for plain z9, not the 171 above, because the earlier simulation grouped some files differently. The largest packs (56.9 MB at z9, 14.9 MB at z10) are the same.

Branch: https://github.com/timohueser/OpenBikeComputer/tree/prototype/cold-range-read

Activity

  1. timohueser commented on Nov 20, 2026

    @timohueser
    OwnerAuthor

    Closed in the backlog cleanup: all 3 sub-issues are closed (#3076, #3077, #3088). Prod has the new pack rule: the tile Worker's /version gives grids:[3,4], and the active prod release.json has the release grid {format:3, zoom:9} and manifest format 2 (#3146). The iOS build with reader C waits for iOS work and is in #3741.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions