Skip to content

Snow A. Bake: planner/snow writes snow, snow-levels and snow-seasons, and the TrailView snow bake goes #3129

Closed

Description

@timohueser

Status: approved
Part of #3125.

TL;DR

Goal: The step planner/snow writes the three snow archives snow, snow-levels and snow-seasons of a region, and a release has no TrailView snow bands.
Problem: A release has 46 TrailView snow band archives from a step of their own, and the planner reads per-year tiles of 18 bytes per pixel. #3125 has the measurements.
Decision: join in tools/planner_snow.py walks the per-year tiles of each block once. For each tile, it writes the snow and snow-levels tile of the runs and the 16 snow-seasons tiles of the worst case, so it holds only one path of the walk in memory. The Rust plan gives planner/snow three outputs with one grid each, and the step planner/trailview-snow, its layer and its Python bake go. The release lists, the release check, the data UI card, the TrailView test box and the four specs follow.
Rejected: see #3125
Systems: planner-map-baking, trailview, data-engine, device-map-builder
Contracts: specs/planner-snow-tiles.md, specs/trailview-tiles.md, specs/obc-data.md, specs/planner-release.md
Size: L (4 systems, a breaking change of the snow formats)

Facts

  • join keeps one tile per block in memory. In tools/planner_snow.py, join runs join_block for each zoom 6 block in a process pool. join_block makes each per-year tile of the block that meets the bounds with season_tile, and writes it gzip-compressed into a spill file of its own. Only the zoom 6 tile of each block stays in memory. above then blends zoom 0 to 5 from those tiles with parent. write_archive always writes tile type unknown with gzip.
  • A prepared file holds every ancestor of its tiles down to zoom 6. "Zoom levels" of specs/planner-snow-tiles.md: a season has a tile of zoom 6 to 12 where it has data or a tile below it. season_tile gives each season its own tile, else for zoom 9 to 12 the part of its nearest ancestor of zoom 8 or more.
  • The TrailView bake already builds a 512 × 512 tile from four children. In tools/trailview/snow.py, encode puts the four children north-west first. A missing child takes its part of the parent tile (fill), each pixel twice in each direction, but only when the parent is of zoom 8 or more. parents gives a parent only when at least one child exists. day_step(band) gives the day of a band.
  • tiles.lossless writes two bytes per pixel as red and green, with blue 0. See tools/trailview/tiles.py. tools/planner_trailview_canopy.py already imports the module tiles, and its step runs in the uv group planner-snow, which includes Pillow through planner-maps (pyproject.toml).
  • A map grid reads one archive. step of tools/planner_grid_maps.py refuses a step with more than one file. In planner.rs, snow is in the maps list, so map_grid makes planner/snow/grid from all outputs of the step. The four grids of trailview-colour each read one file with layer_files and name the step in their step option.
  • The bands are named in five places. PLANNER_TRAILVIEW_SNOW, SNOW_BANDS, optional, the check "the layer trailview-snow reads snow" and the test trailview_snow_reads_the_snow_archive_and_each_band_has_a_grid in planner.rs. pointer in catalog.rs. DATA_LAYERS in tools/planner_runtime.py. archive in tools/planner_verify.py, which also expects a snow tile of type unknown with seasons × 131,072 bytes. planner_grid_index.py refuses a kind of DATA_LAYERS whose metadata lacks the key that DATA_LAYERS names.
  • The snow card is the only card with with. In contents.ts, only the snow card sets with, and only layerFlags reads it.
  • The TrailView test box bakes the bands. box.sh runs tools.planner_snow box into snow.pmtiles, then tools.trailview.snow into snow-days/, and lists the 46 band names in the page config.
  • The prepared source links the per-year sections of the snow spec. "snow" of specs/obc-data.md defines its files by planner-snow-tiles.md#zoom-levels.

Plan

The rules of the three archives are The formats of #3125.

  1. tools/planner_snow.py: the bands and the runs.

    • BANDS (46) and day_step(band) move here from tools/trailview/snow.py. LEVELS names the five levels in the order of the levels table: any, quarter, half, three quarters, every.
    • runs(planes, level) -> (first, last): from per-year planes of shape (seasons, 2, N), the first and the last band of the run of level for each pixel, as two uint8 arrays of N. 253 when the level holds no band, 255 when no season has data.
    • run_image(first, last) -> bytes: one lossless WebP of 512 × 512 pixels, red the first band, green the last band, with tiles.lossless.
    • worst(children) -> planes: from four snow-seasons tiles {(dx, dy): planes (seasons, 2, 64, 64) or None}, the tile one zoom up by the worst-case rule. A quarter of None stays 255; the caller fills it.
    • snow_tile(children, own) -> (snow, levels) or None: the bodies of a snow and a snow-levels tile from the runs of the four per-year children, north-west first, None where the join lacks the child. A missing child of zoom 9 to 12 takes its part of own, the runs of the per-year tile itself, each pixel twice in each direction, as encode of tools/trailview/snow.py does. A quarter with no child and no fill is 255 in red and green (no data), not 0 as in encode: 0 and 0 would be a run of band 0. None when no child exists or no pixel has data.
  2. tools/planner_snow.py join_block: a depth-first walk over the per-year tiles of the block that meet the bounds, from zoom 6 to 12. For each tile, with its planes from season_tile:

    • It walks the four children first. Each child gives its runs of the five levels and its 16 snow-seasons tiles of zoom z + 3.
    • It writes the snow and snow-levels tile z/x/y from snow_tile.
    • It makes the 16 snow-seasons tiles of zoom z + 2 over the tile. At zoom 12, each is its 64 × 64 part of the planes. Above, each is worst of the four tiles one zoom deeper. A tile one zoom deeper that is not written to snow-seasons, because the walk lacks it or because it has no data, is missing: its quarter takes its part of the planes. It writes each that meets the bounds and has data, gzip-compressed.
    • It returns its runs and its 16 tiles to its parent.

    The bodies of the three archives go into the spill file of the block. join_block returns {archive: {tile id: (offset, length)}}, the per-year zoom 6 tile as today, and whether it read a tile of zoom 9 or more. The walk holds at most the tiles of one path from zoom 6 to 12 and their children, so the memory of a worker does not grow with the block.

  3. tools/planner_snow.py join(files, bounds, seasons, output, credit): output is a folder.

    • above keeps one recursion for zoom 0 to 5 that does what the walk does, with the planes of parent as today. At zoom 5, the children are the zoom 6 tiles of the blocks, and their snow-seasons tiles of zoom 8 come back from the spill files.
    • write_archive gets the tile type: webp with no compression for snow and snow-levels, unknown with gzip for snow-seasons and for the season files of prepare.
    • join writes snow.pmtiles, snow-levels.pmtiles and snow-seasons.pmtiles with the metadata of the formats. source is copernicus-hr-wsi when a block read a tile of zoom 9 or more, as today. It refuses when snow has no tile, and returns {archive: (tiles, bytes)}.
  4. tools/planner_snow.py step, box, main and the module text: step writes the three archives into the step output and records {"snow": {"tiles", "bytes"}, "snow-levels": …, "snow-seasons": …} as its metrics. box --output DIR writes the three archives into DIR.

  5. Delete tools/trailview/snow.py and tools/trailview/tests/test_snow.py.

  6. host/obc-data-steps/src/planner.rs:

    • snow writes snow.pmtiles, snow-levels.pmtiles and snow-seasons.pmtiles. PLANNER_SNOW is whole("planner/snow", 2, …): format 2, because its readers need the new data, so the update review shows the warning of the web planner. Its revision goes back to the 1 of whole.
    • map_grid(source, kind, archive: Option<&str>, bounds): with an archive, the grid is <source>/<archive>/grid, reads only <archive>.pmtiles and sets the option step to the source. The on("snow") branch of Planner::steps adds the producer with Client::None and the grids planner/snow/<kind>/grid for the kinds snow, snow-levels and snow-seasons. snow leaves the maps list. The four trailview-colour grids in trailview use map_grid too, with the same names and options as today.
    • Delete PLANNER_TRAILVIEW_SNOW, SNOW_BANDS, trailview-snow in optional, the check "the layer trailview-snow reads snow", the on("trailview-snow") block, its arm in each_step_belongs_to_the_layer_of_its_producer and the test trailview_snow_reads_the_snow_archive_and_each_band_has_a_grid.
  7. host/obc-data-steps/src/planner/catalog.rs: DATA_LAYERS gets snow-levels and snow-seasons (11 names). pointer adds no band names, and the comment on DATA_LAYERS loses the bands.

  8. tools/planner_runtime.py: DATA_LAYERS maps snow, snow-levels and snow-seasons to snow_format. The band entries and their comment go.

  9. tools/planner_verify.py archive: snow and snow-levels expect tile type webp, and snow-seasons unknown. A snow tile is one image of 512 × 512 pixels. A snow-levels tile is four such images that fill the body. A snow-seasons tile has 2 × seasons × 4,096 bytes. A helper images(data) splits a body by the RIFF size of each image; the colour check uses it too. The band check goes.

  10. builder/web/src/lib/data/contents.ts: the snow card loses with. No card uses with after that, so the field with of Content and its part of layerFlags go. contents.test.ts loses trailview-snow from its flags.

  11. builder/web/tests/browser/trailview/box.sh: tools.planner_snow box writes the three archives into a temporary folder, and the script moves them into the ground folder. The cache check looks for snow-seasons.pmtiles, so a ground folder of an earlier bake makes the archives again, and the script deletes its snow-days/. The tools.trailview.snow run goes. The page config lists snow, snow-levels and snow-seasons instead of the 46 bands. harness.js and link.mjs serve the bands until Snow B. TrailView: the snow of a date from the run of snow #3127 changes them, so the box page shows no snow between this issue and Snow B. TrailView: the snow of a date from the run of snow #3127.

  12. specs/planner-snow-tiles.md gets the formats of Snow: one date-free snow layer for TrailView and the planner, and small per-year tiles for routes #3125. The per-year sections stay: "Seasons", "Tile body", "Byte values", "Zoom levels", "Sources" and "Forest", because the prepared source snow is defined by them. The sections "Archive" and "Metadata" become the three archives: "Bands", "Levels", "Runs", "snow and snow-levels", "snow-seasons", "Metadata" and "Readers". The readers are the planner snow layer and the TrailView ground stream.

  13. The other specs:

    • specs/trailview-tiles.md: "Snow days", tools/trailview/snow.py and the snow days in the version line go. The introduction says that TrailView reads the snow of a date from snow of the snow spec.
    • specs/obc-data.md: the row planner/snow names the three outputs. A tile of zoom 6 or more of snow and snow-levels, and of zoom 8 or more of snow-seasons, that lies inside the bounds has the same bytes in each region whose step reads its block. planner/snow/<kind>/grid reads one archive, as the kind snow, snow-levels or snow-seasons. The row planner/trailview-snow, its grid sentence, the part trailview-snow and "trailview-snow needs snow" go. The producer snow says that planner/snow makes the three archives.
    • specs/planner-release.md: maps/ holds snow.pmtiles, snow-levels.pmtiles and snow-seasons.pmtiles.
  14. Tests:

    • tools/tests/test_planner_snow.py: runs gives the run of a pixel for each level, the earliest of two runs of equal length, 253 and 255. A snow-levels tile holds four images in the order any, quarter, three quarters, every. worst takes the worst case of four pixels. In JoinTest, a zoom 14 snow-seasons tile holds the bytes of its part of the per-year tile, a zoom 13 tile over a zoom 12 per-year tile with two parts of no data has two quarters from worst and two from the per-year tile of zoom 11, and where the join lacks a per-year tile of zoom 11, the zoom 12 tile over it takes all four quarters from the per-year tile of zoom 10. JoinTest checks each archive that join wrote with planner_verify.archive. The fixtures hold each ancestor of their tiles down to zoom 6, as a prepared file does. StepTest checks the output folder and the metrics.
    • tools/tests/test_planner_grid_maps.py: test_trailview_archives_verify_as_their_raster_tiles has a snow tile of 512 × 512 that passes and one of 256 × 256 that fails, and a snow-levels tile of four images, instead of trailview-snow-04.
    • planner.rs: a new test snow_has_a_grid_for_each_archive checks that each of the three grids reads its archive as its kind.

Files

  • tools/planner_snow.py: change: the runs, the worst case, the walk of join_block, and the three archives of join, step and box.
  • tools/trailview/snow.py, tools/trailview/tests/test_snow.py: delete.
  • host/obc-data-steps/src/planner.rs: change: the three outputs and grids of planner/snow, map_grid for one archive of a step, no planner/trailview-snow.
  • host/obc-data-steps/src/planner/catalog.rs: change: the three snow layers, no bands.
  • tools/planner_runtime.py: change: the three snow layers with snow_format, no bands.
  • tools/planner_verify.py: change: the checks of the three snow kinds.
  • builder/web/src/lib/data/contents.ts, builder/web/src/lib/data/contents.test.ts: change: no trailview-snow, no with.
  • builder/web/tests/browser/trailview/box.sh: change: the three snow archives, no bands.
  • specs/planner-snow-tiles.md, specs/trailview-tiles.md, specs/obc-data.md, specs/planner-release.md: change: the formats of Snow: one date-free snow layer for TrailView and the planner, and small per-year tiles for routes #3125.
  • tools/tests/test_planner_snow.py, tools/tests/test_planner_grid_maps.py: change: the tests above.

Done when

  • obc test -p obc-data-steps passes.
  • obc exec cargo clippy -p obc-data-steps --all-targets -- -D warnings is clean, and cargo fmt --all makes no change.
  • obc test affected --base origin/develop passes. It selects python.repository-tools (test_planner_snow.py, test_planner_grid_maps.py), python.trailview and web.builder-vitest (contents.test.ts).
  • obc suites check passes, because a test file goes.
  • On a machine with the CDSE keys, builder/web/tests/browser/trailview/box.sh writes snow.pmtiles, snow-levels.pmtiles and snow-seasons.pmtiles into the ground folder of the box.
  • Depends on obc data: the prepared source snow (format 1), and planner/snow bakes from it #2937.

Generated by Claude Code

Activity

  1. timohueser commented on Nov 6, 2026

    @timohueser
    OwnerAuthor

    The walk of join_block: algorithm and invariants

    This is the algorithm that I implement for step 1 to 3 of the Plan, with the rules of "The formats" of #3125.

    Words. T = z/x/y is a per-year tile that the walk visits. present is the set of tiles of the season files of the block that meet the bounds. P(T) is the planes (seasons, 2, 256, 256) of T from season_tile. R(T) is runs(P(T)), shape (5, 2, 256, 256): the first and the last band of the run of each level. Q(T) is the 16 snow-seasons tiles of zoom z + 2 over T: {(i, j): planes (seasons, 2, 64, 64) or None}, where (i, j) is the column and the row in T.

    visit(T), from the zoom 6 tile of the block, depth first:

    1. For each child C of T (zoom z + 1 ≤ 12), north-west first: visit(C) when C is in present, else None. A prepared file holds every ancestor of its tiles down to zoom 6, so the walk from the zoom 6 tile reaches each tile of present.
    2. Then read P(T) and compute R(T).
    3. snow_tile({C: R(C) or None}, own) gives the snow and snow-levels tile z/x/y. own is R(T) when the children are of zoom 9 to 12 (z ≥ 8), else None. At z = 12, T has no children, so there is no tile.
    4. Q(T): tile (i, j) covers the pixels [64 j, 64 j + 64) × [64 i, 64 i + 64) of P(T). Its four tiles one zoom deeper are all in one child, C = (i ÷ 2, j ÷ 2): the tiles (2 (i mod 2) + dx, 2 (j mod 2) + dy) of Q(C). The tile is worst of these four. A quarter whose deeper tile is None takes the same 32 × 32 pixels of P(T), at the same resolution. At z = 12, or when C is missing, all four quarters come from P(T): the tile is the 64 × 64 part of P(T).
    5. Write each tile of Q(T) that meets the bounds and has a pixel that is not 255, gzip-compressed. Each other tile becomes None in Q(T): for the parent, it is a missing tile.
    6. Return R(T) and Q(T).

    above (zoom 0 to 5) does steps 3 to 6 with P(T) = parent of the children's planes. At zoom 6, the children are the block tops: join_block returns P and R of its zoom 6 tile, and above reads their zoom 8 snow-seasons tiles back from the spill files. The zooms align: per-year zoom 0 to 12 give snow zoom 0 to 11 (one coarser) and snow-seasons zoom 2 to 14 (two finer).

    Memory bound. A level of the walk holds the results of at most 4 children: R (640 KiB) and Q (16 × seasons × 8 KiB, 1.25 MiB for 10 seasons), and the planes of one tile while it makes it (seasons × 128 KiB). For 10 seasons, that is about 8 MiB per level and 7 levels from zoom 6 to 12: less than 60 MiB per worker. The bodies go into the spill file at once. So the memory of a worker does not grow with the block.

    runs, for each pixel. k = the seasons that are not 255. c(b) = the seasons with snow on the day of band b, day = min(4 b + 2, 182): onset ≤ day ≤ melt-out, or 254. A scan over the 46 bands keeps the longest run; a run replaces the best one only when it is strictly longer, so the earliest run wins on a tie.

    • k = 0: 255 in both bytes, for each level.
    • k ≥ 1 and the level holds no band: 253 in both bytes.
    • Else the first and the last band of the run.

    worst, for each season and each 2 × 2 group of pixels one zoom deeper. In this order:

    • one of them is 254: 254 in both planes;
    • else one of them is dated: the earliest dated onset and the latest dated melt-out;
    • else one of them is 253: 253 (all those with data are 253);
    • else 255.

    A quarter whose child is None stays 255; the caller fills it from P(T).

    Fill rules.

    • snow and snow-levels: a missing child of zoom 9 to 12 takes its part of R(T), each pixel twice in each direction. A missing child of zoom 8 or less is 255 in red and green (no data). Without a child, or when no pixel of the half level has data, there is no tile.
    • snow-seasons: a deeper tile that the walk lacks, that has no data, or that does not meet the bounds is missing, and its quarter takes its part of P(T). A tile outside the bounds lies wholly outside them, so the pixels inside the bounds do not depend on the bounds.

    Two small deviations from the Plan.

    • runs(planes) gives all five levels at once, shape (5, 2, …), instead of runs(planes, level). The walk always needs the five, and k and c(b) are the same for each level, so one scan makes them all.
    • join_block returns the runs of its zoom 6 tile with its planes, so above does not compute them again in the main process.

    Generated by Claude Code

  2. timohueser commented on Nov 6, 2026

    @timohueser
    OwnerAuthor

    Deviations from the Plan in the pull request

    1. runs(planes) gives all five levels at once, shape (5, 2, …), not runs(planes, level). k and c(b) are the same for each level, and the walk always needs the five, so one scan over the 46 bands makes them all.
    2. join_block returns the planes and the runs of its zoom 6 tile. So above does not compute the runs of each block top again in the main process.
    3. The box page config lists only snow, not snow, snow-levels and snow-seasons (step 11). Snow B. TrailView: the snow of a date from the run of snow #3127 merged into snow-formats first: its harness.js and link.mjs serve and name only snow, the one archive that TrailView reads. The two other names would point at files that nothing serves. box.sh still writes all three archives into the ground folder.
    4. The precedence of worst. The epic lists the cases as "the earliest onset and the latest melt-out of those that are dated; 254 when one of them is 254; 253 …; 255 …". When a 2 × 2 group has a dated pixel and a 254 pixel, two cases apply. I take 254 first, because the rule is the worst case and 254 has no snow-free day. The spec says it so: "by the first rule that applies: 254 when one of them is 254; the earliest onset and the latest melt-out of those that are dated; 253 when one of them is 253; else 255".
    5. A snow-seasons tile outside the bounds is missing for its parent. The walk does not write it, so the parent quarter over it takes its part of the per-year tile. That quarter lies wholly outside the bounds, so the bytes inside the bounds do not change with the bounds.
    6. specs/README.md also changes: its row of planner-snow-tiles.md names the three archives.

    Generated by Claude Code

  3. timohueser commented on Nov 6, 2026

    @timohueser
    OwnerAuthor

    #3174 merged into the integration branch snow-formats, after an adversarial review and green CI. The review found no bug: a brute-force probe matched runs and worst on random planes, and every tile of a real join (357 snow-seasons tiles, 15 snow and snow-levels tiles) against an independent reference. Its two small gaps are fixed in aca1403: a byte check of the tiles above the blocks, and stricter planner_verify checks of the three kinds. This issue closes when snow-formats merges into develop with the re-bake of #3125. Open step of Done when: box.sh with the CDSE keys.


    Generated by Claude Code

  4. timohueser commented on Nov 6, 2026

    @timohueser
    OwnerAuthor

    Done: #3174 merged into snow-formats, and #3175 merged it into develop (3469850).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions