Repository navigation
Description
Status: approved
Part of #3125.
TL;DR
Goal: The step planner/snow writes the three snow archives snow, snow-levels and snow-seasons of a region, and a release has no TrailView snow bands.
Problem: A release has 46 TrailView snow band archives from a step of their own, and the planner reads per-year tiles of 18 bytes per pixel. #3125 has the measurements.
Decision: join in tools/planner_snow.py walks the per-year tiles of each block once. For each tile, it writes the snow and snow-levels tile of the runs and the 16 snow-seasons tiles of the worst case, so it holds only one path of the walk in memory. The Rust plan gives planner/snow three outputs with one grid each, and the step planner/trailview-snow, its layer and its Python bake go. The release lists, the release check, the data UI card, the TrailView test box and the four specs follow.
Rejected: see #3125
Systems: planner-map-baking, trailview, data-engine, device-map-builder
Contracts: specs/planner-snow-tiles.md, specs/trailview-tiles.md, specs/obc-data.md, specs/planner-release.md
Size: L (4 systems, a breaking change of the snow formats)
Facts
joinkeeps one tile per block in memory. Intools/planner_snow.py,joinrunsjoin_blockfor each zoom 6 block in a process pool.join_blockmakes each per-year tile of the block that meets the bounds withseason_tile, and writes it gzip-compressed into a spill file of its own. Only the zoom 6 tile of each block stays in memory.abovethen blends zoom 0 to 5 from those tiles withparent.write_archivealways writes tile typeunknownwith gzip.- A prepared file holds every ancestor of its tiles down to zoom 6. "Zoom levels" of
specs/planner-snow-tiles.md: a season has a tile of zoom 6 to 12 where it has data or a tile below it.season_tilegives each season its own tile, else for zoom 9 to 12 the part of its nearest ancestor of zoom 8 or more. - The TrailView bake already builds a 512 × 512 tile from four children. In
tools/trailview/snow.py,encodeputs the four children north-west first. A missing child takes its part of the parent tile (fill), each pixel twice in each direction, but only when the parent is of zoom 8 or more.parentsgives a parent only when at least one child exists.day_step(band)gives the day of a band. tiles.losslesswrites two bytes per pixel as red and green, with blue 0. Seetools/trailview/tiles.py.tools/planner_trailview_canopy.pyalready imports the moduletiles, and its step runs in the uv groupplanner-snow, which includes Pillow throughplanner-maps(pyproject.toml).- A map grid reads one archive.
stepoftools/planner_grid_maps.pyrefuses a step with more than one file. Inplanner.rs,snowis in themapslist, somap_gridmakesplanner/snow/gridfrom all outputs of the step. The four grids oftrailview-coloureach read one file withlayer_filesand name the step in theirstepoption. - The bands are named in five places.
PLANNER_TRAILVIEW_SNOW,SNOW_BANDS,optional, the check "the layer trailview-snow reads snow" and the testtrailview_snow_reads_the_snow_archive_and_each_band_has_a_gridinplanner.rs.pointerincatalog.rs.DATA_LAYERSintools/planner_runtime.py.archiveintools/planner_verify.py, which also expects asnowtile of typeunknownwithseasons× 131,072 bytes.planner_grid_index.pyrefuses a kind ofDATA_LAYERSwhose metadata lacks the key thatDATA_LAYERSnames. - The snow card is the only card with
with. Incontents.ts, only the snow card setswith, and onlylayerFlagsreads it. - The TrailView test box bakes the bands.
box.shrunstools.planner_snow boxintosnow.pmtiles, thentools.trailview.snowintosnow-days/, and lists the 46 band names in the page config. - The prepared source links the per-year sections of the snow spec. "
snow" ofspecs/obc-data.mddefines its files byplanner-snow-tiles.md#zoom-levels.
Plan
The rules of the three archives are The formats of #3125.
-
tools/planner_snow.py: the bands and the runs.BANDS(46) andday_step(band)move here fromtools/trailview/snow.py.LEVELSnames the five levels in the order of the levels table: any, quarter, half, three quarters, every.runs(planes, level) -> (first, last): from per-year planes of shape (seasons, 2, N), the first and the last band of the run oflevelfor each pixel, as twouint8arrays of N. 253 when the level holds no band, 255 when no season has data.run_image(first, last) -> bytes: one lossless WebP of 512 × 512 pixels, red the first band, green the last band, withtiles.lossless.worst(children) -> planes: from foursnow-seasonstiles {(dx, dy): planes (seasons, 2, 64, 64) or None}, the tile one zoom up by the worst-case rule. A quarter of None stays 255; the caller fills it.snow_tile(children, own) -> (snow, levels) or None: the bodies of asnowand asnow-levelstile from the runs of the four per-year children, north-west first, None where the join lacks the child. A missing child of zoom 9 to 12 takes its part ofown, the runs of the per-year tile itself, each pixel twice in each direction, asencodeoftools/trailview/snow.pydoes. A quarter with no child and no fill is 255 in red and green (no data), not 0 as inencode: 0 and 0 would be a run of band 0. None when no child exists or no pixel has data.
-
tools/planner_snow.pyjoin_block: a depth-first walk over the per-year tiles of the block that meet the bounds, from zoom 6 to 12. For each tile, with its planes fromseason_tile:- It walks the four children first. Each child gives its runs of the five levels and its 16
snow-seasonstiles of zoom z + 3. - It writes the
snowandsnow-levelstile z/x/y fromsnow_tile. - It makes the 16
snow-seasonstiles of zoom z + 2 over the tile. At zoom 12, each is its 64 × 64 part of the planes. Above, each isworstof the four tiles one zoom deeper. A tile one zoom deeper that is not written tosnow-seasons, because the walk lacks it or because it has no data, is missing: its quarter takes its part of the planes. It writes each that meets the bounds and has data, gzip-compressed. - It returns its runs and its 16 tiles to its parent.
The bodies of the three archives go into the spill file of the block.
join_blockreturns {archive: {tile id: (offset, length)}}, the per-year zoom 6 tile as today, and whether it read a tile of zoom 9 or more. The walk holds at most the tiles of one path from zoom 6 to 12 and their children, so the memory of a worker does not grow with the block. - It walks the four children first. Each child gives its runs of the five levels and its 16
-
tools/planner_snow.pyjoin(files, bounds, seasons, output, credit):outputis a folder.abovekeeps one recursion for zoom 0 to 5 that does what the walk does, with the planes ofparentas today. At zoom 5, the children are the zoom 6 tiles of the blocks, and theirsnow-seasonstiles of zoom 8 come back from the spill files.write_archivegets the tile type:webpwith no compression forsnowandsnow-levels,unknownwith gzip forsnow-seasonsand for the season files ofprepare.joinwritessnow.pmtiles,snow-levels.pmtilesandsnow-seasons.pmtileswith the metadata of the formats.sourceiscopernicus-hr-wsiwhen a block read a tile of zoom 9 or more, as today. It refuses whensnowhas no tile, and returns {archive: (tiles, bytes)}.
-
tools/planner_snow.pystep,box,mainand the module text:stepwrites the three archives into the step output and records{"snow": {"tiles", "bytes"}, "snow-levels": …, "snow-seasons": …}as its metrics.box --output DIRwrites the three archives into DIR. -
Delete
tools/trailview/snow.pyandtools/trailview/tests/test_snow.py. -
host/obc-data-steps/src/planner.rs:snowwritessnow.pmtiles,snow-levels.pmtilesandsnow-seasons.pmtiles.PLANNER_SNOWiswhole("planner/snow", 2, …): format 2, because its readers need the new data, so the update review shows the warning of the web planner. Its revision goes back to the 1 ofwhole.map_grid(source, kind, archive: Option<&str>, bounds): with an archive, the grid is<source>/<archive>/grid, reads only<archive>.pmtilesand sets the optionstepto the source. Theon("snow")branch ofPlanner::stepsadds the producer withClient::Noneand the gridsplanner/snow/<kind>/gridfor the kindssnow,snow-levelsandsnow-seasons.snowleaves themapslist. The fourtrailview-colourgrids intrailviewusemap_gridtoo, with the same names and options as today.- Delete
PLANNER_TRAILVIEW_SNOW,SNOW_BANDS,trailview-snowinoptional, the check "the layer trailview-snow reads snow", theon("trailview-snow")block, its arm ineach_step_belongs_to_the_layer_of_its_producerand the testtrailview_snow_reads_the_snow_archive_and_each_band_has_a_grid.
-
host/obc-data-steps/src/planner/catalog.rs:DATA_LAYERSgetssnow-levelsandsnow-seasons(11 names).pointeradds no band names, and the comment onDATA_LAYERSloses the bands. -
tools/planner_runtime.py:DATA_LAYERSmapssnow,snow-levelsandsnow-seasonstosnow_format. The band entries and their comment go. -
tools/planner_verify.pyarchive:snowandsnow-levelsexpect tile typewebp, andsnow-seasonsunknown. Asnowtile is one image of 512 × 512 pixels. Asnow-levelstile is four such images that fill the body. Asnow-seasonstile has 2 ×seasons× 4,096 bytes. A helperimages(data)splits a body by the RIFF size of each image; the colour check uses it too. The band check goes. -
builder/web/src/lib/data/contents.ts: the snow card loseswith. No card useswithafter that, so the fieldwithofContentand its part oflayerFlagsgo.contents.test.tslosestrailview-snowfrom its flags. -
builder/web/tests/browser/trailview/box.sh:tools.planner_snow boxwrites the three archives into a temporary folder, and the script moves them into the ground folder. The cache check looks forsnow-seasons.pmtiles, so a ground folder of an earlier bake makes the archives again, and the script deletes itssnow-days/. Thetools.trailview.snowrun goes. The page config listssnow,snow-levelsandsnow-seasonsinstead of the 46 bands.harness.jsandlink.mjsserve the bands until Snow B. TrailView: the snow of a date from the run of snow #3127 changes them, so the box page shows no snow between this issue and Snow B. TrailView: the snow of a date from the run of snow #3127. -
specs/planner-snow-tiles.mdgets the formats of Snow: one date-free snow layer for TrailView and the planner, and small per-year tiles for routes #3125. The per-year sections stay: "Seasons", "Tile body", "Byte values", "Zoom levels", "Sources" and "Forest", because the prepared sourcesnowis defined by them. The sections "Archive" and "Metadata" become the three archives: "Bands", "Levels", "Runs", "snowandsnow-levels", "snow-seasons", "Metadata" and "Readers". The readers are the planner snow layer and the TrailView ground stream. -
The other specs:
specs/trailview-tiles.md: "Snow days",tools/trailview/snow.pyand the snow days in the version line go. The introduction says that TrailView reads the snow of a date fromsnowof the snow spec.specs/obc-data.md: the rowplanner/snownames the three outputs. A tile of zoom 6 or more ofsnowandsnow-levels, and of zoom 8 or more ofsnow-seasons, that lies inside the bounds has the same bytes in each region whose step reads its block.planner/snow/<kind>/gridreads one archive, as the kindsnow,snow-levelsorsnow-seasons. The rowplanner/trailview-snow, its grid sentence, the parttrailview-snowand "trailview-snowneedssnow" go. The producersnowsays thatplanner/snowmakes the three archives.specs/planner-release.md:maps/holdssnow.pmtiles,snow-levels.pmtilesandsnow-seasons.pmtiles.
-
Tests:
tools/tests/test_planner_snow.py:runsgives the run of a pixel for each level, the earliest of two runs of equal length, 253 and 255. Asnow-levelstile holds four images in the order any, quarter, three quarters, every.worsttakes the worst case of four pixels. InJoinTest, a zoom 14snow-seasonstile holds the bytes of its part of the per-year tile, a zoom 13 tile over a zoom 12 per-year tile with two parts of no data has two quarters fromworstand two from the per-year tile of zoom 11, and where the join lacks a per-year tile of zoom 11, the zoom 12 tile over it takes all four quarters from the per-year tile of zoom 10.JoinTestchecks each archive thatjoinwrote withplanner_verify.archive. The fixtures hold each ancestor of their tiles down to zoom 6, as a prepared file does.StepTestchecks the output folder and the metrics.tools/tests/test_planner_grid_maps.py:test_trailview_archives_verify_as_their_raster_tileshas asnowtile of 512 × 512 that passes and one of 256 × 256 that fails, and asnow-levelstile of four images, instead oftrailview-snow-04.planner.rs: a new testsnow_has_a_grid_for_each_archivechecks that each of the three grids reads its archive as its kind.
Files
tools/planner_snow.py: change: the runs, the worst case, the walk ofjoin_block, and the three archives ofjoin,stepandbox.tools/trailview/snow.py,tools/trailview/tests/test_snow.py: delete.host/obc-data-steps/src/planner.rs: change: the three outputs and grids ofplanner/snow,map_gridfor one archive of a step, noplanner/trailview-snow.host/obc-data-steps/src/planner/catalog.rs: change: the three snow layers, no bands.tools/planner_runtime.py: change: the three snow layers withsnow_format, no bands.tools/planner_verify.py: change: the checks of the three snow kinds.builder/web/src/lib/data/contents.ts,builder/web/src/lib/data/contents.test.ts: change: notrailview-snow, nowith.builder/web/tests/browser/trailview/box.sh: change: the three snow archives, no bands.specs/planner-snow-tiles.md,specs/trailview-tiles.md,specs/obc-data.md,specs/planner-release.md: change: the formats of Snow: one date-free snow layer for TrailView and the planner, and small per-year tiles for routes #3125.tools/tests/test_planner_snow.py,tools/tests/test_planner_grid_maps.py: change: the tests above.
Done when
obc test -p obc-data-stepspasses.obc exec cargo clippy -p obc-data-steps --all-targets -- -D warningsis clean, andcargo fmt --allmakes no change.obc test affected --base origin/developpasses. It selectspython.repository-tools(test_planner_snow.py,test_planner_grid_maps.py),python.trailviewandweb.builder-vitest(contents.test.ts).obc suites checkpasses, because a test file goes.- On a machine with the CDSE keys,
builder/web/tests/browser/trailview/box.shwritessnow.pmtiles,snow-levels.pmtilesandsnow-seasons.pmtilesinto the ground folder of the box. - Depends on obc data: the prepared source snow (format 1), and planner/snow bakes from it #2937.
Generated by Claude Code
Activity
timohueser commentedon Nov 6, 2026
The walk of
join_block: algorithm and invariantsThis is the algorithm that I implement for step 1 to 3 of the Plan, with the rules of "The formats" of #3125.
Words. T = z/x/y is a per-year tile that the walk visits.
presentis the set of tiles of the season files of the block that meet the bounds. P(T) is the planes (seasons, 2, 256, 256) of T fromseason_tile. R(T) isruns(P(T)), shape (5, 2, 256, 256): the first and the last band of the run of each level. Q(T) is the 16snow-seasonstiles of zoom z + 2 over T: {(i, j): planes (seasons, 2, 64, 64) or None}, where (i, j) is the column and the row in T.visit(T), from the zoom 6 tile of the block, depth first:visit(C)when C is inpresent, else None. A prepared file holds every ancestor of its tiles down to zoom 6, so the walk from the zoom 6 tile reaches each tile ofpresent.snow_tile({C: R(C) or None}, own)gives thesnowandsnow-levelstile z/x/y.ownis R(T) when the children are of zoom 9 to 12 (z ≥ 8), else None. At z = 12, T has no children, so there is no tile.worstof these four. A quarter whose deeper tile is None takes the same 32 × 32 pixels of P(T), at the same resolution. At z = 12, or when C is missing, all four quarters come from P(T): the tile is the 64 × 64 part of P(T).above(zoom 0 to 5) does steps 3 to 6 with P(T) =parentof the children's planes. At zoom 6, the children are the block tops:join_blockreturns P and R of its zoom 6 tile, andabovereads their zoom 8snow-seasonstiles back from the spill files. The zooms align: per-year zoom 0 to 12 givesnowzoom 0 to 11 (one coarser) andsnow-seasonszoom 2 to 14 (two finer).Memory bound. A level of the walk holds the results of at most 4 children: R (640 KiB) and Q (16 × seasons × 8 KiB, 1.25 MiB for 10 seasons), and the planes of one tile while it makes it (seasons × 128 KiB). For 10 seasons, that is about 8 MiB per level and 7 levels from zoom 6 to 12: less than 60 MiB per worker. The bodies go into the spill file at once. So the memory of a worker does not grow with the block.
runs, for each pixel. k = the seasons that are not 255. c(b) = the seasons with snow on the day of band b, day = min(4 b + 2, 182): onset ≤ day ≤ melt-out, or 254. A scan over the 46 bands keeps the longest run; a run replaces the best one only when it is strictly longer, so the earliest run wins on a tie.worst, for each season and each 2 × 2 group of pixels one zoom deeper. In this order:A quarter whose child is None stays 255; the caller fills it from P(T).
Fill rules.
snowandsnow-levels: a missing child of zoom 9 to 12 takes its part of R(T), each pixel twice in each direction. A missing child of zoom 8 or less is 255 in red and green (no data). Without a child, or when no pixel of the half level has data, there is no tile.snow-seasons: a deeper tile that the walk lacks, that has no data, or that does not meet the bounds is missing, and its quarter takes its part of P(T). A tile outside the bounds lies wholly outside them, so the pixels inside the bounds do not depend on the bounds.Two small deviations from the Plan.
runs(planes)gives all five levels at once, shape (5, 2, …), instead ofruns(planes, level). The walk always needs the five, and k and c(b) are the same for each level, so one scan makes them all.join_blockreturns the runs of its zoom 6 tile with its planes, soabovedoes not compute them again in the main process.Generated by Claude Code
timohueser commentedon Nov 6, 2026
Deviations from the Plan in the pull request
runs(planes)gives all five levels at once, shape (5, 2, …), notruns(planes, level). k and c(b) are the same for each level, and the walk always needs the five, so one scan over the 46 bands makes them all.join_blockreturns the planes and the runs of its zoom 6 tile. Soabovedoes not compute the runs of each block top again in the main process.snow, notsnow,snow-levelsandsnow-seasons(step 11). Snow B. TrailView: the snow of a date from the run of snow #3127 merged intosnow-formatsfirst: itsharness.jsandlink.mjsserve and name onlysnow, the one archive that TrailView reads. The two other names would point at files that nothing serves.box.shstill writes all three archives into the ground folder.worst. The epic lists the cases as "the earliest onset and the latest melt-out of those that are dated; 254 when one of them is 254; 253 …; 255 …". When a 2 × 2 group has a dated pixel and a 254 pixel, two cases apply. I take 254 first, because the rule is the worst case and 254 has no snow-free day. The spec says it so: "by the first rule that applies: 254 when one of them is 254; the earliest onset and the latest melt-out of those that are dated; 253 when one of them is 253; else 255".snow-seasonstile outside the bounds is missing for its parent. The walk does not write it, so the parent quarter over it takes its part of the per-year tile. That quarter lies wholly outside the bounds, so the bytes inside the bounds do not change with the bounds.specs/README.mdalso changes: its row ofplanner-snow-tiles.mdnames the three archives.Generated by Claude Code
timohueser commentedon Nov 6, 2026
#3174 merged into the integration branch
snow-formats, after an adversarial review and green CI. The review found no bug: a brute-force probe matchedrunsandworston random planes, and every tile of a realjoin(357snow-seasonstiles, 15snowandsnow-levelstiles) against an independent reference. Its two small gaps are fixed in aca1403: a byte check of the tiles above the blocks, and stricterplanner_verifychecks of the three kinds. This issue closes whensnow-formatsmerges intodevelopwith the re-bake of #3125. Open step of Done when:box.shwith the CDSE keys.Generated by Claude Code
timohueser commentedon Nov 6, 2026
Done: #3174 merged into
snow-formats, and #3175 merged it intodevelop(3469850).