Deck.gl data explorer in the catalog

[mrshll]

Deck.gl data explorer in the catalog — plan

PR: #250 (draft) · Issue: dynamical-org/meta#236 · Branch: feat/deckgl-explorer

Goal

Visitors to a catalog dataset page can see the data itself: a map of one variable, read straight from our published Icechunk store in the browser with deck.gl. No tile server, proxy or pre-rendered images.

Where this stands

• Research and spike: done. The spike rendered noaa-gfs-forecast temperature_2m straight from the live Icechunk store in a browser, with values matching xarray exactly. Evidence and screenshots are below.
• Marsh's scope decisions (2026-09-25) are recorded in the next section.
• All 26 catalog products are on staging at commit f4f7a62, with CI green, now on phones too: 16 materialized (Track 1) and 10 virtual (Track 2). The second round (Marsh's staging feedback) is done: an empty-map preview, Init time and member selects, Stop loading instead of Unload, and the GEFS no-data fix.
• Staging is the Cloudflare Pages branch preview, https://feat-deckgl-explorer.dynamical-org.pages.dev, which deploys on push. Nothing merges to production.

Decisions (Marsh, 2026-09-25)

#	Decision	Answer
1	Products	All of them, to try in staging, with the map behind an explicit user action.
2	Variable	A select for all available variables.
3	Map vs time series	A map with a lead slider only; no point series.
4	Placement	Inline "Explore" section behind "Load interactive map".
5	Loading limit	None for this first staging phase.
6	Mobile	First "depends on bandwidth findings, probably disable", then (2026-09-26) "can we enable deck gl on mobile just to try". Enabled on every screen and pointer type.
7	Virtual stores	Included: "include them so we can try it in staging". This needs a browser GRIB decoder packaged as a zarrita codec, behind the same button. If it needs COOP/COEP or other page-wide changes, stop and report first.

Tracks

Track 1: materialized products (every one the explorer can read)

• The explorer core (explorer/, plain JS ESM built by Vite into public/explorer/) is generic over dims. It covers:

  • lead_time or time sliders;
  • an ensemble_member select (default member 0);
  • a select for any other dim, such as pressure_level;
  • a variable select;
  • regular lat/lon grids, plus projected grids where proj4js can express the CRS (HRRR Lambert, HRDPS rotated pole: being checked).

• The catalog integration adds the Explore section for enabled products, lazy-loads on click, quotes a per-product "about N MB to start", and shows a note instead of the button on mobile.

• Tests: offline unit tests; offline e2e against a tiny fixture Icechunk store; a live smoke test on GFS.

• All 16 materialized products render (headless Chromium, 758-px map). The "Estimated … ~N MB" caption on each page comes from this table:

  Product	Default view	~MB to first frame	Notes
  noaa-gfs-forecast	CONUS	9	105-lead blocks; scrubbing inside a block costs 0 bytes
  noaa-gfs-analysis	CONUS	46	time chunks of 1,440 steps; texture window 128
  noaa-gefs-forecast-35-day	CONUS	55	member 0
  noaa-gefs-analysis	CONUS	14
  noaa-hrrr-forecast-48-hour	CONUS	58	Lambert; proj4 built from CF (checked against the GRIB grid)
  noaa-hrrr-analysis	Houston area	44	CONUS would be about 1.5 GB
  noaa-mrms-conus-analysis-hourly	Houston area	9	CONUS would be about 387 MB
  ecmwf-aifs-single-forecast	CONUS	12
  ecmwf-aifs-ens-forecast	CONUS	138	every member decodes per chunk
  ecmwf-ifs-ens-forecast-15-day-0-25-degree	CONUS	195	every member decodes per chunk
  ecmwf-ifs-ens-forecast-46-day-daily-1-5-degree	CONUS	9	opens at +24 h (a 24 h mean is NaN at +0 h); pressure_level select
  ecmwf-ifs-ens-forecast-46-day-6-hourly-1-5-degree	CONUS	18	opens at +6 h
  dwd-icon-eu-forecast-5-day	Europe	52
  eccc-hrdps-forecast	Canada	114	rotated pole (ob_tran)
  nasa-imerg-analysis-early	CONUS	88
  nasa-imerg-analysis-late	CONUS	89

• A GPU-memory estimate is shown as a warning when a view needs a lot of texture memory. It never blocks loading (Marsh: no limits in staging).

Track 2: virtual products (10 -virtual stores)

• Needs a browser GRIB2 decoder with no threads, SharedArrayBuffer or COOP/COEP. It must be registered in zarrita as the gribberish codec (honouring var, adjust_longitude_range, north_up). scale_offset is already built into zarrita 0.7.5.
• icechunk-js must reach the virtual refs (NOAA/ECMWF buckets) through CORS with no redirects.
• Prior art: samn/deep-purple, a HRRR map that reads our virtual HRRR stores in the browser (review: /tmp/deep-purple-review.md).
  • It has a pure-TypeScript GRIB2 decoder registered as the zarrita gribberish codec: DRS 5.0/5.2/5.3 plus bitmap, grid 3.30 only, no WASM or COOP/COEP.
  • It runs on our exact icechunk-js / zarrita versions and is cross-tested against native gribberish.
  • It has no license. Ideas only until its author adds one. Marsh: could you ask Sam for MIT or Apache-2.0 on src/lib/grib/?
  • Ideas we're adopting in our own code: open the mutable repo object with no-cache, so a reload never opens a stale snapshot; probe the final lead so a virtual store opens on a run whose final lead exists (this doesn't prove every lead, member or variable is written).
• A spike is running first: it inventories GRIB packing templates per product, checks upstream CORS, decodes chunks in the browser, and compares them with Python.
• Estimate:
  • If deep-purple is licensed: about 2–3 hours. Vendor its decoder, add grid 3.0 and the 0..360 longitude rewrap for GFS/GEFS/AIFS, cross-check against Python, enable.
  • If not: about 1 day to write our own decoder from the GRIB2 spec.
  • Either way, a template that neither covers (e.g. ECMWF CCSDS, 5.42) stops that product and comes back to Marsh. So does anything needing page-wide changes.

What the research established (2026-09-25)

Stack.

• Rendering is Development Seed's deck.gl-raster, not deck.gl core. ZarrLayer takes a zarrita array, a selection for the non-spatial dims and GeoZarr-style spatial metadata. It tiles by inner chunk and reprojects on the GPU (WebGL2).
• icechunk-js itself is pure TypeScript: no WASM, workers or COOP/COEP. The tested codec pipeline does load a Blosc WASM module (205 KB gzip).
• The spike pinned @developmentseed/deck.gl-zarr/deck.gl-raster/geozarr 0.8.1, @deck.gl/* 9.4.0, @luma.gl/* 9.4.2, zarrita 0.7.5, icechunk-js 0.6.0.

Prior art.

• deck.gl-raster's dynamical-zarr-ecmwf example renders our legacy (pre-Icechunk) ECMWF zarr. Its synthetic transform has no spatial:registration. For our GFS grid that setting was required, so the example may be half a cell off; this needs checking against the legacy store before any upstream report.
• nldas-icechunk shows the Icechunk store path.

Our stores.

• There are 26 STAC collections, 16 materialized and 10 virtual. Each has an icechunk-https asset on regional S3.
• HTTP CORS and Range probes passed on objects in all ten dynamical-* buckets: HEAD, OPTIONS with range, and Range GET returning 206. Browser reads were exercised on GFS only.
• Materialized stores are chunked for point series. GFS temperature_2m is (init, lead, lat, lon) with chunks (1,105,121,121) and shards (1,210,726,726), blosc-zstd. So one tile is 30°×30° × 105 leads: 6.1 MB decoded, 0.7–1.5 MB compressed.
• The sampled virtual GFS temperature_2m array uses scale_offset + gribberish codecs over s3://noaa-gfs-bdp-pds GRIB byte ranges. Other virtual arrays were not inspected.
• In this GFS snapshot ingested_forecast_length is all-NaN, so it can't be used as a completeness signal.

Site.

• 11ty with no bundler. Page JS is public/*.mjs or inline. public/vendor/ already self-hosts versioned ESM with immutable caching.
• No CSP or COOP/COEP.
• Playwright e2e stubs the network from fixtures (pipeline.spec.mjs).

Spike evidence (GFS temperature_2m, headless Chromium + SwiftShader)

Step	1280×800	390×844 (DPR 3)
Cold load: repo, snapshot, manifests, coords, run probe	11 req, 0.56 MB, 2.2 s	11 req, 0.56 MB, 1.6 s
First frame at CONUS, lead 0	7 req, 6.5 MB; first frame 2.7 s from nav, complete 3.2 s	13 req, 12.6 MB; 2.2 s / 4.5 s
Lead 0→48, 48→104 (same block)	0 bytes, ~15 ms	0 bytes, ~15 ms
Lead 104→105 (next block)	6 req, 6.4 MB, 1.5 s	12 req, 12.5 MB, 2.2 s
Pan CONUS→Europe	7 req, 6.7 MB, 0.8 s	13 req, 12.8 MB
Additional transfer on zoom-out to global, after regional exploration	60 req, 57.8 MB, 32 s, 72 tiles	36 req, 35 MB, 14 s, 60 tiles at zoom 0 (not the whole globe), RSS 1.7 GB
JS	~600 KB gzip fetched (index 349 KB + blosc WASM 205 KB)

• Errors: with S3 blocked at start, the page shows a readable error in 0.3 s. With S3 failing after the first frame, the tested block-change failures blanked the tiles and showed an error, with no stale field. A same-block pan with partial failures was not tested separately; it becomes a required test.
• Caveats: timings are from software WebGL2 on the test host and say nothing about phone or Safari performance. No real device was tested. A full-block global view is 72 tiles × 6.1 MB, about 440 MB of texture; that is the payload, not a measured phone GPU allocation.

Screenshots

The images live on the orphan branch screenshots/deckgl-explorer; delete it after merge.

Spike (throwaway page, not site styling):

CONUS, +48 h	Europe, +48 h	Phone, zoomed out

Other spike shots:

• CONUS +0 h
• phone CONUS
• S3 blocked at start
• S3 fails after first frame
• cell registration check

Placement mockups (decision 4; the spike map is embedded as-is, so its control panel is oversized):

A: inline section, after activation	B: link on the dataset page, then a dedicated page

More mockup shots:

• A, before activation: desktop, phone
• B: link on the dataset page, explore page on phone

Mockup notes:

• Inline at content width (about 780 px) is cramped. The real version puts a compact control strip under the map, not overlaid.
• The mock's "~7 MB" button label is wrong for phones, where the load was 13 MB. The real label will say "Load interactive map" and quote a range.

Implementation plan

1. Explorer source in explorer/: plain JS ESM with JSDoc, so npm test imports its pure modules directly.
   • It is its own npm package with exact-pinned direct deps and a committed explorer/package-lock.json. That is a deliberate exception to the repo's gitignored-lockfile policy, so the tested dependency set (including the single proj4 instance) survives Pages rebuilds. It records every platform's optional native bindings (rolldown, lightningcss, fsevents), which CI's npm 11 requires for npm ci.
   • An eleventy.before hook in .eleventy.js builds it with Vite into public/explorer/ (gitignored), once per Eleventy process.
     • The hook runs npm ci whenever package.json or package-lock.json differs from the last successful install, or Vite is missing.
     • A failed install or build fails the site build.
     • The hook exists because the Pages build doesn't run npm run build; the first preview shipped without the bundle.
     • .node-version pins Pages to Node 22, which Vite 8 needs and which the explorer was tested on.
     • In npm start, restart after editing explorer/.
   • The virtual codec's wasm is committed, prebuilt (explorer/src/grib/, with SHA256SUMS). Its crate source and pinned build live in explorer/gribwasm/, so no site build needs Rust.
2. Spatial adapter: synthesize GeoZarr attrs from the store's own 1-D coordinate arrays.
   • Uniform spacing is checked; spatial:registration: "node".
   • Both latitude orders and both longitude conventions are handled.
   • The CF sphere is drawn as EPSG:4326 degrees.
   • Projected grids get a proj4 string, from the CF grid mapping or a per-product override.
   • A local EPSG resolver avoids the default epsg.io fetch.
3. Dims:
   • init_time: the latest init found by a bounded probe opens by default. An "Init time" select lists the variable's newest 20 inits. An explicit choice never falls back to another run, and an unwritten slice shows "No data" rather than a blank Ready.
   • lead_time or time: the slider.
   • ensemble_member: a select labelled with coordinate values, default member 0.
   • Any other dim: a select.
   • A variable select covers all float data variables.
4. Layer: one ZarrLayer per (variable, block, pinned indices).
   • All block steps sit in an r32float texture array, and the step is picked on the GPU.
   • One block is kept at a time; real lead or time coordinates are read.
   • Pitch is frozen, request concurrency bounded, and textures disposed.
   • Older results never draw under newer labels.
   • No loading limit this phase (decision 5). A "Stop loading" button, shown only while loading, aborts reads and keeps the drawn field. The earlier Unload/Load mode is removed.
5. Colour and labels: turbo colormap. The range is −40…50 for °C; otherwise the 2nd–98th percentile of the first block, frozen per variable. Labels show init / lead / valid in UTC, or the analysis time. Loading and error states. Site tokens around the canvas, working in both themes.
6. Catalog integration (_data/explorer.js, catalog-pages.njk):
   • An Explore section after the summary, for enabled products only.
   • Before activation the box shows an empty map: a build-time SVG of borders for the initial view (0.5–6 KB gzip), with the button over it. No explorer bundle and no weather reads until the click.
   • The button reads "Load interactive map", with "about N MB to start" per product. The explorer JS is dynamically imported on click.
   • The button shows on every screen size and pointer type (mobile enabled 2026-09-26 to try).
7. Upstream workarounds (small and commented): EPSG:3857 with +over; minZoom ≥ 0; wrapLongitude on the borders.
8. Tests:
   • Unit (offline): grid transform, proj4 from CF, dims classification, block math, time labels, shard-index probe.
   • E2E, offline (a tiny generated Icechunk fixture served by page.route):
     • JS stays lazy until the click;
     • a rendered pixel registers to the right cell;
     • a block-crossing slider move;
     • out-of-order responses;
     • S3 blocked;
     • a phone-width load that draws without widening the page.
   • E2E, live: a GFS smoke test.
   • The full suite and the build run after integration.
9. Track 2 (virtual): a gribberish + scale_offset codec module for the explorer, float64 input, one lead per chunk. Virtual-ref options live where icechunk-js opens the store. Enabled per product in _data/explorer.js once the spike proves the templates.
10. Possible upstream reports to Development Seed (after verification, your call): the 3857 dateline wrap; the minZoom ≤ −1 hang; the ECMWF example's registration, if the legacy store confirms it.

Out of scope for this phase

Point series, ensemble summaries, animation autoplay, prefetching the next block, and production.

Work plan

• Meta issue (meta#236), assigned to Marsh
• Research: libraries, stores (measured bytes, CORS), site
• Browser feasibility spike on the current Icechunk store
• Placement mockups (A inline, B dedicated page)
• Marsh decides scope (2026-09-25)
• Per-product readability inventory (materialized): all 16 render
• Track 1: explorer core (implementer A)
• Track 1: catalog integration, build, tests (implementer B)
• Track 1: branch preview deployed (https://feat-deckgl-explorer.dynamical-org.pages.dev)
• Track 2: GRIB codec spike (implementer C)
• Track 2: integrate the codec, enable the virtual products, redeploy the preview
• Review (opposite vendor): 3 passes, the maximum. Pass 1: 7 findings, pass 2: 4, pass 3: 2. All fixed; pass 3's fixes are unreviewed

Log

2026-09-25 — research and spike

• Filed the issue (meta#236). Three researchers covered the libraries, our stores and the site; a Codex second-look challenged each step.
• Corrections along the way:
  • The store report dropped the 105-lead chunk factor from its map-frame math.
  • The library report's 0.7.0 pin set is superseded by npm view.
  • The ECMWF example is only cheap because the legacy zarr was chunked differently; the current Icechunk ECMWF store costs 143 MB for CONUS.
• The spike (Claude implementer) passed every criterion. I re-ran its Python cross-check independently and got max |diff| 0.0.
• The second-look's plan review led to these changes:
  • loading-limit wording: a viewport-aware threshold, not camera minZoom;
  • numerical coverage stated exactly;
  • the ~7 MB label dropped in favour of a range;
  • mobile acceptance and the forecast horizon added as decisions;
  • upstream claims softened until verified.
• A Claude implementer made the placement mockups in a throwaway worktree, since removed.

2026-09-25 — Marsh's decisions, build started

• Marsh (relayed by the supervisor): all products, a variable select, a lead slider, inline behind the button, no limits in staging, mobile probably off.
• I clarified that virtual stores need a browser GRIB decoder (/tmp/dynorg-deckgl-d7.md). Marsh then included them too.
• Seated:
  • researcher-4: per-product inventory;
  • implementer A: explorer core;
  • implementer B: catalog integration and tests;
  • implementer C: GRIB codec spike.
• A and B work in disjoint files under a shared contract; I commit.
• Codex helpers lost network (their DNS fails), so researcher-4's inventory came back empty and researcher-5 couldn't clone deep-purple. I reported this to the supervisor. The per-product inventory moved into implementer A's acceptance (browser-measured first-view MB and proj4 per product). I reviewed deep-purple myself from a host clone.

2026-09-25 — Track 2 spike passed

• Decoder: gribberish 1.8.0 (MIT) built for wasm32-unknown-unknown with wasm-bindgen: 112 KB gzip, no threads, no SharedArrayBuffer, no COOP/COEP, no C deps (pure-Rust CCSDS fallback).
• Templates in use: DRS 5.3 and 5.0 (NOAA) and 5.42 (AIFS); grids 3.0 and 3.30. All are covered.
• Correctness: 13 sampled chunks spanning all 10 virtual stores are bit-exact against Python (max |diff| 0.0, NaN masks identical). scale_offset is zarrita's built-in codec.
• Upstream reads: icechunk-js reaches all 6 upstream hosts through their regional endpoints with 0 redirects; CORS passes. ecmwf-forecasts (about 5% of AIFS refs, late 2024 to early 2025) throttles with 503 and no CORS headers, so it needs a retry fetchClient.
• Found and fixed in app code: a whole-globe chunk rendered as one tile misregisters by 1–3 cells (17/54 points), because deck.gl-raster's mesh refinement hits its 10k-iteration cap. A 121×121 facade over the decoded chunk gives 54/54.
• Cost: cold open is 6.65 MB of Icechunk metadata (the snapshot alone is 6 MB, with a 1.5–2.4 s main-thread parse). After that, a global frame is 0.51 MB and each lead change is one about-510 KB request, compared with 58 MB for a global frame of the materialized store.
• Plan for staging: decoder + codec + tile facade + retry. The Worker offload (for the snapshot parse) is deferred. C's full estimate is 6–8 engineer-days to production quality; the staging subset is about 2–3 days.
• The deep-purple decoder wasn't used: no license, and it lacks 5.42.

2026-09-25 — Track 1 on staging; review pass 1

• Committed 2fa9997 (core + integration, GFS only) and 10cdc71 (all 16 products, eleventy build hook).
• The first preview had no bundle: Pages doesn't run npm run build. Fixed with the eleventy.before hook, which I verified with a bare npx @11ty/eleventy in a clean worktree. On the clean commit, all unit tests and the 10 offline e2e specs pass. GFS on the live preview: ready, 8.6 MB.
• Review pass 1 (Codex) found seven issues. All are accepted:
  1. retry stays stuck after a failed coordinate read;
  2. Unload, then a selection change, strands the map;
  3. an analysis with an empty final window opens blank;
  4. the GPU guard blocks, which contradicts "no limits" (C first added "Load anyway"; after the second-look it becomes advisory only);
  5. the build hook skipped npm ci on a lockfile change — fixed in a5b11de;
  6. partial-failure e2e coverage was missing;
  7. this plan was stale.
     Implementer C is fixing 1–4, implementer B is adding tests for 1, 2, 3 and 6, and this entry covers 7.

2026-09-25 — second-look on pass 1; virtual code committed

• Committed 8495a46: C's virtual hook-up (lazy gribberish codec, whole-grid tile facade, upstream retry, final-lead probe) and review fixes 1–4. On a clean checkout, 269 unit tests and the 10 committed offline specs pass. No virtual product is enabled on a page yet.
• The second-look pushed back on two things:
  • "Load anyway" is still a gate, so the GPU estimate becomes advisory only (C);
  • the mtime-based build skip is fragile.
• The hook now builds once per Eleventy process and reinstalls when either npm file changes; .node-version pins Pages to Node 22 (9b17abb).
• Also in progress:
  • the MRMS all-zero colour range re-establishes on the first non-constant sample (C);
  • the retry wording no longer claims "throttled" (C);
  • the Houston views are labelled as such (B).
• Fixed a plan edit of mine that had overwritten two screenshot links.

2026-09-25 — all 26 on staging

• Committed 0e30809 (advisory GPU estimate, provisional colour range, facade keys and abort) and 2fda2b5 (10 virtual rows, captions, the phone overflow fix, e2e for pass-1 findings 1/2/3/6, and a live virtual spec).

• On a clean checkout: 272 unit tests and 66 Playwright specs pass (explorer offline + live, agent-prompt, pipeline).

• Staging checked on the preview:

  Product	Download to first frame
  noaa-gfs-forecast-virtual	7.2 MB
  noaa-hrrr-forecast-48-hour-virtual	2.8 MB
  ecmwf-aifs-single-forecast-virtual	0.8 MB
  noaa-gefs-forecast-16-day-0-5-degree-virtual	24 MB, mostly its 23.9 MB snapshot

  All reached Ready. The wasm and codec chunks are served lazily from Pages.

• Virtual per-product results (C, headless Chromium): all 10 render. Registration is 45/45 on each of GFS, HRRR-48, AIFS and GEFS 0.5°. Each lead or time change is one request of 0.14–1.22 MB.

• Known limitations:

  • the snapshot parse (GFS virtual: a 6 MB snapshot, about 2 s on the main thread) isn't in a Worker yet;
  • (fixed in b71df8a) an accumulation's +0 h is NaN in virtual stores; the explorer now opens on the first lead with data;
  • the 98th-percentile range saturates heavy precipitation cores.

• Blocked: review pass 2 couldn't run. Codex returned 401 Unauthorized: Incorrect API key on its first request. I reported it to the supervisor. The review is opposite-vendor, so it waits for Codex auth instead of switching to a Claude reviewer. The target stays frozen at 2fda2b5.

2026-09-25 — review pass 2

• Codex auth came back at about 23:20Z, and I re-seated the pass-2 reviewer on the frozen 2fda2b5.
• Pass-1 verification:
  • findings 1, 3, 6 and 7: resolved;
  • finding 2: partly resolved;
  • findings 4 and 5: resolved in code, with no committed regression test.
• New findings, all accepted:
  1. (P2) While unloaded, a level change reverts a pending variable change.
  2. (P2) Unload and clear don't cancel queued or unowned whole-grid reads.
  3. (P2) Virtual accumulations open blank at +0 h.
  4. (P3) A stale rejection can evict a newer facade cache entry.
• Fixes:
  • C fixes 1–4 in explorer/;
  • B adds e2e for 1 and 3, for the advisory GPU estimate (pass-1 Zarr v3 docs support + gefs zarr v3 dataset #4 coverage), and for out-of-order draws through the facade;
  • I moved the build hook into lib/explorer-build.js with 6 unit tests (pass-1 Add display of optional variable comment field to docs #5 coverage), commit b9457a0.
• Pass 3, the last one allowed, will run on the fixed commit.
• Pass-2 fixes were committed in b71df8a (C: pending selection, facade cancellation with waiter counting, empty-first-lead search, the stale-rejection guard, abortable backoff, probe-failure wording) and 82571bb (B: 5 e2e specs, each failing on the old bundle for its bug).
• On a clean checkout of 82571bb: 288 unit tests and 71 Playwright specs pass, including live GFS and GFS-virtual.

2026-09-25 — review pass 3 (final) and fixes

• Pass 3 on 82571bb confirmed:
  • pass-2 findings 1, 3 and 4: resolved;
  • pass-2 finding 2: partly resolved;
  • pass-1 Fix a typo in the GEFS code sample #2, Zarr v3 docs support + gefs zarr v3 dataset #4 and Add display of optional variable comment field to docs #5 coverage: resolved.
• It found two new issues:
  1. (P2, blocked the staging trial) During a loaded variable switch, the old level select could still fire, drawing the old variable under the new dropdown.
  2. (P2) Abandoned reads that never settle kept holding facade concurrency slots.
• Both were fixed in 26b391a (C), with B's e2e in e17dbf2; that spec fails on 82571bb. No fourth pass is allowed, so these fixes are unreviewed. My verification on a clean checkout of e17dbf2: 293 unit tests and 72 Playwright specs pass, including the explorer offline + live, agent-prompt and pipeline specs.

Retro

• What each review pass bought:

  • Pass 1 (7 findings): real bugs a visitor would hit — Retry stuck, Unload stranding the map, blank analyses, the memory guard blocking against "no limits" — plus a build-hook flaw and a stale plan. Clearly worth it.
  • Pass 2 (4 findings): mostly the virtual track's new concurrency (facade cancellation, a stale-rejection race), a real wrong-variable draw while unloaded, and blank virtual accumulations. It also caught that two pass-1 fixes had no regression test. Worth it: two of its bugs were introduced by pass-1 fixes and the virtual integration.
  • Pass 3 (2 findings): the loaded counterpart of the pass-2 selection bug, which would have misled Marsh on staging, and a never-settling-read edge case. The first justified the pass; the second is hardening.
  • Churn was low: no pass re-litigated an earlier decision.

• Second-look: the highest-leverage calls came before any code was written:

  • make the browser read cost the first gate;
  • correct the 105-lead chunk math;
  • don't call "Load anyway" a warning.

• What went wrong:

  • The Codex sandbox lost network, and later auth, mid-session. Two research seats produced nothing, and pass 2 waited on an outside fix.
  • Two of my own plan edits went wrong: a line-number sed clobbered screenshot links, and a write of the deep-purple review silently failed before I told the supervisor it was done.
  • Pages doesn't run npm run build. That was learned from a broken first preview rather than checked beforehand.

• What worked:

  • Disjoint file ownership between implementers under one written contract.
  • A frozen commit for each review pass.
  • Verifying every commit in a clean worktree rather than the shared one.

• Final staging check on e17dbf2's preview (all Ready):

  Product	Download to first frame
  GFS	8.6 MB
  GFS virtual	7.2 MB
  HRRR 48 h	58 MB
  HRDPS	115 MB
  AIFS virtual	0.8 MB
  MRMS	0.6 MB (the newest time chunk had just started)

• Follow-ups, not blocking:

  • move the snapshot parse into a Worker;
  • draw zero precipitation as transparent rather than the darkest colour;
  • use a precipitation colour scale that doesn't saturate heavy cores;
  • upstream reports to Development Seed (the 3857 dateline wrap, the minZoom ≤ −1 hang, the globe-sized tile misregistration).

2026-09-26 — CI fix

• CI's render job failed on e17dbf2. Its e2e web server runs eleventy, whose explorer hook runs npm ci. Under CI's Node 24 / npm 11, that refused the lockfile as out of sync: it held only the linux-x64-gnu builds of rolldown and lightningcss, and no fsevents.
• I had verified only with npm 10 (Node 22), locally and on Pages, which accepted it. That was a gap in my checks.
• Fixed in 1f31b27: regenerated with npm 11 --package-lock-only. That added 25 entries and changed no locked version. npm ci passes with npm 11 and npm 10.
• On 1f31b27: render 93 passed (all e2e, including explorer offline + live), and unit, CodeQL and Cloudflare Pages pass.
• This was a lockfile-only change, so no review pass.

2026-09-26 — Marsh's staging feedback, round 2

Marsh: "instead of the example image, show the map empty (no data) with the button overlayed. adding controls for init time and ensemble member (if ensemble) would be cool. What is the 'unload' button for? GEFS seems to fail at any of the 6-hourly steps."

• Empty-map preview (1ab5eee):
  • A build-time SVG of world-atlas borders for each product's initial view, projected and fitted exactly as the live map fits.
  • Clipped at build time: +0.5–6 KB gzip per page, less than the thumbnail JPEG it replaces.
  • aria-hidden, with the button over it. No explorer bundle and no weather reads before the click.
  • The Houston-area views show only the Gulf coastline; no extra map assets were added.
• GEFS "fails at 6-hourly steps":
  • Root cause: a partly written newest run. On GEFS 35-day, the default init's later lead chunks exist but hold only NaN, and the explorer reported Ready over a blank map. The 3-hourly → 6-hourly switch (+240 h) was coincidence: it was presumably where the run had been written to when Marsh looked.
  • Fix (bf69c47, refined in dc4e4de): each block records which steps have values. When no visible tile has a value at the chosen step, the state is "No data". The message says where values end and that an earlier init may have them.
  • Regression tests fail on the old code.
  • On staging: +246 h and +276 h draw. Past +384 h the newest run says "No data … values end at +384 h"; the 09-24 run draws at +480 h.
• Init time and ensemble member (bf69c47):
  • Init time: a select of the variable's newest 20 inits at the pinned snapshot. An explicit choice never falls back to another run or advances the lead.
  • ensemble_member: a select labelled with coordinate values.
  • Missing data and transport failure are distinct states, and Retry repeats the exact selection.
• Unload → Stop loading:
  • Unload stopped all reads and freed the map's GPU memory, returning to the pre-load state: a safety valve for heavy products. It was confusing and caused two earlier review bugs.
  • The persistent unload/reload mode is removed. A "Stop loading" button shows only while loading: it aborts reads, keeps what's drawn, and Retry or any new selection resumes.
• Also fixed:
  • a pre-existing race (20d353b): two slider moves within one frame destroyed the textures of the layer on screen and hung in "loading";
  • a test-only CodeQL alert (8f84af5).
• Review of this change set:
  • Pass 1 (Codex, on 20d353b) found 4 issues:
    1. Stop didn't stop queued tiles;
    2. a slider move during an init/member change was overwritten;
    3. Stop during startup was undone;
    4. cached off-screen tiles made a blank view read Ready.
       All were fixed in dc4e4de, and E closed two test gaps in bd11232 (a real mid-load level change; an ensemble fixture and member-switch spec).
  • Pass 2 (Codex, on bd11232): all four resolved, and no staging blockers. The one remaining approximation is documented: a partly visible edge tile with values only off-screen counts as data.
• Verification on a clean checkout of bd11232: 297 unit tests; explorer offline e2e 108/108 over 3 repeats with no retries; live + agent-prompt + pipeline 49/49. CI on bd11232 is all green: render (e2e) 106 passed, plus unit, CodeQL, Analyze and Cloudflare Pages.

2026-09-26 — mobile enabled

• Marsh: "can we enable deck gl on mobile just to try". f4f7a62 removes the small-screen/touch gate and the note: the button shows everywhere.
• The phone spec now loads the map at 390 px and checks it draws without horizontal scroll.
• Staging, iPhone 13 emulation (touch, DPR 3, software WebGL): GFS Ready at 7 MB, GFS-virtual Ready at 7.2 MB, scrollWidth 390. The controls stack under the map.
• Not tested on a real phone GPU. Texture memory (about 6 MB per materialized tile) is the device risk on heavy products.
• CI on f4f7a62: all green (render, unit, CodeQL, Analyze, Pages).

[cloudflare-workers-and-pages]

Deploying dynamical-org with    Cloudflare Pages

Latest commit:	f4f7a62
Status:	✅  Deploy successful!
Preview URL:	https://14f05039.dynamical-org.pages.dev
Branch Preview URL:	https://feat-deckgl-explorer.dynamical-org.pages.dev

View logs