Skip to content

Add combined NOAA GFS forecast and analysis virtual quickstart - #78

Draft
aldenks wants to merge 3 commits into
mainfrom
gfs-virtual-quickstart
Draft

aldenks wants to merge 3 commits into
mainfrom
gfs-virtual-quickstart

Conversation

@aldenks

@aldenks aldenks commented Sep 18, 2026

Copy link
Copy Markdown
Member

One quickstart covering both noaa-gfs-forecast-virtual and noaa-gfs-analysis-virtual, following the two-dataset shape of ucsb-chc-chirps-analysis.ipynb and the AGENTS.md intro/question structure.

Temporary direct open

Neither dataset is in the STAC catalog yet, so dynamical_catalog.open(...) cannot resolve them. Cell 2 is the only deviation: it opens the two Icechunk stores directly with anonymous S3 storage plus authorize_virtual_chunk_access for s3://noaa-gfs-bdp-pds/, and opens the pressure_level group there too so no later cell needs it. The cell header comment carries the three dynamical_catalog.open(...) lines that replace it at publish time; everything downstream is unchanged by that swap. The install line stays dynamical-catalog cartopy (icechunk arrives transitively); no new project dependency.

The catalog documentation links in the intro point at the expected https://dynamical.org/catalog/noaa-gfs-{forecast,analysis}-virtual/ pages, which do not exist until the STAC publish.

What it shows and why

Every example uses something the materialized noaa-gfs-forecast / noaa-gfs-analysis (25 variables each) do not carry, and none reuses their events or those of the HRRR virtual quickstarts (Idalia, Mumbai monsoon, Feb 2024 atmospheric river, Shanghai; Buffalo Nov 2022, Dec 2022 arctic front, June 2023 smoke, Greenfield, Milton, March 2025 outbreak):

Section Dataset Field Event
Map analysis convective_available_potential_energy_surface 31 March 2023 tornado outbreak, 21 UTC
Time series analysis snow_thickness_surface Boston through the 29 January 2022 nor'easter, 27 Jan–1 Feb
How far ahead did the forecast carry Eunice's gusts? forecast + analysis wind_gust_surface Storm Eunice at The Needles; inits 5/3/1 days before vs analysis
When did the polar vortex reverse in Feb 2023? analysis pressure_level wind_u at 10/30/100 hPa, 60°N zonal mean, 12-hourly Feb 2023 SSW; first easterly at 10 hPa is 2023-02-16T00 in the data
Animation analysis composite_reflectivity, hourly, 37 frames Hurricane Beryl, 2024-06-30T18 → 07-02T06

All reads are bounded to fixed time slices and regions; the whole notebook makes ~600 chunk reads and executes in ~40 s locally.

Facts re-derived from the stores (not taken from the brief)

Read on 2026-09-17 via xr.open_datatree on readonly_session("main"); snapshots forecast SVZ00W5RRRW4Y6PCCVYG, analysis 5Q0Z4PQ0620AA2F576C0.

  • forecast root has two variables the analysis lacks: convective_precipitation_run_total_surface, total_precipitation_run_total_surface; no analysis-only variables.
  • forecast: 255 root + 16 pressure_level + 3 height_above_mean_sea_level = 274 data variables; dims (init_time: 7864, lead_time: 209, latitude: 721, longitude: 1440); init_time 2021-05-01T00 → 2026-09-17T18 every 6 h; lead 0–384 h, hourly to 120 h then 3-hourly.
  • analysis: 253 + 16 + 3 = 272; dims (time: 47182, latitude: 721, longitude: 1440); time 2021-05-01T00 → 2026-09-17T21 hourly.
  • pressure_level: 57 levels, 1000 → 0.01 hPa. height_above_mean_sea_level: 8 heights, 305–4572 m (note its dims are (time, latitude, longitude, height) — height last).
  • Virtual chunks are one global field per step: (1, 1, 721, 1440) forecast, (1, 721, 1440) analysis.
  • Materialized twins: 25 variables each; the only materialized variable absent from the virtual stores is the derived precipitation_surface.

Checks

  • uv run .internal/run_notebooks.py noaa-gfs-forecast-analysis-virtual.ipynb — 2.7 MB, ~40 s
  • uv run .internal/run_notebooks.py --isolated noaa-gfs-forecast-analysis-virtual.ipynb — passes against only the install line's packages
  • uv run python tests/test_notebook_execution.py noaa-gfs-forecast-analysis-virtual.ipynb — OK
  • uv run python .internal/check_install_lines.py — all 24 install lines covered
  • python -m unittest tests/test_notebook_coordinator.py tests/test_notebook_deps.py tests/test_notebook_isolation.py — 12 tests OK

The new notebook lands in CI shard 2 (sorted, stride 3).

🤖 Generated with Claude Code

@aldenks

aldenks commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

Ready for review. Reviewed by a separate agent; all five findings applied in 550adb9.

Two things to decide before merge, not before review:

  1. The catalog documentation links do not resolve yet. noaa-gfs-forecast-virtual and noaa-gfs-analysis-virtual are not in the STAC catalog — the publish is a later stage — so the two dynamical.org/catalog/... links in the intro are soft-404s today (the catalog is a client-side app, so they return the 200 shell and find nothing). google-weathernext2-forecast-historical+operational-virtual.ipynb (Add corrected WeatherNext 2 virtual forecast quickstart #73) shipped the same way. These must be re-checked after the STAC publish, together with item 2.

  2. Cell 2 is a temporary store-open that needs swapping at publish time. Because the datasets are uncataloged, the notebook opens the Icechunk stores directly with anonymous credentials. The deviation is confined to that one cell (verified: no other cell touches the store-open path), and its header comment carries the exact four-line replacement, ending with the trailing forecast line so the repr that the next cell references survives the swap.

Unrelated, but easier to change now than later: both stores place the level dimension last in both vertical groups — pressure_level is (time, latitude, longitude, pressure_level) with chunks (1, 721, 1440, 1), and height_above_mean_sea_level likewise. It is consistent across both datasets and invisible to readers through xarray’s labelled .sel, so the notebook says nothing about it. Worth confirming the ordering is intended before the STAC publish, since it is hard to change afterwards.

Review notes: the earlier draft used the November 2022 Buffalo lake-effect event, which is already the centerpiece of noaa-hrrr-analysis-virtual.ipynb — same provider and same dataset shape, and HRRR resolves it far better at 3 km. It now uses the 29 January 2022 nor’easter at Boston, a synoptic-scale event where a global 0.25° product is the appropriate instrument.

@aldenks aldenks self-assigned this Sep 18, 2026
Colab can mishandle '+' in a notebook path, so use a dash instead. Pure
rename: the notebook content is byte-identical and no other file in the
repo referenced the old name.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant