NNJA Sat Obs: ATMS/AMSUA/MHS/AMSUB (stacked on #966)#968
Conversation
Exercise the three-operation store seam through the synchronous __call__ path so cleanup delegation is verified, including that cleanup still runs when the request fails mid-flight.
AMSU-A validation — NNJA aggregate vs UFS/GSIReviewer question: Does the new NNJA AMSU-A datasource preserve the aggregate microwave observations used by the selected UFS/GSI replay across every represented platform and physical channel? Validation layer: NNJA aggregate-to-UFS/GSI inclusion and value parity for the six-hour cycle centered on 2024-01-01 00 UTC.
The numeric source for these figures is Metop-B — channels 1–15 (4 pages)
Metop-C — channels 1–15 (4 pages)
NOAA-15 — channels 1–15 (4 pages)
NOAA-18 — channels 1–15 (4 pages)
NOAA-19 — channels 1–15 (4 pages)
|
Validation method — locating NNJA observations at UFS/GSI diagnostic locationsThe UFS/GSI files are treated as a selected diagnostic population, not as the raw-source population oracle. The comparison asks whether each diagnostic row can be traced uniquely to the corresponding NNJA aggregate observation and whether its value agrees after any documented GSI-side transform.
This establishes UFS/GSI inclusion and value parity. It does not prove that every finite NNJA observation is scientifically usable, or that UFS/GSI selection is the only valid quality policy. Exact input hashes, matching manifests, and figure paths are in |
AMSU-B validation — 2004 UFS/GSI parity and late-life NOAA-16 source degradationReviewer questions: Does NNJA reproduce the AMSU-B observations selected by UFS/GSI, and do the aggregates expose enough metadata to identify late-life channel degradation? The identity and one-to-one assignment rules are documented in the shared matching procedure. Observation values are not used to choose matches. 2004 UFS/GSI selection-parity result
2004 platform/channel evidence (3 figures)NOAA-15 — channels 1–5
NOAA-16 — channels 1–5
NOAA-17 — channels 1–5
Historical NOAA-16 source-quality findingThe late-life behavior is best described as progressive distribution broadening, not assigned a causal label such as “corruption.” Channels 3–5 broaden substantially from 2004 through 2012/2013 and into 2014; by May 2014, channels 4–5 are absent from the audited aggregate. An independent Earth2Bufr decode agrees with Earth2Studio on all 36 public fields for the January 2014 sample (channel RMS differences approximately (5–8 × 10^{-6}) K), so the abnormal distributions are source content rather than an Earth2Studio decoding artifact. The audited NNJA aggregate’s channel-, scan-, granule-, and geolocation-quality fields are null, and the available correction field is constant rather than observation-varying QC. In other words, NNJA does not expose a usable per-row flag that says when these late-life channels remain scientifically usable. Historical GSI Practical consequence: finite late-life NOAA-16 AMSU-B values should not automatically be treated as training-ready. A platform × channel × time validity policy is required. NOAA-16 historical source audit (2 figures)January 2014 independent-source parity
Distribution evolution: 2004, 2012, and January 2014
Reproduction details and exact source hashes are in |
MHS validation — NNJA aggregate vs UFS/GSIReviewer question: Does the new NNJA MHS datasource preserve the observations selected by the UFS/GSI replay across every represented platform and physical channel? The identity and one-to-one assignment rules are documented in the shared matching procedure. Observation values are excluded from matching.
UFS/GSI is a selected diagnostic population, not a raw-population completeness oracle. The maps and histograms show the complete finite NNJA population alongside the selected and uniquely matched populations; no geographic binning or display thinning is used. MHS platform/channel evidence (3 figures)Metop-B — channels 1–5
Metop-C — channels 1–5
NOAA-19 — channels 1–5
The numeric source is |
ATMS validation — NNJA aggregate vs UFS/GSI after GSI spatial averagingReviewer question: Does the new NNJA ATMS datasource reproduce the observations selected by UFS/GSI across every represented platform and physical channel when the diagnostic’s preprocessing is modeled correctly? The identity and one-to-one assignment rules are documented in the shared matching procedure. Observation values are excluded from matching. Result
Why ATMS uses a transformed comparisonFor ATMS, the UFS/GSI diagnostic is not compared directly with a single raw aggregate sample. The validation first applies the documented GSI spatial/beamwidth averaging to NNJA The figures therefore show both the complete untransformed NNJA source population and the correctly transformed uniquely matched population. This validates the GSI-facing preprocessing path without hiding the raw NNJA distribution. NOAA-21 has finite NNJA observations for all 22 channels but no diagnostic rows because the replay configuration marks that platform passive ( Suomi-NPP — channels 1–22 (6 figures)
NOAA-20 — channels 1–22 (6 figures)
NOAA-21 — channels 1–22, passive in replay (6 figures)
Maps and histograms use complete populations with no geographic binning or display thinning. The numeric source is |
Important NotesEncoded field versus physical quantityATMS is the only feed in this PR that encodes separate
For the legacy feeds,
For reference, see these NOAA GSI issues https://redirect.github.com/NOAA-EMC/GSI/issues/341 (traditionally assimilated ta not tb) and the follow up https://redirect.github.com/NOAA-EMC/GSI/issues/380 (identified N15/N16 AMSU-A was actually tb). QuantizationNCEP already quantized the input to 0.01 K. This is what the GSI consumes. |
| self._handle_fetch_failure(file_uri_set, exc) | ||
| raise | ||
|
|
||
| df = self._compile_dataframe(async_tasks, schema) |
There was a problem hiding this comment.
I think this loses the fix from #971: _compile_dataframe is CPU-bound (and internally blocks on pool futures), so calling it synchronously inside fetch stalls the shared fsspec IO loop that _sync_async dispatches every data source's fetch onto. #966 kept it as df = await asyncio.to_thread(self._compile_dataframe, async_tasks, schema) in utils_ncep.py, but the move into _NCEPObsSourceBase.fetch here dropped the to_thread wrapper. Suggest restoring:
df = await asyncio.to_thread(self._compile_dataframe, async_tasks, schema)There was a problem hiding this comment.
Yes. I will update this once the base PR is merged, bringing in the fix from main. (This branch is also out of date with the base PR branch since it is currently being revised.)
2 participants
Earth2Studio Pull Request
Description
Stacked PR — draft until #966 merges. This branch is stacked on the NCEP observation refactor, so that PR’s changes appear here until it lands. Once merged, this branch will be rebased onto main, leaving only the NNJA satellite-observation implementation, tests, documentation, and validation. Until then, review only the commits starting with 57fdf2e (feat: add NNJA aggregate microwave observations).
This PR adds
earth2studio.data.NNJAObsSat, a historical NCEP aggregatemicrowave observation source backed by NNJA. It supports ATMS, MHS, AMSU-A,
and AMSU-B through one source-faithful long DataFrame API while reusing the
private NCEP request and NNJA S3/cache foundation.
What Changed
TMBRasatmsand encodedTMANTexplicitly asatms_antenna_temperature.location, view/solar geometry, available source quality/calibration fields,
and message/subset ordering provenance.
incomplete instead of returning an apparently successful partial result.
continuous decode window.
Compatibility
NNJAObsConvbehavior is unchanged and retains its permissivemissing-cycle policy.
pandas.DataFrame; no compact orstreaming API is introduced.
applied by the source.
Validation
The public source was executed on complete six-hour NNJA files, producing
117,615,825 ATMS/MHS/AMSU-A rows plus a 9,150,780-row AMSU-B historical
holdout. External evidence is kept separate by product boundary:
MHS, and AMSU-A; values agree at binary roundoff.
candidates. ATMS is compared after the official beamwidth transform; the
other sensors use the aggregate TMBR value.
no unmatched UFS rows, 160 ambiguous identities, and zero residual on all
38,025 unique pairs.
Metop overlaps; the Metop conversion remains tolerance-qualified.
NNJAObsConvsame-input output remains exact.On the current reviewed head, 40 focused NNJA data/lexicon tests pass with two
slow network tests deselected; Ruff, Black, source-module mypy, and
git diff --checkalso pass. The complete execution packet must be rerun on thefinal reviewed commit before the PR is marked ready.
Reviewer figures cover every channel and platform. Each page shows the full
unthinned NNJA population, UFS, NNJA-minus-UFS residuals at unique identities,
and overlaid full-population histograms. Temperature maps share a scale; the
residual map uses a separate zero-centered scale. Maps use opaque points and no
geographic binning or display thinning.
Checklist
Dependencies