Skip to content

Use quantile delta mapping (QDM) to bias adjust forecast, reanalysis - #17

Open
brews wants to merge 5 commits into
ClimateImpactLab:mainfrom
brews:qdm
Open

Use quantile delta mapping (QDM) to bias adjust forecast, reanalysis#17
brews wants to merge 5 commits into
ClimateImpactLab:mainfrom
brews:qdm

Conversation

@brews

@brews brews commented Aug 4, 2026

Copy link
Copy Markdown
Member

This PR adds prototype quantile-delta mapping (QDM) bias adjustment of ERA5 and season forecast ensembles to GMFD before mortality projection.

To run with this change:

  1. Download forecast ensemble with scripts/00-download_forecast.py. This now downloads multiple years' worth of forecast ensembles initialized at a particular month. The data is downloaded to a local directory. Currently, this is 1981 - 2026 for May. I let this run overnight. It downloads under 70 GiB of data.

  2. Run scripts/01-parse_forecast.py. This parses the downloaded data and writes its output to POREALLAS_PARSED_FORECAST_URI.

  3. Run scripts/01-parse_era5.py and scripts/01-parse_gmfd.py. I run this on a daskhub cluster hosted on Google Cloud. It uses ERA5 data from Google's ARCO store (as before) and the CIL's internal GMFD data store. These write output to POREALLAS_PARSED_ERA5_URI and POREALLAS_PARSED_GMFD_URI, respectively.

  4. Run scripts/02-bias_adjust_era5.py and scripts/02-bias_adjust_forecast.py. These use the data parsed above as input. Output is written to POREALLAS_ERA5_URI and POREALLAS_TAS_FORECAST_URI, respectively. Bias adjustment for ERA5 takes about 3 minutes on my laptop. Bias adjustment of the forecast ensemble takes roughly 2.5 hours.

  5. Now you can project mortality as you would have before by running scripts/05-project_effects.py. This writes full, "hot only" and "cold only" projections of the forecast and baseline period.

A few additional things to note:

  • Parsed files have their calendar converted to "noleap" and so output data files now also use a "noleap" calendar. This was needed because the QDM implementation needs input data to use the same calendar. Pandas and geopandas might have trouble using this calendar. You can convert it back to a standard calendar for analysis with Xarray's convert_calendar.
  • All the output Zarr Stores in these scripts use consolidated metadata.

The README and example.env have been updated to reflect these changes.

@brews brews self-assigned this Aug 4, 2026
@brews brews added the enhancement New feature or request label Aug 4, 2026
brews added 3 commits August 3, 2026 20:21
coldonly projection was merged into the other project_effects script.
Removing this to clean up the interface.
@codecov-commenter

codecov-commenter commented Aug 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.79%. Comparing base (3ef4ab8) to head (e018cb9).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #17   +/-   ##
=======================================
  Coverage   69.79%   69.79%           
=======================================
  Files           3        3           
  Lines          96       96           
=======================================
  Hits           67       67           
  Misses         29       29           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@brews
brews marked this pull request as ready for review August 4, 2026 16:20
@brews

brews commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

After talking with the group, I'm holding off on merging this because it has breaking changes to #16. Otherwise, this should be ready to go, AFAIK.

I need to switch to work on another project, and I'll be out for a week+ on Friday. I'm moving the assignment to @kemccusker so she has the power.

@brews brews assigned kemccusker and unassigned brews Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants