Skip to content

Add the rubem calibrate command - #344

Merged
camilabillerbeck merged 11 commits into
mainfrom
feat/343-calibrate
Sep 22, 2026
Merged

camilabillerbeck merged 11 commits into
mainfrom
feat/343-calibrate

Conversation

@soaressgabriel

@soaressgabriel soaressgabriel commented Sep 16, 2026 •

Copy link
Copy Markdown
Collaborator

Checklist

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?
  • Added tests for changed code.
  • Updated documentation for changed code.

Description

Based on #342 (feat/341-python-api): the workers run the model through rubem.api.Model. This pull request carries the whole calibration command; it replaces #346 and #350, whose work is folded in here. The series is meant to be reviewed commit by commit:

  1. Package SciPy as the optional calibration extra: rubem[calibration] (scipy>=1.16.2, the first release with CPython 3.14 wheels on Linux and Windows), in the dev extra and in the CI test installation; the dependency guard in rubem._deps; ci/dev-requirements.in committed (the lock header pointed at an uncommitted file) and ci/dev-requirements.lock recompiled keeping every pin (scipy==1.18.1, numpy==2.5.3 added; pip-audit --strict clean).
  2. Objective, station metrics and series readers (rubem/calibration/objective.py): the NSE over the pairs valid in both series (NaN, negative values, -9999 and the PCRaster missing value are gaps), the objective 1000 (100 (1 - NSE))^2, StationMetrics per shared station (pairs, means, standard deviations, Pearson r, RMSE, NSE), evaluate_series (alignment on shared steps and ids, spin-up, a chosen selection of stations), read_series for the model's table layout and for PCRaster time series with their header; head-less files are refused.
  3. Decision space (parameters.py): eight free parameters, w_3 derived, w_1 + w_2 <= 1; per-run bounds that only narrow the settings ranges; fixed parameters that leave the vector; the weight cases (one fixed, one collapsed to a value, both fixed).
  4. Runner (_worker.py, runner.py): one spawned, silenced worker per evaluation running Model.run() in-process on a derived format 1.0 document; atomic JSON records; differential_evolution (best1exp, sobol, x0, no polishing, deferred) over executor.map; the pre-flight of the observed series; observed.csv, evaluations.csv, stations.csv, best_<variable>.csv, result.json and the calibrated configuration; progress through rubem.progress; interrupt and out-of-memory handling.
  5. Command (rubem calibrate): options for the variable, spin-up, generations, population, seed, workers, temporary directory, --bound, --fix, --stations, --init, --strategy, --polish/--no-polish; the progress and the closing summary; exit codes as rubem run.
  6. Dataset marker and smoke test: dataset marker, RUBEM_DATASET_DIR with a skip reason, and the Ipojuca smoke (six steps, a drainage network generated once, best objective exactly 0.0; without the fixed network it is 4.4e-7).
  7. Documentation: the calibration page, the command in the user guide, the entry point in the API page.
  8. Changelog: one entry under Added.

Related Issue

Motivation and context

  • The corrected soil moisture coefficient (Fix units soil moisture equation for ch #321) invalidated the published parameter sets of the three basins (Pin dataset provenance and normalization for integration testing #217), and the model had no reproducible calibration: each parameter was edited by hand. The command implements the method the model was designed around, from the SciPy API and the Nash-Sutcliffe definition, with the process isolation PCRaster requires, per-run control of the search box, a chosen set of stations, per-station goodness-of-fit and visible progress.

Pending, tracked in #343: the rejection of candidates whose weights violate C_wp <= 1 needs the check merged by #336, which this branch does not contain until it is rebased onto main. Design notes left in the code: a repeated time step of an observed table is aligned on its first row; a BrokenProcessPool in the final re-run of the best candidate is reported without the --workers advice (that pool runs one task); the per-generation progress re-reads the records (negligible beside the model runs).

How has this been tested

  • The tree of this branch is the one verified on the former stack (Bring rubem calibrate to parity with the previous calibration workflow #350), byte for byte, except the changelog, whose three entries became one: git diff --stat against that head lists doc/source/changelog.rst only.
  • python -m pytest -q -p no:cacheprovider --ignore=tests/integration/doc -n 2: 1445 passed, 2 skipped (Linux, Python 3.13, PCRaster 4.4.2, SciPy 1.18.1). Every numeric expectation was re-derived by hand a second time; mutation checks (record written before the run, constraint dropped, w_3 ignoring w_2, mask accepting negatives, record without station metrics, constraint kept with a fixed weight) are caught.
  • End to end through the command line on the synthetic dataset, with the default options (best objective 0, 70 evaluations in one generation) and with --fix x=0 --bound alpha=1:10 --stations 1 --init latinhypercube (7 free parameters, artifacts consistent: stations.csv recomputed from best_arn.csv bit for bit); no lingering process.
  • RUBEM_DATASET_DIR=<local datasets> python -m pytest tests/dataset: 1 passed in about 13 s.
  • uvx ruff@0.16.4 format --check . and uvx ruff@0.16.4 check .: clean. sphinx-build -W -b html doc/source with the documentation's Python 3.13 environment: no warnings. uvx pip-audit --strict -r ci/dev-requirements.lock: no known vulnerabilities.

Screenshots

  • N/A

@codecov

codecov Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.76366% with 38 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.36%. Comparing base (dc80787) to head (99023b4).

Files with missing lines Patch % Lines
rubem/calibration/runner.py 95.40% 12 Missing and 5 partials ⚠️
rubem/calibration/_worker.py 90.90% 7 Missing and 2 partials ⚠️
rubem/calibration/objective.py 95.58% 4 Missing and 4 partials ⚠️
rubem/calibration/parameters.py 98.50% 1 Missing and 1 partial ⚠️
rubem/cli.py 97.87% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #344      +/-   ##
==========================================
+ Coverage   92.87%   93.36%   +0.48%     
==========================================
  Files          64       68       +4     
  Lines        4365     5260     +895     
  Branches      558      675     +117     
==========================================
+ Hits         4054     4911     +857     
- Misses        248      274      +26     
- Partials       63       75      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Add the rubem[calibration] extra (scipy>=1.16.2, the first release with
CPython 3.14 wheels on Linux and Windows), include it in the dev extra
and in the CI test installation, and give rubem._deps the guard of the
optional dependency, whose guidance names the extra. Commit the
ci/dev-requirements.in the audit lock was compiled from, with scipy
added, and recompile the lock keeping every existing pin.
…ders

rubem.calibration.objective computes the Nash-Sutcliffe efficiency over
the pairs valid in both series (NaN, negative values, -9999 and the
PCRaster missing value are gaps; fewer than two pairs or a constant
observed record give no value), the objective 1000 (100 (1 - NSE))^2 and,
per shared station, the pairs, the means and standard deviations of both
series, the Pearson correlation, the RMSE and the NSE. evaluate_series
aligns a simulated and an observed series on their shared time steps and
station ids after a spin-up window and averages the NSE over a chosen
selection of stations while measuring the others. read_series reads the
table layout of the model's time series ("0;<id>;...") and PCRaster time
series with their header; a file without a header is refused naming the
two accepted layouts, and a byte order mark is ignored.
The search runs over the eight free calibration parameters, with w_3
derived as 1 - w_1 - w_2 and the linear constraint w_1 + w_2 <= 1.
decision_space(fixed, bounds) builds the vector of one run: a fixed
parameter leaves the vector and is re-inserted in the nine parameters
every candidate runs with; a per-run bound may only narrow the range of
the application settings; fixing one weight narrows the other so the
derived w_3 stays in range and drops the pair constraint, a narrowing
that leaves it one value fixes it as well, and fixing both weights
checks w_3 before the search.
_worker.evaluate is the module-level entry point a spawned worker runs:
an inadmissible candidate is rejected without a run; otherwise the
format 1.0 document of the configuration is copied with the candidate's
parameters, every raster series disabled and only the target time series
enabled into a temporary directory, the model runs in-process through
the Python API, the table is compared with the observed series and one
JSON record (parameters, NSE, per-station metrics, objective, start
time, error) is written atomically; failures are recorded and ranked
last. runner.calibrate loads and validates the configuration once,
refuses an observed series that shares no time step or station with what
the configuration samples, writes observed.csv, and drives SciPy's
differential_evolution over a spawn ProcessPoolExecutor with one task
per silenced worker. It then consolidates evaluations.csv, runs the best
candidate once more to write best_<variable>.csv and stations.csv, and
writes result.json and the calibrated configuration in the input's own
format. Progress goes through the rubem.progress logger; an interrupted
or failed search still consolidates the evaluations, and a worker killed
by the system is reported with the advice to lower the worker count.
The command takes the configuration, the observed series and the run
directory, and exposes the search as options: the variable, the spin-up
window, the generations, the population, the seed, the workers, the
temporary directory, per-run bounds (--bound NAME=MIN:MAX), fixed
parameters (--fix NAME=VALUE), the stations of the objective, the
initialization, the strategy and the polishing. It prints the progress
lines and a closing summary listing every artifact, and exits without a
traceback on configuration and calibration errors.
…basin

Tests marked dataset need the local copy of the published datasets, which
RUBEM_DATASET_DIR names; when it is unset, or does not name a directory,
they are skipped with the reason, so the suite stays self-contained
elsewhere. The smoke test rebases the manifest of the Ipojuca basin onto
the local copy, restricts the window to six monthly steps, generates the
drainage network once and names it in the configuration (the accumulated
runoff is routed on a network lddcreate does not reproduce between
calls), takes the series of a base run as the observations, and checks
that a one-generation differential evolution from five members finds the
configuration again with an objective of exactly zero.
Add the calibration page: what the command fits, the objective with its
masks and guards, the free parameters with their bounds, the per-run
bounds and fixed parameters, the derived weight, the differential
evolution settings and the evaluation budget, the observed series
layouts and the fixed drainage network a calibration on the accumulated
runoff needs, the artifacts, the process model, what a run prints and
hints for large machines. Document the command in the user guide and
name the Python entry point in the API page.
soaressgabriel and others added 3 commits September 22, 2026 10:45
# Conflicts:
#	doc/source/api.rst
#	pyproject.toml
# Conflicts:
#	doc/source/changelog.rst
#	rubem/cli.py
- Introduced the --allow-blocking-problems flag in the CLI for the calibrate command to allow searches even with input validation errors.
- Updated CalibrationSettings to include allow_blocking_problems parameter, and related documentation in calibration.rst, changelog.rst, and userguide.rst to reflect the new functionality.
- Added unit tests to verify the behavior of the new option in both the calibration runner and CLI.
@camilabillerbeck
camilabillerbeck merged commit 87818d9 into main Sep 22, 2026
17 checks passed
@camilabillerbeck
camilabillerbeck deleted the feat/343-calibrate branch September 22, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add the rubem calibrate command (differential evolution on the station streamflow)

2 participants