Skip to content

Document the calibration method and add the dataset smoke test - #346

Closed
soaressgabriel wants to merge 12 commits into
feat/343-calibratefrom
feat/345-calibration-docs
Closed

soaressgabriel wants to merge 12 commits into
feat/343-calibratefrom
feat/345-calibration-docs

Conversation

@soaressgabriel

Copy link
Copy Markdown
Collaborator

Checklist

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?
  • Added tests for changed code.
  • Updated documentation for changed code.

Description

Stacked on #344 (base feat/343-calibrate).

  • doc/source/calibration.rst, a new page in the toctree between the preprocessing and the API pages: what the command fits and against what; the objective (the Nash-Sutcliffe efficiency over the pairs valid in both series, the masked values, the two guards, the average over stations, the spin-up window, FO = 1000 (100 (1 - NSE))^2); the eight free parameters with the bounds of appsettings.json, w3 = 1 - (w1 + w2), the constraint and the starting point; the differential evolution settings with a reason for each and the evaluation budget (max(5, 8 popsize) rounded up to a power of two for the Sobol' initialization; 128 members and up to 12,928 runs with the defaults, 64 members with --popsize 5; the convergence test SciPy applies); the observed series formats and the fixed drainage network a calibration on arn requires (RASTERS.ldd, with the PCRaster recipe); the artifacts; the process model (one spawned interpreter per evaluation, silenced workers, failures recorded per evaluation); running on a cluster (--workers, memory, --temp-dir, shorter windows, continuing from a calibrated configuration). Linked from the user guide's command section and from the API page, which states that rubem.calibration is not part of the stable surface yet.
  • dataset pytest marker and the RUBEM_DATASET_DIR convention in tests/conftest.py: tests marked dataset are skipped, with the reason, when the variable is unset or does not name a directory, whether or not they request the dataset_dir fixture; the suite stays self-contained in CI.
  • tests/dataset/test_calibrate_ipojuca.py: the manifest of the Ipojuca basin is rebased onto the local copy, the window restricted to six monthly steps, the drainage network generated once and named in the configuration, GDAL's auxiliary metadata disabled so nothing is written beside the dataset; a base run with the manifest's own parameters gives the observed series and a one-generation differential evolution from a five-member initial population with two workers finds that configuration again: best objective exactly 0.0 (without the fixed network it is 4.4e-7, which is what the exact comparison guards), the artifacts consistent, no process left behind.
  • Changelog entry under Added.

Related Issue

Motivation and context

How has this been tested

  • python -m pytest -q -p no:cacheprovider --ignore=tests/integration/doc -n 2 without RUBEM_DATASET_DIR: 1301 passed, 2 skipped (the smoke skipped with its reason).
  • RUBEM_DATASET_DIR=<local datasets> python -m pytest -q -p no:cacheprovider tests/dataset: 1 passed in about 10 s (Ipojuca, six steps, ten evaluations on two workers).
  • Every statement of the page was checked a second time against the code and the installed SciPy source (formulas, defaults, bounds, budget arithmetic, artifact fields, the LDD requirement).
  • uvx ruff@0.16.4 format --check . and uvx ruff@0.16.4 check .: clean.
  • sphinx-build -W -b html doc/source with the documentation's Python 3.13 environment: no warnings.

Screenshots

  • N/A

rubem.calibration.objective computes the Nash-Sutcliffe efficiency over
the pairs valid in both series (NaN, -9999 and the PCRaster missing value
masked, a zero-variance guard), the objective 1000 (100 (1 - NSE))^2 and
the alignment of a simulated and an observed series on their shared time
steps and station ids after a spin-up window; read_series reads the CSV
layout of the model's time series tables and PCRaster TSS files.
parameters.py maps the eight free parameters to the decision vector, with
w_3 derived from the other two weights, the bounds of the application
settings and the linear constraint w_1 + w_2 <= 1. _worker.py is the
module-level entry point a spawned worker runs: it builds the candidate's
format 1.0 document with every raster series disabled and only the target
time series enabled, runs the model in-process through the Python API,
compares the table with the observed series and records one JSON file
atomically; failures are recorded and ranked last. runner.py loads and
validates the configuration once, drives SciPy's differential_evolution
over a spawn ProcessPoolExecutor with one task per silenced worker, and
writes evaluations.csv, result.json and the calibrated configuration.
The command takes the configuration, the observed series and the run
directory, exposes the search settings as options, prints the documented
progress lines and exits without a traceback on configuration and
calibration errors. rubem._deps gains the optional calibration
dependency guard, with the guidance to install rubem[calibration].
Add the rubem[calibration] extra (scipy>=1.16.2, the first release with
CPython 3.14 wheels on Linux and Windows), include it in the dev extra
and in the CI test installation, commit the ci/dev-requirements.in the
audit lock was compiled from, with scipy added, and recompile the lock
keeping every existing pin.
…search

An observed series that shares no time step with the simulated steps
after the spin-up, or no station id with the sample locations raster,
makes every evaluation fail after its simulation has run, so a wrong
file cost a whole generation of model runs before the calibration
failed. The parent now settles both cases before the run directory is
prepared and before any worker starts, warns once about observed
stations the configuration does not sample, and skips the station check
for the zones aggregation, whose ids are only known at run time. The
reproducibility statement of the runner is qualified: ordering the rows
by candidate removes the scheduling order, but two runs of one seed
evaluate the same candidates only when the simulation itself is
reproducible, which on the real basins needs a fixed LDD raster.
Tests marked dataset need the local copy of the published datasets, which
the environment variable names; when it is unset, or does not name a
directory, they are skipped with the reason, whether or not they request
the dataset_dir fixture, so the suite stays self-contained elsewhere.
Rebase the manifest onto the local copy, restrict the window to six
monthly steps, generate the drainage network once and name it in the
configuration (the accumulated runoff is routed on a network lddcreate
does not reproduce between calls), disable GDAL's auxiliary metadata so
nothing is written beside the dataset, take the series of a base run as
the observations, and check that a one-generation differential evolution
from five members finds the configuration again with an objective of
exactly zero, consistent artifacts and no process left behind.
Add the calibration page: what the command fits, the objective and its
masks and guards, the free parameters with their bounds and the derived
weight, the differential evolution settings and the evaluation budget,
the observed series formats and the fixed drainage network a calibration
on the accumulated runoff needs, the artifacts, the process model and
hints for large machines; link it from the user guide and the API page.
…arch

The calibration page said every mismatch between the observations and
the configuration surfaced inside the evaluations; the runner now refuses
a series with no shared time step or no shared station before the search
starts, and only a series whose shared stations all lack an efficiency
still fails each evaluation after its simulation.
@codecov

codecov Bot commented Sep 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.58%. Comparing base (deb85e2) to head (5d0d77f).

Additional details and impacted files
@@                 Coverage Diff                 @@
##           feat/343-calibrate     #346   +/-   ##
===================================================
  Coverage               92.58%   92.58%           
===================================================
  Files                      65       65           
  Lines                    4573     4573           
  Branches                  557      557           
===================================================
  Hits                     4234     4234           
  Misses                    270      270           
  Partials                   69       69           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@soaressgabriel

Copy link
Copy Markdown
Collaborator Author

Superseded by #344, which now carries the calibration page, the dataset marker and the Ipojuca smoke test as part of one pull request for the calibration command (issue #343).

@soaressgabriel
soaressgabriel deleted the feat/345-calibration-docs branch October 8, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document the calibration method and add the dataset smoke test

1 participant