Repository navigation
Document the calibration method and add the dataset smoke test - #346
Closed
soaressgabriel wants to merge 12 commits into
Closed
soaressgabriel wants to merge 12 commits into
soaressgabriel wants to merge 12 commits into
Conversation
rubem.calibration.objective computes the Nash-Sutcliffe efficiency over the pairs valid in both series (NaN, -9999 and the PCRaster missing value masked, a zero-variance guard), the objective 1000 (100 (1 - NSE))^2 and the alignment of a simulated and an observed series on their shared time steps and station ids after a spin-up window; read_series reads the CSV layout of the model's time series tables and PCRaster TSS files.
parameters.py maps the eight free parameters to the decision vector, with w_3 derived from the other two weights, the bounds of the application settings and the linear constraint w_1 + w_2 <= 1. _worker.py is the module-level entry point a spawned worker runs: it builds the candidate's format 1.0 document with every raster series disabled and only the target time series enabled, runs the model in-process through the Python API, compares the table with the observed series and records one JSON file atomically; failures are recorded and ranked last. runner.py loads and validates the configuration once, drives SciPy's differential_evolution over a spawn ProcessPoolExecutor with one task per silenced worker, and writes evaluations.csv, result.json and the calibrated configuration.
The command takes the configuration, the observed series and the run directory, exposes the search settings as options, prints the documented progress lines and exits without a traceback on configuration and calibration errors. rubem._deps gains the optional calibration dependency guard, with the guidance to install rubem[calibration].
Add the rubem[calibration] extra (scipy>=1.16.2, the first release with CPython 3.14 wheels on Linux and Windows), include it in the dev extra and in the CI test installation, commit the ci/dev-requirements.in the audit lock was compiled from, with scipy added, and recompile the lock keeping every existing pin.
…search An observed series that shares no time step with the simulated steps after the spin-up, or no station id with the sample locations raster, makes every evaluation fail after its simulation has run, so a wrong file cost a whole generation of model runs before the calibration failed. The parent now settles both cases before the run directory is prepared and before any worker starts, warns once about observed stations the configuration does not sample, and skips the station check for the zones aggregation, whose ids are only known at run time. The reproducibility statement of the runner is qualified: ordering the rows by candidate removes the scheduling order, but two runs of one seed evaluate the same candidates only when the simulation itself is reproducible, which on the real basins needs a fixed LDD raster.
Tests marked dataset need the local copy of the published datasets, which the environment variable names; when it is unset, or does not name a directory, they are skipped with the reason, whether or not they request the dataset_dir fixture, so the suite stays self-contained elsewhere.
Rebase the manifest onto the local copy, restrict the window to six monthly steps, generate the drainage network once and name it in the configuration (the accumulated runoff is routed on a network lddcreate does not reproduce between calls), disable GDAL's auxiliary metadata so nothing is written beside the dataset, take the series of a base run as the observations, and check that a one-generation differential evolution from five members finds the configuration again with an objective of exactly zero, consistent artifacts and no process left behind.
Add the calibration page: what the command fits, the objective and its masks and guards, the free parameters with their bounds and the derived weight, the differential evolution settings and the evaluation budget, the observed series formats and the fixed drainage network a calibration on the accumulated runoff needs, the artifacts, the process model and hints for large machines; link it from the user guide and the API page.
…arch The calibration page said every mismatch between the observations and the configuration surfaced inside the evaluations; the runner now refuses a series with no shared time step or no shared station before the search starts, and only a series whose shared stations all lack an efficiency still fails each evaluation after its simulation.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## feat/343-calibrate #346 +/- ##
===================================================
Coverage 92.58% 92.58%
===================================================
Files 65 65
Lines 4573 4573
Branches 557 557
===================================================
Hits 4234 4234
Misses 270 270
Partials 69 69 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
4 tasks done
4 tasks done
soaressgabriel
force-pushed
the
feat/343-calibrate
branch
from
September 18, 2026 19:15
deb85e2 to
e88ed52
Compare
4 tasks done
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checklist
Description
Stacked on #344 (base
feat/343-calibrate).doc/source/calibration.rst, a new page in the toctree between the preprocessing and the API pages: what the command fits and against what; the objective (the Nash-Sutcliffe efficiency over the pairs valid in both series, the masked values, the two guards, the average over stations, the spin-up window,FO = 1000 (100 (1 - NSE))^2); the eight free parameters with the bounds ofappsettings.json,w3 = 1 - (w1 + w2), the constraint and the starting point; the differential evolution settings with a reason for each and the evaluation budget (max(5, 8 popsize)rounded up to a power of two for the Sobol' initialization;128members and up to12,928runs with the defaults,64members with--popsize 5; the convergence test SciPy applies); the observed series formats and the fixed drainage network a calibration onarnrequires (RASTERS.ldd, with the PCRaster recipe); the artifacts; the process model (one spawned interpreter per evaluation, silenced workers, failures recorded per evaluation); running on a cluster (--workers, memory,--temp-dir, shorter windows, continuing from a calibrated configuration). Linked from the user guide's command section and from the API page, which states thatrubem.calibrationis not part of the stable surface yet.datasetpytest marker and theRUBEM_DATASET_DIRconvention intests/conftest.py: tests markeddatasetare skipped, with the reason, when the variable is unset or does not name a directory, whether or not they request thedataset_dirfixture; the suite stays self-contained in CI.tests/dataset/test_calibrate_ipojuca.py: the manifest of the Ipojuca basin is rebased onto the local copy, the window restricted to six monthly steps, the drainage network generated once and named in the configuration, GDAL's auxiliary metadata disabled so nothing is written beside the dataset; a base run with the manifest's own parameters gives the observed series and a one-generation differential evolution from a five-member initial population with two workers finds that configuration again: best objective exactly0.0(without the fixed network it is4.4e-7, which is what the exact comparison guards), the artifacts consistent, no process left behind.Related Issue
Motivation and context
How has this been tested
python -m pytest -q -p no:cacheprovider --ignore=tests/integration/doc -n 2withoutRUBEM_DATASET_DIR: 1301 passed, 2 skipped (the smoke skipped with its reason).RUBEM_DATASET_DIR=<local datasets> python -m pytest -q -p no:cacheprovider tests/dataset: 1 passed in about 10 s (Ipojuca, six steps, ten evaluations on two workers).uvx ruff@0.16.4 format --check .anduvx ruff@0.16.4 check .: clean.sphinx-build -W -b html doc/sourcewith the documentation's Python 3.13 environment: no warnings.Screenshots