Skip to content

Bring rubem calibrate to parity with the previous calibration workflow #348

Closed as duplicate

Description

@soaressgabriel

Description

A comparison of rubem calibrate (#343, #345) with the calibration workflow the model was calibrated with before (a SciPy differential evolution driving one model process per evaluation, with a spreadsheet handoff and per-station workbooks) found capabilities of that workflow the command does not offer yet, and definitions that differ in ways a user of the previous workflow would notice.

Proposed solution

Required for parity:

  • Per-parameter bounds for a run (--bound NAME=MIN:MAX, repeatable; CalibrationSettings.bounds), narrowing the ranges of appsettings.json only, recorded in result.json. The previous workflow searched alpha in 1 to 10, the weights in 0.1 to 1 and RCD in 0.1 to 10.
  • Fixed parameters (--fix NAME=VALUE, repeatable; CalibrationSettings.fixed): the coordinate leaves the decision vector, the value is kept in the nine emitted parameters and in result.json. The previous workflow pinned x at 0.
  • Negative observed values are missing values, as the previous workflow's obs >= 0 mask treated them; the pre-flight reports, per station, how many observed values the mask dropped.
  • Progress on the terminal: the budget line, one line per generation (best objective and best NSE) and the final summary go through the rubem.progress logger the command line already routes to standard output, as rubem run does.

Also delivered, from the same comparison:

  • Observed file without a header (;-separated, first cell of the first line not the step column label) is refused with a message naming the expected layouts (the model's 0;<id>;... table or a PCRaster time series with its header); PCRaster time series files without the header are refused too (Write the PCRaster header in the time series outputs #347 makes the model write the header).
  • Per-station goodness-of-fit in every evaluation record (valid pairs, mean and standard deviation of both series, Pearson correlation, RMSE, NSE); stations.csv and best_<variable>.csv for the best candidate (its simulated series beside the observed one) in the run directory.
  • --stations ID[,ID...]: the stations that enter the objective; the others are still reported, which gives the calibration/validation split the previous workflow obtained by averaging three of its four stations.
  • observed.csv in the run directory: per station, the valid pairs against the compared steps and the summary statistics of the observations, plus the coverage of the compared window logged before the search.
  • A run interrupted or crashed after the search started still gets its evaluations.csv; a worker killed by the system (out of memory) is reported with the advice to lower --workers.
  • A wall-clock started_at column in the evaluation records and table, so the order of evaluation and the convergence curve can be reconstructed.
  • --init, --strategy and --polish/--no-polish on the command line, to reproduce the previous settings (latinhypercube, polishing on) when wanted.
  • The calibration page records the definitions that changed on purpose (exact weights derivation instead of a penalty with a 0.01 tolerance, steps counted as months, a point station must mark one cell, no polishing by default, the new budget arithmetic, memory per worker) and how to find the per-station efficiency of the winner.

Alternative solutions

Accepting the head-less observed file by numbering its columns 1..N would silently mislabel stations whenever a header is present but malformed; refusing it with the expected layouts is safer and the conversion is one header line.

Additional context

Follows #343 and #345; depends on #347 for the header of the time series the model writes.

Activity

  1. soaressgabriel commented on Sep 18, 2026

    @soaressgabriel
    CollaboratorAuthor

    Absorbed into #343, which is now the single issue for the calibration command; every item listed here is part of its scope and is delivered by #344.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions