Skip to content

statistical_test.md

Latest commit

 

History

History
47 lines (36 loc) · 2.31 KB

statistical_test.md

File metadata and controls

47 lines (36 loc) · 2.31 KB
Raw

Statistical Comparison (autorank)

statistical_test.py runs a paired statistical comparison across multiple models' per-case metric CSVs using autorank. It merges the CSVs by case ID, prints mean performance per model, runs the ranking test, and saves the paired data, a textual report, a LaTeX table and a result plot.

Each input CSV must hold per-case results (one row per case). Note that the test_metrics.csv written by the training framework is per-epoch, not per-case, so it cannot be used directly (see testing.md).

How To Run

python statistical_test.py \
  --csv-files /path/to/ModelA.csv /path/to/ModelB.csv /path/to/ModelC.csv \
  --metric-column DSC_aggregated_mean \
  --id-column subject_id \
  --order descending \
  --output-prefix autorank_results \
  --output-dir /path/to/results

Options

Option Default Description
--csv-files required One CSV per model (at least 2).
--metric-column required Metric column to compare, e.g. DSC_aggregated_mean or DSC_mean.
--id-column subject_id Case ID column used to pair rows across models.
--alpha 0.05 Significance level.
--order descending descending when higher is better, ascending when lower is better.
--output-prefix autorank_results Prefix of all output file names.
--output-dir . Directory for all outputs (created if missing).

Input rules

  • Each CSV has one row per case, identified by --id-column. An AVERAGE row, if present, is dropped automatically.
  • The model name is the CSV filename without extension, so filenames must be unique.
  • Rows whose metric is missing or not numeric are dropped.
  • Duplicate case IDs within a CSV raise an error.
  • Only cases present in every CSV are compared (inner join); the number of paired cases is printed.

Outputs

Written to --output-dir:

  • <prefix>_paired_data.csv: the merged table used for the test (case ID + one column per model).
  • <prefix>_report.txt: autorank's textual report.
  • <prefix>_table.tex: LaTeX table of the results.
  • <prefix>_plot.png: critical-difference / confidence-interval plot.

The mean per model, the full autorank result, the report and the LaTeX table are also printed to stdout.