Pre-register the YOLO-baseline evaluation & checkpoint-selection protocol (#71)

docs/model_comparison.md

@@ -43,8 +43,13 @@ ckpt preemptions — `y11l_pano` and `y26_pano` share batch 4 / imgsz 1280 and b
 recovered. Figures and the full ruling-out are in the training record; the stabilized rerun is
 tracked in #70 (whose original small-batch hypothesis this evidence refutes) and the caveat
 write-up in #72. The reported baseline will be the best-val checkpoint, selected on val, per the
-protocol in #71 — with the caveat that today's checkpoints are undertrained relative to a stable
-schedule and so give a **lower bound** on supervised-YOLO performance.
+**pre-registered protocol** committed in the
+[training record](../scripts/model_comparison/yolo_baseline/README.md#pre-registered-evaluation--checkpoint-selection-protocol-issue-71)
+(#71) *before* any benchmark eval of any YOLO checkpoint: `best.pt` = Ultralytics best-val-fitness
+with `patience=20`, config choice on val only, headline F1 at the Ultralytics-default conf 0.25,
+same matcher/GT/radius as the whole roster, test bundles touched once — with the caveat that
+today's checkpoints are undertrained relative to a stable schedule and so give a **lower bound**
+on supervised-YOLO performance.
 
 Three classes of challenger, which fail differently and are worth keeping distinct:
 

scripts/model_comparison/compare.py

@@ -16,9 +16,14 @@
 Each --models token is a provider (rampnet/gemini/qwen/owlv2/gdino/molmo) or
 provider:model_id to pin a variant, so several models from the same provider
 compare side by side. Detectors that emit calibrated scores (RampNet, OWLv2,
-Grounding DINO) additionally get AP, a PR curve (--pr-out) and a threshold sweep
-(--sweep); chat VLMs have no score to rank by, so they get one operating point.
-See docs/model_comparison.md.
+Grounding DINO, YOLO) additionally get AP, a PR curve (--pr-out) and a threshold
+sweep (--sweep); chat VLMs have no score to rank by, so they get one operating
+point. See docs/model_comparison.md.
+
+The supervised YOLO baseline (--models yolo:<best.pt>) is evaluated under the
+pre-registered checkpoint-selection & eval protocol in
+scripts/model_comparison/yolo_baseline/README.md (issue #71): checkpoint and
+config chosen on val only, headline F1 at conf 0.25, test bundles touched once.
 """
 import argparse
 import hashlib

scripts/model_comparison/yolo_baseline/README.md

@@ -30,7 +30,8 @@ in git, bulk/binary/regenerable out):
   the few relevant lines are quoted below.
 - **Benchmark-city eval numbers** — these `results.csv` values are the **internal YOLO
   val-split proxy** used to judge *training health*, **not** the reported baseline. The
-  reported numbers come from local `compare.py` benchmark eval on each `best.pt`.
+  reported numbers come from local `compare.py` benchmark eval on each `best.pt`, under
+  the pre-registered protocol below.
 
 ## Config grid
 
@@ -116,6 +117,68 @@ The honest caveat: these checkpoints are **undertrained** relative to a stable s
 any benchmark number from them is a **lower bound** on supervised-YOLO performance, and must
 be reported as such.
 
+## Pre-registered evaluation & checkpoint-selection protocol (issue #71)
+
+Fixed in writing 2026-07-28, **before any benchmark evaluation of any YOLO
+checkpoint**. As of the commit adding this section, no YOLO checkpoint has been scored
+on any benchmark bundle (city or `manual_gold`); the only numbers observed are the
+internal val-split curves above (`runs/*/results.csv`, mirrored in #51's status
+comments). The point is to make the baseline defensible against the standard
+"you sandbagged / cherry-picked the baseline" review: every selection decision is made
+on validation data, under rules written down first, applied identically to the
+stabilized rerun (#70) when it lands.
+
+1. **Checkpoint selection — best-val, never test.** Each config's reported checkpoint
+   is its `best.pt` exactly as Ultralytics saves it: highest **fitness** (default
+   weighting, 0.1·mAP50 + 0.9·mAP50-95) on that config's own YOLO val split, with
+   `patience=20`. This refines #71's draft wording ("best val mAP50"): `best.pt` is
+   the only per-epoch artifact the runs keep, so re-ranking `results.csv` by raw mAP50
+   would mean a second, hand-rolled selector — and on every curve committed so far the
+   two rules choose the same epoch anyway (all five configs: epoch 1, checked
+   2026-07-28). RampNet's published checkpoint was selected by the same principle:
+   best **validation loss** on the dataset's val split (`stage_two/train.py`), never a
+   benchmark number.
+2. **Config selection — on val, and it controls emphasis only.** Which config
+   headlines each family (YOLO11 vs YOLO26) in the main text is decided by the same
+   internal val fitness at `best.pt`. Stated caveat: tiles and pano configs have
+   different val sets (perspective crops vs whole equirects), so the cross-geometry
+   comparison is a judgment call on non-identical data — which is why this choice is
+   only about emphasis. **Every trained config's benchmark row goes in the appendix /
+   full table regardless; selection drops nothing.** Test is touched once, at the end,
+   after these choices are fixed.
+3. **Benchmark eval — the identical path as every roster model.** `compare.py`'s
+   `yolo` provider: boxes → box centers → the `rampnet/detection_eval.py` matcher at
+   radius 0.022, on the same bundles and the same GT as the rest of the roster,
+   detections cached at the 0.05 floor (`--yolo-conf 0.05`). Tiles configs are scored
+   through the same perspective rig as the VLMs (`--tiling perspective`,
+   `--yolo-imgsz 1024`); pano configs whole-pano (`--tiling none --yolo-imgsz 1280`) —
+   each model in its training geometry.
+4. **Operating point — fixed in advance, not tuned on test.** The headline metric is
+   **F1 at confidence 0.25**, Ultralytics' default predict confidence — the
+   "configure the baseline the way its authors recommend" choice, fixed here while no
+   test number exists. The full threshold sweep is still reported, with its best-F1
+   row flagged as tune-on-test, exactly as the harness already does for every
+   confidence-carrying model.
+5. **AP, with the truncation caveat.** AP/PR come from the full cached range (0.05
+   floor). They are **not** directly comparable to RampNet's committed AP, whose
+   detections were extracted at the 0.5 peak threshold (a truncated curve — see
+   `docs/model_comparison.md`); the low-floor RampNet extraction is the #54/#78 line
+   of work.
+6. **Reporting an unstable run.** A checkpoint from inside the collapse dip would
+   understate recall catastrophically (see the failure signature above); best-val
+   selection avoids the dip by construction. A `best.pt` from an instability-affected
+   run **is** reportable, but only with the standing caveat: it is a **lower bound**
+   on supervised-YOLO performance, superseded when the #70 stabilized rerun lands —
+   which will be selected, evaluated, and reported under this same protocol,
+   unchanged.
+7. **Seeds (aspirational).** All runs are `seed=0`. If ckpt capacity allows, ≥3 seeds
+   of the headline configs → mean ± std, which makes single-run instability commentary
+   moot. Not a blocker for reporting the lower-bound numbers.
+
+The paper's "Baseline protocol" appendix paragraph is this section, condensed.
+`docs/model_comparison.md` links here from its baseline-in-progress note, and the
+protocol governs the results that will replace that note.
+
 ## Provenance
 
 - **Training code:** `scripts/model_comparison/run_yolo_train.slurm` + the repo-root