From df90ec97b8cd672813410e523eab5c758cfa8431 Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Tue, 28 Jul 2026 22:33:53 -0700 Subject: [PATCH] Pre-register the YOLO-baseline eval & checkpoint-selection protocol (#71) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Committed before any benchmark evaluation of any YOLO checkpoint — the five recovery runs (jobs 37745358-363) are still training on ckpt-g2 and the only numbers observed anywhere are the internal val-split curves already in this record. That timing is the whole point: selection rules written down first, test bundles touched once. The protocol section in the training record fixes, in writing: 1. Checkpoint selection: best.pt as Ultralytics saves it (best val fitness, 0.1*mAP50 + 0.9*mAP50-95, patience=20) — refines #71's "best val mAP50" draft to the artifact the runs actually keep; verified both rules pick the same epoch (1) on all five committed curves. RampNet parity noted: its checkpoint was likewise best-val (loss), never a benchmark number. 2. Config/headline selection on val only; controls emphasis only — every trained config's benchmark row is reported regardless. 3. Benchmark eval through the identical compare.py path as the whole roster (boxes -> centers -> detection_eval matcher, radius 0.022, 0.05 cache floor; tiles via the perspective rig at 1024, pano whole-pano at 1280). 4. Headline operating point fixed in advance: F1 at conf 0.25, Ultralytics' own default — not tuned on test. Sweep still reported with the tune-on-test flag as usual. 5. AP with the standing truncation caveat vs RampNet's 0.5-peak extraction. 6. Instability-affected best.pt is reportable only as a lower bound, superseded by the #70 rerun under this same protocol, unchanged. 7. Seeds aspirational (all runs seed=0). Cross-referenced from compare.py's module docstring and from the baseline-in-progress note in docs/model_comparison.md, per the issue's deliverable list. Co-Authored-By: Claude Fable 5 --- docs/model_comparison.md | 9 ++- scripts/model_comparison/compare.py | 11 +++- .../model_comparison/yolo_baseline/README.md | 65 ++++++++++++++++++- 3 files changed, 79 insertions(+), 6 deletions(-) diff --git a/docs/model_comparison.md b/docs/model_comparison.md index 2822b46..147fa2a 100644 --- a/docs/model_comparison.md +++ b/docs/model_comparison.md @@ -43,8 +43,13 @@ ckpt preemptions — `y11l_pano` and `y26_pano` share batch 4 / imgsz 1280 and b recovered. Figures and the full ruling-out are in the training record; the stabilized rerun is tracked in #70 (whose original small-batch hypothesis this evidence refutes) and the caveat write-up in #72. The reported baseline will be the best-val checkpoint, selected on val, per the -protocol in #71 — with the caveat that today's checkpoints are undertrained relative to a stable -schedule and so give a **lower bound** on supervised-YOLO performance. +**pre-registered protocol** committed in the +[training record](../scripts/model_comparison/yolo_baseline/README.md#pre-registered-evaluation--checkpoint-selection-protocol-issue-71) +(#71) *before* any benchmark eval of any YOLO checkpoint: `best.pt` = Ultralytics best-val-fitness +with `patience=20`, config choice on val only, headline F1 at the Ultralytics-default conf 0.25, +same matcher/GT/radius as the whole roster, test bundles touched once — with the caveat that +today's checkpoints are undertrained relative to a stable schedule and so give a **lower bound** +on supervised-YOLO performance. Three classes of challenger, which fail differently and are worth keeping distinct: diff --git a/scripts/model_comparison/compare.py b/scripts/model_comparison/compare.py index 2b4a4a4..78e92e1 100644 --- a/scripts/model_comparison/compare.py +++ b/scripts/model_comparison/compare.py @@ -16,9 +16,14 @@ Each --models token is a provider (rampnet/gemini/qwen/owlv2/gdino/molmo) or provider:model_id to pin a variant, so several models from the same provider compare side by side. Detectors that emit calibrated scores (RampNet, OWLv2, -Grounding DINO) additionally get AP, a PR curve (--pr-out) and a threshold sweep -(--sweep); chat VLMs have no score to rank by, so they get one operating point. -See docs/model_comparison.md. +Grounding DINO, YOLO) additionally get AP, a PR curve (--pr-out) and a threshold +sweep (--sweep); chat VLMs have no score to rank by, so they get one operating +point. See docs/model_comparison.md. + +The supervised YOLO baseline (--models yolo:) is evaluated under the +pre-registered checkpoint-selection & eval protocol in +scripts/model_comparison/yolo_baseline/README.md (issue #71): checkpoint and +config chosen on val only, headline F1 at conf 0.25, test bundles touched once. """ import argparse import hashlib diff --git a/scripts/model_comparison/yolo_baseline/README.md b/scripts/model_comparison/yolo_baseline/README.md index 861fe6c..578cc76 100644 --- a/scripts/model_comparison/yolo_baseline/README.md +++ b/scripts/model_comparison/yolo_baseline/README.md @@ -30,7 +30,8 @@ in git, bulk/binary/regenerable out): the few relevant lines are quoted below. - **Benchmark-city eval numbers** — these `results.csv` values are the **internal YOLO val-split proxy** used to judge *training health*, **not** the reported baseline. The - reported numbers come from local `compare.py` benchmark eval on each `best.pt`. + reported numbers come from local `compare.py` benchmark eval on each `best.pt`, under + the pre-registered protocol below. ## Config grid @@ -116,6 +117,68 @@ The honest caveat: these checkpoints are **undertrained** relative to a stable s any benchmark number from them is a **lower bound** on supervised-YOLO performance, and must be reported as such. +## Pre-registered evaluation & checkpoint-selection protocol (issue #71) + +Fixed in writing 2026-07-28, **before any benchmark evaluation of any YOLO +checkpoint**. As of the commit adding this section, no YOLO checkpoint has been scored +on any benchmark bundle (city or `manual_gold`); the only numbers observed are the +internal val-split curves above (`runs/*/results.csv`, mirrored in #51's status +comments). The point is to make the baseline defensible against the standard +"you sandbagged / cherry-picked the baseline" review: every selection decision is made +on validation data, under rules written down first, applied identically to the +stabilized rerun (#70) when it lands. + +1. **Checkpoint selection — best-val, never test.** Each config's reported checkpoint + is its `best.pt` exactly as Ultralytics saves it: highest **fitness** (default + weighting, 0.1·mAP50 + 0.9·mAP50-95) on that config's own YOLO val split, with + `patience=20`. This refines #71's draft wording ("best val mAP50"): `best.pt` is + the only per-epoch artifact the runs keep, so re-ranking `results.csv` by raw mAP50 + would mean a second, hand-rolled selector — and on every curve committed so far the + two rules choose the same epoch anyway (all five configs: epoch 1, checked + 2026-07-28). RampNet's published checkpoint was selected by the same principle: + best **validation loss** on the dataset's val split (`stage_two/train.py`), never a + benchmark number. +2. **Config selection — on val, and it controls emphasis only.** Which config + headlines each family (YOLO11 vs YOLO26) in the main text is decided by the same + internal val fitness at `best.pt`. Stated caveat: tiles and pano configs have + different val sets (perspective crops vs whole equirects), so the cross-geometry + comparison is a judgment call on non-identical data — which is why this choice is + only about emphasis. **Every trained config's benchmark row goes in the appendix / + full table regardless; selection drops nothing.** Test is touched once, at the end, + after these choices are fixed. +3. **Benchmark eval — the identical path as every roster model.** `compare.py`'s + `yolo` provider: boxes → box centers → the `rampnet/detection_eval.py` matcher at + radius 0.022, on the same bundles and the same GT as the rest of the roster, + detections cached at the 0.05 floor (`--yolo-conf 0.05`). Tiles configs are scored + through the same perspective rig as the VLMs (`--tiling perspective`, + `--yolo-imgsz 1024`); pano configs whole-pano (`--tiling none --yolo-imgsz 1280`) — + each model in its training geometry. +4. **Operating point — fixed in advance, not tuned on test.** The headline metric is + **F1 at confidence 0.25**, Ultralytics' default predict confidence — the + "configure the baseline the way its authors recommend" choice, fixed here while no + test number exists. The full threshold sweep is still reported, with its best-F1 + row flagged as tune-on-test, exactly as the harness already does for every + confidence-carrying model. +5. **AP, with the truncation caveat.** AP/PR come from the full cached range (0.05 + floor). They are **not** directly comparable to RampNet's committed AP, whose + detections were extracted at the 0.5 peak threshold (a truncated curve — see + `docs/model_comparison.md`); the low-floor RampNet extraction is the #54/#78 line + of work. +6. **Reporting an unstable run.** A checkpoint from inside the collapse dip would + understate recall catastrophically (see the failure signature above); best-val + selection avoids the dip by construction. A `best.pt` from an instability-affected + run **is** reportable, but only with the standing caveat: it is a **lower bound** + on supervised-YOLO performance, superseded when the #70 stabilized rerun lands — + which will be selected, evaluated, and reported under this same protocol, + unchanged. +7. **Seeds (aspirational).** All runs are `seed=0`. If ckpt capacity allows, ≥3 seeds + of the headline configs → mean ± std, which makes single-run instability commentary + moot. Not a blocker for reporting the lower-bound numbers. + +The paper's "Baseline protocol" appendix paragraph is this section, condensed. +`docs/model_comparison.md` links here from its baseline-in-progress note, and the +protocol governs the results that will replace that note. + ## Provenance - **Training code:** `scripts/model_comparison/run_yolo_train.slurm` + the repo-root