Preserve the supervised-YOLO baseline training record (#51)

[jonfroehlich]

What

Preserves the supervised-YOLO baseline experiment (issue #51) as a durable,
reproducible record in the repo, and surfaces its in-flight status in the central
results doc.

Adds scripts/model_comparison/yolo_baseline/:

• README.md — config grid, the training-instability finding, and a provenance block
  (toolchain, dataset + prep params, hyperparameters, Slurm job IDs, dates).
• runs/<config>/{results.csv,args.yaml} — per-epoch curves + resolved config for all six
  configs (y11x_tiles has args.yaml only — it was dropped before epoch 1).
• plot_training_curves.py + figures/*.png — four diagnostic figures, regenerable from
  the committed CSVs alone (CPU-only, no network, no checkpoints).

Plus a "supervised baseline in progress" note in docs/model_comparison.md so the
baseline's absence from the results tables reads as not yet run, not withheld.

Why

Per the "record scientific progress in GitHub" standing rule (CLAUDE.md): the runs'
outputs live on /gscratch/scrubbed, which auto-purges after ~21 idle days. This captures
the perishable record before it's lost, following the repo convention (curated small-text
results tracked; weights/logs stay out).

The finding — and a correction to an earlier version of this PR

Fresh per-epoch data pulled from the live runs falsified three claims this branch
previously carried. They are corrected in the record, and the correction changes what
should happen next:

previously claimed	what the data shows
"YOLO26 and the tiles configs trained cleanly"	All five remaining configs hit the same collapse. The tiles runs were two epochs into it when that was written.
"Small-batch BatchNorm/EMA instability"	Refuted by the grid. y11l_pano and y26_pano share batch 4 / imgsz 1280 and both collapsed, then both recovered. Batch spans 2/4/6 and imgsz 1024/1280 with no separation.
"The collapsed runs hold a non-reportable epoch-1 pretrained artifact, not worth keeping"	By epoch 1 the model has trained on all 150k images — the score is real. Those best.pt files are genuine best-val checkpoints and currently the only supervised baseline we have.

What actually happens: validation mAP peaks at epoch 1, collapses at epoch 3, and
recovers as the LR decays — in every config. Epoch 3 is exactly where the warmup ramp
peaks (lr/pg0 0.010 → 0.029, warmup_epochs=3). Training loss falls straight through
the collapse while val cls_loss explodes 1.2 → 8.2, with no NaN or AMP error. The ckpt
preemptions are scattered across epochs 2–13 and do not line up with it. optimizer=auto
resolves to MuSGD(lr=0.01, momentum=0.9) in every job.

So the clean test is a lower peak LR or longer warmup, not a larger batch — which is
what #70 currently proposes. Worth re-scoping before that rerun burns another week.

The failure signature is a recall collapse at sustained high precision: the model stops
firing rather than starting to guess. A checkpoint caught inside the dip would understate
recall catastrophically.

Overlap with #76 — resolved

#76 (merged) carries the canonical training code at scripts/model_comparison/run_yolo_train.slurm
and the repo-root hyak_yolo_runbook.sh. This branch's duplicate snapshots under
yolo_baseline/ are deleted here, so merging leaves exactly one copy of each. The
provenance those snapshots preserved is in git history.

Scope / notes for review

• No benchmark-city eval numbers yet — these results.csv values are the internal YOLO
  val-split proxy (training health), not the reported baseline. Benchmark eval is pending, and
  is now unblocked: every run's best.pt is already on disk.
• Weights are not committed (*.pt/*.pth now gitignored). The README has a TODO for the
  durable URL.
• Any benchmark number from today's checkpoints is a lower bound — they are undertrained
  relative to a stable schedule, and the README says so.

Related: #51 (parent), #70, #71, #72.

---

🤖 Generated with Claude Code

[jonfroehlich]

Heads-up from #76 (the code half of #51), which was opened concurrently — the two
PRs now touch disjoint files, but there's one conflict git won't flag because the
paths differ.

This PR adds scripts/model_comparison/yolo_baseline/run_yolo_train.slurm and
.../yolo_baseline/hyak_yolo_runbook.sh. #76 adds the canonical
scripts/model_comparison/run_yolo_train.slurm and the repo-root
hyak_yolo_runbook.sh. Merging both leaves two copies of each, and:

• the two run_yolo_train.slurm files are byte-identical;
• the two runbooks are not — the copy here is three lines older. It predates
  the ARG_MAX fix to the dataset count display (ls "$DATA/$s"/*.jpg | wc -l
  overflows on the ~150k-file train split, so ls errors to suppressed stderr and
  it prints a false train: 0 on a healthy download).

The difference is display-only and cannot affect the recorded runs, so the record
stands either way. But shipping the pre-fix text as a second live copy is a trap
for the next person who runs the runbook.

Suggested resolution for whichever merges second: keep the canonical copies, drop
the two under yolo_baseline/, and point this README at them — git history
already preserves both texts, and the README can state which version the recorded
runs used.

[jonfroehlich]

Review fixes pushed in 562277a:

• Config table: batches corrected to the as-run args.yaml values (y11x_tiles 3→12 across its two attempts, y26_tiles 6) — batch is the load-bearing variable in the collapse hypothesis, so the table must be exact.
• Provenance: box-size corrected to pitch + ramp 1.8 m per run_yolo_prep.slurm's as-run defaults; the earlier fixed:0.03 matched the runbook default, not the prep launcher. Needs a one-command confirmation on klone before scratch purges: head logs/yolo_prep_*.out prints box=... ramp=....
• Added y11x_tiles's job ID 37745359, inferred from the one gap in the otherwise-contiguous six-job block (its batch-12 resubmit ID went unrecorded; sacct can recover both).
• Dropped the run_yolo_train.slurm / hyak_yolo_runbook.sh snapshots — the canonical copies land via Add the supervised YOLO baseline to the comparison harness (#51) #76 (launcher byte-identical; runbook drift was display-only, documented there). This removes the two-live-copies trap at merge time.
• .gitignore now enforces the stated weights-out policy structurally (*.pt, *.pth).

Data cross-check during review: every number in the status table matched the committed CSVs at snapshot time (epoch counts, ep1 values, current mAP50, the P≈0.94/R≈0.03 flicker, the 4 straight literal-0 epochs, the train-loss trends). One observation about collapse timing is on #70.

🤖 Generated with Claude Code

[jonfroehlich]

Follow-up: the box-size confirmation flagged above is resolved without needing the klone log — the session run notes record it definitively (prep job 37677130, 2026-07-25: pitch + ramp 1.8 m, chosen after overlay QA; fixed:0.03 rejected as too small, gps fell back on 169/200 smoke panos). The y11x_tiles job IDs are also now exact rather than inferred: 37745359 original, 37809205 batch-12 resubmit. Both upgraded in 34bafcd.

🤖 Generated with Claude Code