Refresh the YOLO baseline record: durable weights, ckpt ceiling, fork (#51)

[jonfroehlich]

Refreshes scripts/model_comparison/yolo_baseline/README.md against the live runs. Issue #51.
Documentation only — no code, no behaviour change, no new numbers in the reported tables.

Why

The committed record had drifted from reality in three ways, one of them actively false.

1. A false statement. "Where the weights live" still read:

TODO — stage to Hugging Face (projectsidewalk/…) or lab storage and record the URL here.

The weights are now staged and verified:

/gscratch/makelab/jonf/rampnet_yolo_baseline_51/
  <arm>/{best.pt,last.pt,results.csv,args.yaml}   # 5 arms, 1.8 GB
  MANIFEST.md                                     # job IDs, per-arm metrics, sha256 of every weight

All 10 weight files sha256-verified against source (10 ok, 0 mismatched). They had lived
only on /gscratch/scrubbed, which auto-purges after ~21 idle days, and the run dirs date
from 07-26 — so anyone reading the old text would reasonably conclude the weights were still
one purge from gone, and either redo the staging or worry.

2. Stale numbers, and one that mattered. The status table was the 07-28 snapshot. Beyond
the refresh, the record claimed every checkpoint is undertrained and every benchmark number a
lower bound. That is no longer uniform:

• y26_pano's best.pt is epoch 18 — proven, not inferred: best.pt and last.pt are
  byte-identical (both sha256 aac93e74…50bf). mAP50-95 0.460 vs 0.413 at ep1.
• The other four arms' best.pt are frozen at epoch 1 (mtimes 07-26 20:22/21:42/23:35/23:58,
  each ~one epoch after the 18:10 start, while last.pt is current). For y26_tiles this is
  exact: best_fitness = 0.35579 = its ep1 mAP50-95.

Reporting all five identically would understate y26_pano and overstate the rest, so the
caveat is now per-arm and says "one-epoch model" in those words where that is what it is.

3. Two findings were missing entirely.

The ckpt slice ceiling — a limitation distinct from the LR instability already documented.
The instability says scores got worse; this says an arm cannot advance at all. Across 158
scheduling segments over 3.5 days, the longest slice ever granted was 8.24 h and none
reached 9.5 h. Tiles epoch time rose from ~5.2 h to ~9.5 h under filesystem contention — the
logs warn read: 2.1±1.3 MB/s across 557,413 tiles, so it is I/O-bound, not compute-bound. Since
Ultralytics checkpoints only at epoch boundaries, an epoch longer than the slice can never
complete: both tiles arms consumed ~19 GPU-hours each and logged zero epochs. Same mechanism
that dropped y11x_tiles on 07-27. This is a structural gap, not a slow one, and calling it
"undertrained" would misdescribe it.

The fork. y26_tiles resumed on one of makelab's two gpu-l40s GPUs (job 37889646, 72 h
cap, one GPU left free for students); the ckpt original was not cancelled and still runs. Its
provenance is recorded in full because a fork here is easy to get silently wrong: resume=True
restores every arg from the checkpoint including save_dir, so the copied last.pt needs
project/name/save_dir/model/resume rewritten, or the fork resumes into the original
run directory and corrupts a live training arm. Everything governing the schedule (epochs=60,
lr0, lrf, warmup_epochs, batch, imgsz, seed, patience, data, epoch,
best_fitness) was asserted unchanged on read-back.

Deliberately deferred

Not in this PR, by agreement — all three are worth doing once, against final curves rather
than a moving target:

• re-pulling runs/*/results.csv from scratch,
• regenerating figures/*.png via the committed plot_training_curves.py,
• updating the "Supervised baseline in progress" paragraph in docs/model_comparison.md.

So the committed CSVs still hold the 07-28 snapshot while the new table shows live epochs. The
file now says this explicitly in two places, so the staleness reads as a decision rather than an
oversight — including a note that docs/model_comparison.md is behind this file by design.

Verification

• y26_pano best.pt == last.pt byte-identical: sha256 compared on the cluster.
• Snapshot integrity: 10/10 sha256 match, recorded in MANIFEST.md.
• Fork isolation: original y26_tiles results.csv and last.pt still Jul 28 07:44 / 3 epochs
  after the fork started; the fork resumed from its own path, Resuming training … from epoch 4 to 60 total epochs.
• Segment statistics from sacct -X -D on jobs 37745358,37745360,37745361,37745362,37745363.
• Docs-only change; pytest -q unaffected.

Live status continues in #51. Bearing on #70 (any tiles arm in the stabilized rerun needs a
non-preemptable partition or a shorter epoch, independent of the LR fix) and #72 (the ceiling
belongs alongside the instability caveat).

🤖 Generated with Claude Code (claude-opus-5[1m])