diff --git a/scripts/model_comparison/yolo_baseline/README.md b/scripts/model_comparison/yolo_baseline/README.md index 578cc76..5911da7 100644 --- a/scripts/model_comparison/yolo_baseline/README.md +++ b/scripts/model_comparison/yolo_baseline/README.md @@ -5,7 +5,11 @@ trained on the RampNet dataset on Hyak (klone). It exists so the experiment surv `/gscratch/scrubbed`'s ~21-day auto-purge and is reproducible/defensible for the paper. > **Status:** the runs are in-flight (issue #51); live status is tracked there and the -> finding is summarized in `docs/model_comparison.md`. The training code lives at its +> finding is summarized in `docs/model_comparison.md` — though that summary predates the +> 2026-07-29 findings below (the ckpt slice ceiling, the per-arm `best.pt` split, the fork) +> and is deliberately left to be updated together with the final curves. **This file is the +> current record; `docs/model_comparison.md` is behind it by design, not by oversight.** The +> training code lives at its > canonical paths — `scripts/model_comparison/run_yolo_train.slurm` and the repo-root > `hyak_yolo_runbook.sh` — merged in PR #76. (This record briefly carried as-run > snapshots of both; the only drift from the canonical copies was a display-only @@ -50,20 +54,47 @@ Batches are the as-run values from each run's committed `args.yaml`. `y11x_tiles submitted at batch 3, then resubmitted at batch 12 trying to finish an epoch inside the ckpt scheduling slice (its `args.yaml` is the batch-12 attempt); neither completed one. -## Status & findings (snapshot: 2026-07-28, training in progress) +There is a seventh **run directory** on scratch, `y26_tiles_l40s`, but it is **not a seventh +config**: it is `y26_tiles` itself, resumed from its own epoch-3 checkpoint on a different +partition with every hyperparameter unchanged. See "The `y26_tiles_l40s` fork" below. + +## Status & findings (live check: 2026-07-29 ~20:45 PT, training in progress) Val-split proxy mAP50 from `results.csv`. **Every config peaks at epoch 1, collapses at epoch 3, and recovers as the learning rate decays.** The instability is universal, not per-config — see the figures. -| Config | Epochs | best (ep) | Current mAP50 | Assessment | -|--------------|--------|-----------|---------------|------------| -| `y26_pano` | 14 | 0.738 (1) | **0.659 ↑** | ✅ fully round-tripped: 0.738 → 0.125 @ep6 → 0.659 @ep14, climbing | -| `y11l_pano` | 10 | 0.779 (1) | **0.183 ↑** | 🟡 recovering — 7 epochs near 0, then 0.005 → 0.183 @ep10 | -| `y26_tiles` | 3 | 0.647 (1) | 0.280 ↓ | 🟡 in the dip, too early to call | -| `y11l_tiles` | 3 | 0.655 (1) | 0.042 ↓ | 🟡 in the dip, too early to call | -| `y11x_pano` | 7 | 0.777 (1) | **0.000** | ❌ five straight epochs at literal 0; no recovery yet | -| `y11x_tiles` | 0 | — | — | ⏹ dropped 2026-07-27 (GPU-saturated: epoch ~10 h > ckpt slice) | +> **The numbers in this table are ahead of the committed `runs/*/results.csv` and +> `figures/*.png`**, which are still the 2026-07-28 snapshot (e.g. `y26_pano` has 14 rows +> there vs 18 epochs below). The CSVs and figures will be re-pulled and regenerated once +> the arms stop, so the record ends up internally consistent against *final* curves rather +> than a moving target. Until then, prefer this table for status and the CSVs for the +> per-epoch shape of the collapse. + +| Config | Epochs | best-val (ep) | Latest mAP50 | `best.pt` holds | Assessment | +|------------------|--------|---------------|--------------|-----------------|------------| +| `y26_pano` | 20 | 0.738 (1) | **0.730 ↑** | **ep20 — current** | ✅ round-tripped 0.738 → 0.125 @ep6 → 0.730 @ep20; mAP50-95 0.475 ≫ ep1's 0.413 | +| `y11l_pano` | 14 | 0.779 (1) | **0.767 ↑** | **ep14 — current** | ✅ **unfroze at ep14** — 0.000 @ep4–5 → 0.183 @ep10 → 0.767 @ep14; mAP50-95 0.437 > ep1's 0.423 | +| `y11x_pano` | 9 | 0.777 (1) | 0.061 ↑ | ep1 | ❌ five epochs at literal 0, **and now starved** — no completed epoch in ~15 h (see below) | +| `y26_tiles` | 3 | 0.647 (1) | 0.280 ↓ | ep1 | ⏸ **blocked** — no completed epoch since 2026-07-28; forked to `gpu-l40s` (below) | +| `y11l_tiles` | 3 | 0.655 (1) | 0.042 ↓ | ep1 | ⏸ **blocked** — no completed epoch since 2026-07-28 | +| `y26_tiles_l40s` | 4 | 0.647 (1) | 0.268 ↓ | inherited ep1 | 🆕 fork; **first new epoch landed** (ep4, 4.94 h) — still in the dip, as the LR schedule predicts | +| `y11x_tiles` | 0 | — | — | — | ⏹ dropped 2026-07-27 (GPU-saturated: epoch ~10 h > ckpt slice) | + +**Two arms have now cleared epoch 1, so the grid is genuinely split.** For `y26_pano` the +evidence is direct rather than an mtime inference: its `best.pt` and `last.pt` were +byte-identical at ep18 (both sha256 `aac93e74…50bf`), and it has since advanced to ep20 +(mAP50-95 0.475 vs 0.413 @ep1). **`y11l_pano` unfroze at ep14** on 2026-07-30 02:29Z — +`best.pt` mtime now equals `results.csv` mtime, and mAP50-95 0.437 finally beat ep1's 0.423 +after 12 epochs unbeaten. This was predicted one check earlier and is worth noting as +confirmation that the recovery is real, not noise. + +The remaining **three** arms' `best.pt` files are still frozen at **epoch 1** — `y11x_pano`, +`y11l_tiles`, `y26_tiles`, mtimes 2026-07-26 21:42 / 23:35 / 23:58, each roughly one epoch +after the 18:10 job start, while their `last.pt` files are current. For `y26_tiles` this is +confirmed exactly: the checkpoint's `best_fitness` is 0.35579, which is its epoch-1 mAP50-95 +to five decimals. **This split moves — re-check mtimes at eval time rather than trusting this +table.** ### The instability: collapse tracks the warmup LR peak @@ -106,6 +137,97 @@ precision reads 0 by the 0/0 convention, not from false positives). This matters interpretation — the instability suppresses detections, so a checkpoint caught inside the dip understates recall catastrophically and would badly misrepresent the baseline. +### The ckpt slice ceiling: why the tiles arms stopped advancing (2026-07-29) + +This is a **second, independent limitation** and must not be folded into the LR instability +above. The instability says an arm's scores got worse; this says an arm could not advance at +all. From `sacct -X -D` over **158 scheduling segments** across the five jobs and 3.5 days: + +- **Longest contiguous segment ever granted: 8.24 h.** That is the ckpt slice limit — every + clean `REQUEUED` record ends at ~08:0x elapsed. +- **Segments ≥ 9.5 h: zero.** For any arm, ever. +- The partition also got far choppier from 2026-07-28T17:00: each arm then received ~19.4 h + of compute across 20–26 segments, **16–23 of them under one hour**. On 2026-07-29 alone the + five jobs accumulated **115 segments**. + +Against that ceiling, epoch time decides whether an arm can progress at all: + +| arm | epoch time | vs the 8.2 h ceiling | epochs completed in the ~28 h to 07-29 14:45 PT | +|---|---|---|---| +| `y26_pano`, `y11l_pano` | ~2.6–2.8 h | fits 2–3× per slice | +4, +3 | +| `y11x_pano` | ~3.5–4 h | fits | +2 | +| `y11l_tiles`, `y26_tiles` | 5.2 h → **~9.5 h** | **exceeds** | **0, 0** | + +**Ultralytics checkpoints only at epoch boundaries.** Once an epoch takes longer than the +slice, every preemption discards the entire partial epoch and the arm can never advance — it +burns GPU-hours and logs nothing. `y11x_tiles` was dropped on 07-27 for exactly this (~10 h +epoch); the other two tiles arms are the same failure one notch slower, and it only became +visible when cluster I/O slowed down. + +**The cause is I/O, not compute**, and it is in the training logs verbatim: + +``` +WARNING ⚠️ train: Slow image access detected (ping: 0.6±0.2 ms, read: 2.1±1.3 MB/s, size: 338.1 KB). +``` + +…across 557,413 train tiles. So tiles epoch time is **node- and contention-dependent, not a +fixed property of the config**: ~5.2 h on a good ckpt landing (epochs 1–3, 07-26/27), ~9.5 h +on a contended one (measured 07-29: 54% of an epoch in 5.15 h), and **4.94 h on a dedicated +`gpu-l40s` L40S** with no allocation-mates competing for the filesystem — now **measured**, from +the `time` column of the fork's completed epoch 4 (17,774.6 s). Its epoch 5 is tracking a little +faster (~4.0 h at 6.4 it/s), so treat **~4–5 h** as the dedicated-node range rather than a single +figure. An earlier draft of this file projected 4.1 h from 13% of an in-flight epoch; the real +value came in ~20% higher, which is the usual direction for an early-epoch extrapolation. Note this also means +a faster GPU alone would not help; and `gpu-l40s` nodes report `TMP_DISK=0`, so the dataset +cannot be staged to node-local disk to fix it. + +**Refinement (2026-07-29 20:45 PT): the binding constraint is the *effective* preemption +interval, not the 8.24 h nominal cap.** `y11x_pano` has now also stopped advancing — no completed +epoch in ~15 h — despite an epoch of only ~3.4 h, which fits the nominal ceiling with room to +spare. Its segments that evening ran 0:58, 2:30, 0:09, 0:18, 0:02, 0:27. So the rule is not +"epoch < 8.24 h is safe"; it is **epoch < whatever contiguous run the partition currently +hands out**, and that collapsed to minutes. The arm is *not* wedged — its progress counter is +live (ep10, 12%, 7.2 it/s) — it simply keeps losing partial epochs. Any future capacity plan on +`ckpt` should budget against the *observed* segment distribution, not the slice cap. + +**Consequence for the record.** Left on ckpt, `y26_pano` and `y11l_pano` (epochs ~2.6–2.8 h) +plausibly reach the 60-epoch budget or their `patience` stop, while the two tiles arms — and now +`y11x_pano` — stay frozen at their current `best.pt`. That is a **structural gap, not a slow +one**, and reporting it as "undertrained" would misdescribe it. + +#### The `y26_tiles_l40s` fork (job `37889646`) + +To answer *"does the tiles representation recover like pano did?"* without cancelling +anything, `y26_tiles` was forked onto **one** of makelab's two `gpu-l40s` GPUs +(`--time=72:00:00`, node `g3100`), leaving the other free for students. `y26_tiles` was chosen +over `y11l_tiles` because YOLO26 is the architecture that demonstrably recovers (`y26_pano` +round-tripped) and it sat at 0.280 vs 0.042 — the likelier of the two to yield a usable +number. **The original `y26_tiles` (job `37745360`) was not cancelled and still runs on ckpt.** + +Provenance, because a fork is easy to get silently wrong: + +- Resumed from the sha256-verified durable snapshot's `last.pt` (see "Where the weights live"). +- **Only these keys were rewritten** inside the checkpoint's `train_args`: `project`, `name`, + `save_dir`, `model`, `resume`. **This rewrite is mandatory.** Ultralytics' `resume=True` + restores *every* arg from the checkpoint — including `save_dir` — so an un-rewritten fork + resumes into the **original** run directory and corrupts a live training arm. +- Asserted unchanged on read-back: `epoch=2`, `best_fitness=0.35579`, `epochs=60`, `lr0`, + `lrf`, `warmup_epochs=3.0`, `batch=6`, `imgsz=1024`, `seed=0`, `patience=20`, `data`. So the + fork is a faithful continuation of the same LR schedule, with optimizer and EMA state + intact — Ultralytics confirms `Resuming training … from epoch 4 to 60 total epochs`. +- **Two lineages now share epochs 1–3.** Any reported tiles number must say which lineage it + came from. In practice they may never diverge, since the ckpt original has completed no + epoch since 07-28. +- **What this does and does not buy.** Per-epoch time is unchanged by the move (same shared + filesystem); what changes is the uninterrupted window. At the measured ~4–5 h/epoch, 72 h yields + roughly **14 epochs**, reaching ~ep18 — enough to show whether tiles turns the corner + (`y26_pano` turned by ep9), but **not** a converged 60-epoch tiles run. +- **First fork epoch is in: ep4 = mAP50 0.268 / mAP50-95 0.111**, marginally *below* ep3's + 0.280 / 0.119, with precision 0.497 and recall 0.201. This is the expected place in the + schedule, not a bad sign: ep3 is the warmup LR peak and the pano arms bottomed at ep4–ep6 + before turning. The same "stops firing" signature (recall collapses, precision holds) is + present. **The recovery question stays open until ~ep9–10**, i.e. ~20–25 h of fork time. + ### What is reportable today Epoch 1–2 are **not** a "pretrained artifact" — an earlier version of this record said so @@ -117,6 +239,22 @@ The honest caveat: these checkpoints are **undertrained** relative to a stable s any benchmark number from them is a **lower bound** on supervised-YOLO performance, and must be reported as such. +**As of 2026-07-29 that caveat is per-arm, not uniform** — an earlier version of this record +applied it to the whole grid, which now understates one arm and *overstates* the others: + +| arm | what `best.pt` actually is | how to report it | +|---|---|---| +| `y26_pano` (ep20), `y11l_pano` (ep14) | genuinely trained, still improving | undertrained relative to a 60-epoch stable schedule, but real models. A normal "lower bound" caveat. | +| `y11x_pano`, `y11l_tiles`, `y26_tiles` | **epoch 1 of 60** | "lower bound" is too soft. These are *one-epoch models*, and the write-up must say so in those words — a reader will otherwise assume `best.pt` came from a converged run. | + +The distinction matters in both directions. Discounting `y26_pano` or `y11l_pano` as "1-epoch +models" would understate the supervised baseline; reporting the three frozen arms as merely +"undertrained" would overstate it. + +**This table has already moved once** — `y11l_pano` was in the bottom row at the 14:45 PT check +and moved up by 20:45 PT, six hours later. So **re-check each `best.pt`'s mtime against its +`results.csv` at eval time** and place the arms then; do not inherit this split. + ## Pre-registered evaluation & checkpoint-selection protocol (issue #71) Fixed in writing 2026-07-28, **before any benchmark evaluation of any YOLO @@ -197,7 +335,11 @@ protocol governs the results that will replace that note. earlier draft of this record said `fixed:0.03` — that was the runbook's old default, not what ran.) - **Toolchain:** Ultralytics 8.4.105 · Python 3.11.15 · torch 2.13.0+cu126. -- **Hardware:** NVIDIA L40 (45 GB), 1 GPU/job, Hyak `ckpt-g2` (preemptable/requeue). +- **Hardware:** NVIDIA L40 (45 GB), 1 GPU/job, Hyak `ckpt-g2` (preemptable/requeue). The + `y26_tiles_l40s` fork instead runs on Hyak `gpu-l40s` (makelab's own allocation, + non-preemptable, `MaxTime=UNLIMITED`), 1 GPU, NVIDIA L40S (45 GB), node `g3100`. Because + ckpt schedules onto whatever GPU is idle (l40/l40s/h200), per-epoch wall time varies across + requeues even within one arm — see "The ckpt slice ceiling". - **Hyperparameters (resolved):** `epochs=60`, `patience=20`, `optimizer=auto`, `lr0=0.01`, `lrf=0.01`, `momentum=0.937`, `weight_decay=0.0005`, `warmup_epochs=3.0`, `close_mosaic=10`, `amp=true`, `seed=0`. Per-run detail in each `runs//args.yaml`. @@ -209,14 +351,44 @@ protocol governs the results that will replace that note. batch-12 resubmit 37809205, scancel'd 2026-07-27) · y26_tiles 37745360 · y11l_pano 37745361 · y11x_pano 37745362 · y26_pano 37745363. The job↔config map was read off the `yolo_train_.out` log headers during the 2026-07-27 preemption check. - Dataset-download job 37649635; prep job 37677130. -- **Run dates:** 2026-07-26 → in progress as of 2026-07-28. + Dataset-download job 37649635; prep job 37677130. The `y26_tiles_l40s` fork is job + **37889646** (`gpu-l40s`, submitted 2026-07-29 ~13:55 PT). +- **Run dates:** 2026-07-26 → in progress as of 2026-07-29. +- **Mail flags are not in the launcher.** The five original jobs carry + `MailType=END,FAIL,TIME_LIMIT`, which was passed on the 07-26 sbatch command line, not set by + `run_yolo_train.slurm` (a fresh submission of the same script gets no mail config). On a + preemptable partition that means one email per preemption — 115 of them on 2026-07-29 alone. + **This volume is accepted deliberately** (Jon, 2026-07-29: "I'm fine with still getting the + mail"), so it is recorded here as context for reading the run history, not as a defect to fix. + A future rerun that wants quiet should omit the flags or narrow them to `TIME_LIMIT`; note they + must be set per-job with `scontrol update`, since the launcher never sets them. ## Where the weights live `best.pt` files are **not** in git. Durable homes: -_TODO — stage to Hugging Face (`projectsidewalk/…`) or lab storage and record the URL here._ +**Staged to durable lab storage on 2026-07-29, while the runs were still going:** + +``` +/gscratch/makelab/jonf/rampnet_yolo_baseline_51/ + /{best.pt,last.pt,results.csv,args.yaml} # 5 arms, 1.8 GB total + MANIFEST.md # Slurm job IDs, per-arm epochs/metrics, sha256 of every weight file +``` + +Arms: `y11l_pano`, `y11x_pano`, `y26_pano`, `y11l_tiles`, `y26_tiles`. All **10 weight files +were sha256-verified against the source** at copy time (10 ok, 0 mismatched), and the hashes +are recorded in `MANIFEST.md` so a later copy can be checked against them. + +This was done because the weights existed **only** on `/gscratch/scrubbed`, which auto-purges +after ~21 idle days — and the run directories date from 07-26, so the clock starts the moment +the jobs stop writing. `/gscratch/makelab` is not scrubbed. + +`last.pt` is kept alongside `best.pt` deliberately: `best.pt` is what the #71/#80 protocol +reports, but only `last.pt` carries the optimizer/EMA state needed to *resume* an arm (which is +exactly what the `y26_tiles_l40s` fork above needed). + +Still open: publishing to Hugging Face (`projectsidewalk/…`) if these become a released +artifact rather than an internal baseline. Lab storage is durable but not public. Keep **every** run's `best.pt`, including the ones that collapsed. An earlier version of this record called those non-reportable epoch-1 artifacts and proposed letting them expire;