From 3816a59a03d6fdc312e675b3efc345d7b5d572fc Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Wed, 29 Jul 2026 15:26:43 -0700 Subject: [PATCH 1/4] Refresh the YOLO baseline record: durable weights, ckpt ceiling, fork (#51) The committed record had drifted from the runs in three ways, one of them an actively false statement. "Where the weights live" said `_TODO -- stage to Hugging Face or lab storage_`. The weights are now staged: /gscratch/makelab/jonf/rampnet_yolo_baseline_51/, five arms, best.pt + last.pt + curves + a MANIFEST.md, all ten weight files sha256-verified against source. They had existed only on /gscratch/scrubbed, which auto-purges after ~21 idle days, and the run dirs date from 07-26. A reader of the old text would reasonably conclude the weights were still at risk. The status table was the 07-28 snapshot (epochs 14/10/3/3/7) and is now live (18/13/9/3/3 plus the fork). More consequentially, "every benchmark number from these checkpoints is a lower bound" is no longer uniform: y26_pano's best.pt is epoch 18 -- proven byte-identical to last.pt, sha256 aac93e74..50bf -- while the other four are frozen at epoch 1. Reporting all five the same way would both understate y26_pano and overstate the rest, so the caveat is now per-arm and says "one-epoch model" in those words where that is what it is. Two findings were missing entirely. The ckpt slice ceiling is a limitation distinct from the LR instability already documented: across 158 scheduling segments the longest ever granted was 8.24 h and none reached 9.5 h, while tiles epoch time rose from ~5.2 h to ~9.5 h under filesystem contention (the logs warn `read: 2.1+-1.3 MB/s` over 557,413 tiles). Since Ultralytics checkpoints only at epoch boundaries, an epoch longer than the slice can never complete -- so both tiles arms burned ~19 GPU-hours each and logged zero epochs. That is a structural gap, not a slow one, and it is the same mechanism that dropped y11x_tiles. The response was additive: y26_tiles forked onto one of makelab's two gpu-l40s GPUs (job 37889646, 72 h cap), original left running on ckpt. The fork's provenance is recorded in full because it is easy to get silently wrong -- resume=True restores every arg from the checkpoint including save_dir, so the copied last.pt needs project/name/save_dir/model/resume rewritten or the fork resumes into the live original directory and corrupts a training arm. Everything governing the schedule was asserted unchanged on read-back. Deferred deliberately: re-pulling runs/*/results.csv, regenerating figures/, and updating the docs/model_comparison.md paragraph -- all worth doing once against final curves rather than a moving target. The file now says so explicitly so the staleness reads as a decision rather than an oversight. Co-Authored-By: Claude Opus 5 (1M context) --- .../model_comparison/yolo_baseline/README.md | 170 ++++++++++++++++-- 1 file changed, 156 insertions(+), 14 deletions(-) diff --git a/scripts/model_comparison/yolo_baseline/README.md b/scripts/model_comparison/yolo_baseline/README.md index 578cc76..b06e00d 100644 --- a/scripts/model_comparison/yolo_baseline/README.md +++ b/scripts/model_comparison/yolo_baseline/README.md @@ -5,7 +5,11 @@ trained on the RampNet dataset on Hyak (klone). It exists so the experiment surv `/gscratch/scrubbed`'s ~21-day auto-purge and is reproducible/defensible for the paper. > **Status:** the runs are in-flight (issue #51); live status is tracked there and the -> finding is summarized in `docs/model_comparison.md`. The training code lives at its +> finding is summarized in `docs/model_comparison.md` — though that summary predates the +> 2026-07-29 findings below (the ckpt slice ceiling, the per-arm `best.pt` split, the fork) +> and is deliberately left to be updated together with the final curves. **This file is the +> current record; `docs/model_comparison.md` is behind it by design, not by oversight.** The +> training code lives at its > canonical paths — `scripts/model_comparison/run_yolo_train.slurm` and the repo-root > `hyak_yolo_runbook.sh` — merged in PR #76. (This record briefly carried as-run > snapshots of both; the only drift from the canonical copies was a display-only @@ -50,20 +54,40 @@ Batches are the as-run values from each run's committed `args.yaml`. `y11x_tiles submitted at batch 3, then resubmitted at batch 12 trying to finish an epoch inside the ckpt scheduling slice (its `args.yaml` is the batch-12 attempt); neither completed one. -## Status & findings (snapshot: 2026-07-28, training in progress) +There is a seventh **run directory** on scratch, `y26_tiles_l40s`, but it is **not a seventh +config**: it is `y26_tiles` itself, resumed from its own epoch-3 checkpoint on a different +partition with every hyperparameter unchanged. See "The `y26_tiles_l40s` fork" below. + +## Status & findings (live check: 2026-07-29 ~14:45 PT, training in progress) Val-split proxy mAP50 from `results.csv`. **Every config peaks at epoch 1, collapses at epoch 3, and recovers as the learning rate decays.** The instability is universal, not per-config — see the figures. -| Config | Epochs | best (ep) | Current mAP50 | Assessment | -|--------------|--------|-----------|---------------|------------| -| `y26_pano` | 14 | 0.738 (1) | **0.659 ↑** | ✅ fully round-tripped: 0.738 → 0.125 @ep6 → 0.659 @ep14, climbing | -| `y11l_pano` | 10 | 0.779 (1) | **0.183 ↑** | 🟡 recovering — 7 epochs near 0, then 0.005 → 0.183 @ep10 | -| `y26_tiles` | 3 | 0.647 (1) | 0.280 ↓ | 🟡 in the dip, too early to call | -| `y11l_tiles` | 3 | 0.655 (1) | 0.042 ↓ | 🟡 in the dip, too early to call | -| `y11x_pano` | 7 | 0.777 (1) | **0.000** | ❌ five straight epochs at literal 0; no recovery yet | -| `y11x_tiles` | 0 | — | — | ⏹ dropped 2026-07-27 (GPU-saturated: epoch ~10 h > ckpt slice) | +> **The numbers in this table are ahead of the committed `runs/*/results.csv` and +> `figures/*.png`**, which are still the 2026-07-28 snapshot (e.g. `y26_pano` has 14 rows +> there vs 18 epochs below). The CSVs and figures will be re-pulled and regenerated once +> the arms stop, so the record ends up internally consistent against *final* curves rather +> than a moving target. Until then, prefer this table for status and the CSVs for the +> per-epoch shape of the collapse. + +| Config | Epochs | best-val (ep) | Latest mAP50 | `best.pt` holds | Assessment | +|------------------|--------|---------------|--------------|-----------------|------------| +| `y26_pano` | 18 | 0.738 (1) | **0.717 ↑** | **ep18 — current** | ✅ round-tripped 0.738 → 0.125 @ep6 → 0.717 @ep18; the only arm whose `best.pt` has cleared ep1 | +| `y11l_pano` | 13 | 0.779 (1) | **0.727 ↑** | ep1 | 🟡 recovering hard — 0.000 @ep4–5 → 0.183 @ep10 → 0.727 @ep13; ~1–2 epochs from unfreezing `best.pt` | +| `y11x_pano` | 9 | 0.777 (1) | 0.061 ↑ | ep1 | ❌ five epochs at literal 0; `patience=20` retires it ~ep22, almost certainly still holding ep1 | +| `y26_tiles` | 3 | 0.647 (1) | 0.280 ↓ | ep1 | ⏸ **blocked** — no completed epoch since 2026-07-28; forked to `gpu-l40s` (below) | +| `y11l_tiles` | 3 | 0.655 (1) | 0.042 ↓ | ep1 | ⏸ **blocked** — no completed epoch since 2026-07-28 | +| `y26_tiles_l40s` | 3 (+) | inherited | — | inherited ep1 | 🆕 fork of `y26_tiles` resumed at ep4 on `gpu-l40s`, ~4.1 h/epoch (see "The ckpt slice ceiling") | +| `y11x_tiles` | 0 | — | — | — | ⏹ dropped 2026-07-27 (GPU-saturated: epoch ~10 h > ckpt slice) | + +**`y26_pano`'s `best.pt` is epoch 18, not epoch 1.** Its `best.pt` and `last.pt` are +byte-identical (both sha256 `aac93e74…50bf`), which is direct evidence rather than an +inference from mtimes: mAP50-95 0.460 @ep18 vs 0.413 @ep1. The other four arms' `best.pt` +files are frozen at **epoch 1** — mtimes 2026-07-26 20:22 / 21:42 / 23:35 / 23:58, each +roughly one epoch after the 18:10 job start, while their `last.pt` files are current. For +`y26_tiles` this is confirmed exactly: the checkpoint's `best_fitness` is 0.35579, which is +its epoch-1 mAP50-95 to five decimals. ### The instability: collapse tracks the warmup LR peak @@ -106,6 +130,79 @@ precision reads 0 by the 0/0 convention, not from false positives). This matters interpretation — the instability suppresses detections, so a checkpoint caught inside the dip understates recall catastrophically and would badly misrepresent the baseline. +### The ckpt slice ceiling: why the tiles arms stopped advancing (2026-07-29) + +This is a **second, independent limitation** and must not be folded into the LR instability +above. The instability says an arm's scores got worse; this says an arm could not advance at +all. From `sacct -X -D` over **158 scheduling segments** across the five jobs and 3.5 days: + +- **Longest contiguous segment ever granted: 8.24 h.** That is the ckpt slice limit — every + clean `REQUEUED` record ends at ~08:0x elapsed. +- **Segments ≥ 9.5 h: zero.** For any arm, ever. +- The partition also got far choppier from 2026-07-28T17:00: each arm then received ~19.4 h + of compute across 20–26 segments, **16–23 of them under one hour**. On 2026-07-29 alone the + five jobs accumulated **115 segments**. + +Against that ceiling, epoch time decides whether an arm can progress at all: + +| arm | epoch time | vs the 8.2 h ceiling | epochs completed in the ~28 h to 07-29 14:45 PT | +|---|---|---|---| +| `y26_pano`, `y11l_pano` | ~2.6–2.8 h | fits 2–3× per slice | +4, +3 | +| `y11x_pano` | ~3.5–4 h | fits | +2 | +| `y11l_tiles`, `y26_tiles` | 5.2 h → **~9.5 h** | **exceeds** | **0, 0** | + +**Ultralytics checkpoints only at epoch boundaries.** Once an epoch takes longer than the +slice, every preemption discards the entire partial epoch and the arm can never advance — it +burns GPU-hours and logs nothing. `y11x_tiles` was dropped on 07-27 for exactly this (~10 h +epoch); the other two tiles arms are the same failure one notch slower, and it only became +visible when cluster I/O slowed down. + +**The cause is I/O, not compute**, and it is in the training logs verbatim: + +``` +WARNING ⚠️ train: Slow image access detected (ping: 0.6±0.2 ms, read: 2.1±1.3 MB/s, size: 338.1 KB). +``` + +…across 557,413 train tiles. So tiles epoch time is **node- and contention-dependent, not a +fixed property of the config**: ~5.2 h on a good ckpt landing (epochs 1–3, 07-26/27), ~9.5 h +on a contended one (measured 07-29: 54% of an epoch in 5.15 h), and **~4.1 h on a dedicated +`gpu-l40s` L40S** with no allocation-mates competing for the filesystem. Note this also means +a faster GPU alone would not help; and `gpu-l40s` nodes report `TMP_DISK=0`, so the dataset +cannot be staged to node-local disk to fix it. + +**Consequence for the record.** Left on ckpt, the three pano arms plausibly reach the +60-epoch budget or their `patience` stop, while the two tiles arms stay at epoch 1 +indefinitely. That is a **structural gap, not a slow one**, and reporting it as "undertrained" +would misdescribe it. + +#### The `y26_tiles_l40s` fork (job `37889646`) + +To answer *"does the tiles representation recover like pano did?"* without cancelling +anything, `y26_tiles` was forked onto **one** of makelab's two `gpu-l40s` GPUs +(`--time=72:00:00`, node `g3100`), leaving the other free for students. `y26_tiles` was chosen +over `y11l_tiles` because YOLO26 is the architecture that demonstrably recovers (`y26_pano` +round-tripped) and it sat at 0.280 vs 0.042 — the likelier of the two to yield a usable +number. **The original `y26_tiles` (job `37745360`) was not cancelled and still runs on ckpt.** + +Provenance, because a fork is easy to get silently wrong: + +- Resumed from the sha256-verified durable snapshot's `last.pt` (see "Where the weights live"). +- **Only these keys were rewritten** inside the checkpoint's `train_args`: `project`, `name`, + `save_dir`, `model`, `resume`. **This rewrite is mandatory.** Ultralytics' `resume=True` + restores *every* arg from the checkpoint — including `save_dir` — so an un-rewritten fork + resumes into the **original** run directory and corrupts a live training arm. +- Asserted unchanged on read-back: `epoch=2`, `best_fitness=0.35579`, `epochs=60`, `lr0`, + `lrf`, `warmup_epochs=3.0`, `batch=6`, `imgsz=1024`, `seed=0`, `patience=20`, `data`. So the + fork is a faithful continuation of the same LR schedule, with optimizer and EMA state + intact — Ultralytics confirms `Resuming training … from epoch 4 to 60 total epochs`. +- **Two lineages now share epochs 1–3.** Any reported tiles number must say which lineage it + came from. In practice they may never diverge, since the ckpt original has completed no + epoch since 07-28. +- **What this does and does not buy.** Per-epoch time is unchanged by the move (same shared + filesystem); what changes is the uninterrupted window. At ~4.1 h/epoch, 72 h yields roughly + 17 epochs — enough to show whether tiles turns the corner (`y26_pano` turned by ep9), but + **not** a converged 60-epoch tiles run. + ### What is reportable today Epoch 1–2 are **not** a "pretrained artifact" — an earlier version of this record said so @@ -117,6 +214,20 @@ The honest caveat: these checkpoints are **undertrained** relative to a stable s any benchmark number from them is a **lower bound** on supervised-YOLO performance, and must be reported as such. +**As of 2026-07-29 that caveat is per-arm, not uniform** — an earlier version of this record +applied it to the whole grid, which now understates one arm and *overstates* the others: + +| arm | what `best.pt` actually is | how to report it | +|---|---|---| +| `y26_pano` | epoch 18 of 60, still improving | undertrained relative to a 60-epoch stable schedule, but a genuinely trained model. A normal "lower bound" caveat. | +| `y11l_pano`, `y11x_pano`, `y11l_tiles`, `y26_tiles` | **epoch 1 of 60** | "lower bound" is too soft. These are *one-epoch models*, and the write-up must say so in those words — a reader will otherwise assume `best.pt` came from a converged run. | + +The distinction matters in both directions. Discounting `y26_pano` as "a 1-epoch model" would +understate the supervised baseline; reporting the four frozen arms as merely "undertrained" +would overstate it. Whether `y11l_pano` clears its ep1 fitness before its wall (it is ~1–2 +epochs away) decides which row it belongs in, so **re-check `best.pt`'s mtime at eval time** +rather than trusting this table. + ## Pre-registered evaluation & checkpoint-selection protocol (issue #71) Fixed in writing 2026-07-28, **before any benchmark evaluation of any YOLO @@ -197,7 +308,11 @@ protocol governs the results that will replace that note. earlier draft of this record said `fixed:0.03` — that was the runbook's old default, not what ran.) - **Toolchain:** Ultralytics 8.4.105 · Python 3.11.15 · torch 2.13.0+cu126. -- **Hardware:** NVIDIA L40 (45 GB), 1 GPU/job, Hyak `ckpt-g2` (preemptable/requeue). +- **Hardware:** NVIDIA L40 (45 GB), 1 GPU/job, Hyak `ckpt-g2` (preemptable/requeue). The + `y26_tiles_l40s` fork instead runs on Hyak `gpu-l40s` (makelab's own allocation, + non-preemptable, `MaxTime=UNLIMITED`), 1 GPU, NVIDIA L40S (45 GB), node `g3100`. Because + ckpt schedules onto whatever GPU is idle (l40/l40s/h200), per-epoch wall time varies across + requeues even within one arm — see "The ckpt slice ceiling". - **Hyperparameters (resolved):** `epochs=60`, `patience=20`, `optimizer=auto`, `lr0=0.01`, `lrf=0.01`, `momentum=0.937`, `weight_decay=0.0005`, `warmup_epochs=3.0`, `close_mosaic=10`, `amp=true`, `seed=0`. Per-run detail in each `runs//args.yaml`. @@ -209,14 +324,41 @@ protocol governs the results that will replace that note. batch-12 resubmit 37809205, scancel'd 2026-07-27) · y26_tiles 37745360 · y11l_pano 37745361 · y11x_pano 37745362 · y26_pano 37745363. The job↔config map was read off the `yolo_train_.out` log headers during the 2026-07-27 preemption check. - Dataset-download job 37649635; prep job 37677130. -- **Run dates:** 2026-07-26 → in progress as of 2026-07-28. + Dataset-download job 37649635; prep job 37677130. The `y26_tiles_l40s` fork is job + **37889646** (`gpu-l40s`, submitted 2026-07-29 ~13:55 PT). +- **Run dates:** 2026-07-26 → in progress as of 2026-07-29. +- **Mail flags are not in the launcher.** The five original jobs carry + `MailType=END,FAIL,TIME_LIMIT`, which was passed on the 07-26 sbatch command line, not set by + `run_yolo_train.slurm` (a fresh submission of the same script gets no mail config). On a + preemptable partition that means one email per preemption — 115 of them on 2026-07-29 alone. + Worth omitting, or narrowing to `TIME_LIMIT`, for the #70 rerun. ## Where the weights live `best.pt` files are **not** in git. Durable homes: -_TODO — stage to Hugging Face (`projectsidewalk/…`) or lab storage and record the URL here._ +**Staged to durable lab storage on 2026-07-29, while the runs were still going:** + +``` +/gscratch/makelab/jonf/rampnet_yolo_baseline_51/ + /{best.pt,last.pt,results.csv,args.yaml} # 5 arms, 1.8 GB total + MANIFEST.md # Slurm job IDs, per-arm epochs/metrics, sha256 of every weight file +``` + +Arms: `y11l_pano`, `y11x_pano`, `y26_pano`, `y11l_tiles`, `y26_tiles`. All **10 weight files +were sha256-verified against the source** at copy time (10 ok, 0 mismatched), and the hashes +are recorded in `MANIFEST.md` so a later copy can be checked against them. + +This was done because the weights existed **only** on `/gscratch/scrubbed`, which auto-purges +after ~21 idle days — and the run directories date from 07-26, so the clock starts the moment +the jobs stop writing. `/gscratch/makelab` is not scrubbed. + +`last.pt` is kept alongside `best.pt` deliberately: `best.pt` is what the #71/#80 protocol +reports, but only `last.pt` carries the optimizer/EMA state needed to *resume* an arm (which is +exactly what the `y26_tiles_l40s` fork above needed). + +Still open: publishing to Hugging Face (`projectsidewalk/…`) if these become a released +artifact rather than an internal baseline. Lab storage is durable but not public. Keep **every** run's `best.pt`, including the ones that collapsed. An earlier version of this record called those non-reportable epoch-1 artifacts and proposed letting them expire; From eddd34f4c81653f25e658b23e7aad56d4312ddd6 Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Wed, 29 Jul 2026 15:37:08 -0700 Subject: [PATCH 2/4] Record that the preemption-mail volume is accepted, not a defect (#51) The provenance note said the mail flags were "worth omitting" for the #70 rerun. Jon's call is the opposite -- the emails are fine and easy to delete -- so the record now says the volume is deliberate and explains what it is useful for (reading the run history), while keeping the actionable detail: the flags come from the sbatch command line, not the launcher, so a quieter rerun has to set them per job with scontrol update. Co-Authored-By: Claude Opus 5 (1M context) --- scripts/model_comparison/yolo_baseline/README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/scripts/model_comparison/yolo_baseline/README.md b/scripts/model_comparison/yolo_baseline/README.md index b06e00d..fad0faf 100644 --- a/scripts/model_comparison/yolo_baseline/README.md +++ b/scripts/model_comparison/yolo_baseline/README.md @@ -331,7 +331,10 @@ protocol governs the results that will replace that note. `MailType=END,FAIL,TIME_LIMIT`, which was passed on the 07-26 sbatch command line, not set by `run_yolo_train.slurm` (a fresh submission of the same script gets no mail config). On a preemptable partition that means one email per preemption — 115 of them on 2026-07-29 alone. - Worth omitting, or narrowing to `TIME_LIMIT`, for the #70 rerun. + **This volume is accepted deliberately** (Jon, 2026-07-29: "I'm fine with still getting the + mail"), so it is recorded here as context for reading the run history, not as a defect to fix. + A future rerun that wants quiet should omit the flags or narrow them to `TIME_LIMIT`; note they + must be set per-job with `scontrol update`, since the launcher never sets them. ## Where the weights live From d7cf8f0bad7288b2512211cd6e23faa0a69071dc Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Wed, 29 Jul 2026 15:37:53 -0700 Subject: [PATCH 3/4] Mark the L40S epoch time as projected, not measured (#51) The 4.1 h/epoch figure for the fork was written as if measured. It is an extrapolation from 6.2 it/s at 12,123 of 92,903 iterations -- 13% of the fork'"'"'s first epoch -- and no epoch has completed on that partition yet. In a record whose whole point is that numbers be re-derivable, a projection stated as a measurement is the wrong kind of claim, and this one is load-bearing: it is the basis for "~17 epochs in 72 h." Says where the real number will come from (the time column of y26_tiles_l40s/results.csv) so it gets replaced rather than inherited. Co-Authored-By: Claude Opus 5 (1M context) --- scripts/model_comparison/yolo_baseline/README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/scripts/model_comparison/yolo_baseline/README.md b/scripts/model_comparison/yolo_baseline/README.md index fad0faf..9d96ec1 100644 --- a/scripts/model_comparison/yolo_baseline/README.md +++ b/scripts/model_comparison/yolo_baseline/README.md @@ -166,7 +166,10 @@ WARNING ⚠️ train: Slow image access detected (ping: 0.6±0.2 ms, read: 2.1± …across 557,413 train tiles. So tiles epoch time is **node- and contention-dependent, not a fixed property of the config**: ~5.2 h on a good ckpt landing (epochs 1–3, 07-26/27), ~9.5 h on a contended one (measured 07-29: 54% of an epoch in 5.15 h), and **~4.1 h on a dedicated -`gpu-l40s` L40S** with no allocation-mates competing for the filesystem. Note this also means +`gpu-l40s` L40S** with no allocation-mates competing for the filesystem. That last figure is +**projected, not yet measured** — extrapolated from 6.2 it/s at 12,123/92,903 iterations (13% of +the fork's first epoch); replace it with the `time` column of `y26_tiles_l40s/results.csv` once an +epoch completes. Note this also means a faster GPU alone would not help; and `gpu-l40s` nodes report `TMP_DISK=0`, so the dataset cannot be staged to node-local disk to fix it. From 7afa60c8c6c13f23ebcd19b0cd16b3f5a0ceea4f Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Wed, 29 Jul 2026 20:49:31 -0700 Subject: [PATCH 4/4] Update the record to the 20:45 PT check: two arms cleared ep1, epoch time measured (#51) Three substantive changes, all from live data six hours after the previous check. y11l_pano unfroze its best.pt at ep14 (mAP50-95 0.437 > ep1 0.423 after 12 epochs unbeaten), and y26_pano advanced to ep20. So two of five arms now hold genuinely trained checkpoints and only three are one-epoch models. The per-arm caveat table is re-sorted accordingly, and now says outright that it has already moved once so the split must be re-derived at eval time rather than inherited. The fork'"'"'s epoch time is measured rather than projected: 4.94 h from the time column of its completed ep4, against the 4.1 h I extrapolated from 13% of that epoch -- about 20% low, the usual direction. Its ep5 is tracking ~4.0 h, so the dedicated-node figure is a ~4-5 h range and the 72 h budget buys ~14 epochs rather than ~17. The fork'"'"'s first new epoch also landed: ep4 is slightly below ep3, which is where the LR schedule puts it, so the recovery question stays open until ~ep9. Refines the ceiling finding in a way that matters for planning: y11x_pano has now also stopped advancing -- no epoch in ~15 h -- despite a ~3.4 h epoch that fits the 8.24 h nominal cap with room to spare. Its segments that evening were 0:58, 2:30, 0:09, 0:18, 0:02, 0:27. So the constraint is the effective preemption interval, not the slice cap, and "epoch < 8.24 h is safe" is the wrong rule. The arm is not wedged; its counter is live at ep10. Co-Authored-By: Claude Opus 5 (1M context) --- .../model_comparison/yolo_baseline/README.md | 88 ++++++++++++------- 1 file changed, 56 insertions(+), 32 deletions(-) diff --git a/scripts/model_comparison/yolo_baseline/README.md b/scripts/model_comparison/yolo_baseline/README.md index 9d96ec1..5911da7 100644 --- a/scripts/model_comparison/yolo_baseline/README.md +++ b/scripts/model_comparison/yolo_baseline/README.md @@ -58,7 +58,7 @@ There is a seventh **run directory** on scratch, `y26_tiles_l40s`, but it is **n config**: it is `y26_tiles` itself, resumed from its own epoch-3 checkpoint on a different partition with every hyperparameter unchanged. See "The `y26_tiles_l40s` fork" below. -## Status & findings (live check: 2026-07-29 ~14:45 PT, training in progress) +## Status & findings (live check: 2026-07-29 ~20:45 PT, training in progress) Val-split proxy mAP50 from `results.csv`. **Every config peaks at epoch 1, collapses at epoch 3, and recovers as the learning rate decays.** The instability is universal, not @@ -73,21 +73,28 @@ per-config — see the figures. | Config | Epochs | best-val (ep) | Latest mAP50 | `best.pt` holds | Assessment | |------------------|--------|---------------|--------------|-----------------|------------| -| `y26_pano` | 18 | 0.738 (1) | **0.717 ↑** | **ep18 — current** | ✅ round-tripped 0.738 → 0.125 @ep6 → 0.717 @ep18; the only arm whose `best.pt` has cleared ep1 | -| `y11l_pano` | 13 | 0.779 (1) | **0.727 ↑** | ep1 | 🟡 recovering hard — 0.000 @ep4–5 → 0.183 @ep10 → 0.727 @ep13; ~1–2 epochs from unfreezing `best.pt` | -| `y11x_pano` | 9 | 0.777 (1) | 0.061 ↑ | ep1 | ❌ five epochs at literal 0; `patience=20` retires it ~ep22, almost certainly still holding ep1 | +| `y26_pano` | 20 | 0.738 (1) | **0.730 ↑** | **ep20 — current** | ✅ round-tripped 0.738 → 0.125 @ep6 → 0.730 @ep20; mAP50-95 0.475 ≫ ep1's 0.413 | +| `y11l_pano` | 14 | 0.779 (1) | **0.767 ↑** | **ep14 — current** | ✅ **unfroze at ep14** — 0.000 @ep4–5 → 0.183 @ep10 → 0.767 @ep14; mAP50-95 0.437 > ep1's 0.423 | +| `y11x_pano` | 9 | 0.777 (1) | 0.061 ↑ | ep1 | ❌ five epochs at literal 0, **and now starved** — no completed epoch in ~15 h (see below) | | `y26_tiles` | 3 | 0.647 (1) | 0.280 ↓ | ep1 | ⏸ **blocked** — no completed epoch since 2026-07-28; forked to `gpu-l40s` (below) | | `y11l_tiles` | 3 | 0.655 (1) | 0.042 ↓ | ep1 | ⏸ **blocked** — no completed epoch since 2026-07-28 | -| `y26_tiles_l40s` | 3 (+) | inherited | — | inherited ep1 | 🆕 fork of `y26_tiles` resumed at ep4 on `gpu-l40s`, ~4.1 h/epoch (see "The ckpt slice ceiling") | +| `y26_tiles_l40s` | 4 | 0.647 (1) | 0.268 ↓ | inherited ep1 | 🆕 fork; **first new epoch landed** (ep4, 4.94 h) — still in the dip, as the LR schedule predicts | | `y11x_tiles` | 0 | — | — | — | ⏹ dropped 2026-07-27 (GPU-saturated: epoch ~10 h > ckpt slice) | -**`y26_pano`'s `best.pt` is epoch 18, not epoch 1.** Its `best.pt` and `last.pt` are -byte-identical (both sha256 `aac93e74…50bf`), which is direct evidence rather than an -inference from mtimes: mAP50-95 0.460 @ep18 vs 0.413 @ep1. The other four arms' `best.pt` -files are frozen at **epoch 1** — mtimes 2026-07-26 20:22 / 21:42 / 23:35 / 23:58, each -roughly one epoch after the 18:10 job start, while their `last.pt` files are current. For -`y26_tiles` this is confirmed exactly: the checkpoint's `best_fitness` is 0.35579, which is -its epoch-1 mAP50-95 to five decimals. +**Two arms have now cleared epoch 1, so the grid is genuinely split.** For `y26_pano` the +evidence is direct rather than an mtime inference: its `best.pt` and `last.pt` were +byte-identical at ep18 (both sha256 `aac93e74…50bf`), and it has since advanced to ep20 +(mAP50-95 0.475 vs 0.413 @ep1). **`y11l_pano` unfroze at ep14** on 2026-07-30 02:29Z — +`best.pt` mtime now equals `results.csv` mtime, and mAP50-95 0.437 finally beat ep1's 0.423 +after 12 epochs unbeaten. This was predicted one check earlier and is worth noting as +confirmation that the recovery is real, not noise. + +The remaining **three** arms' `best.pt` files are still frozen at **epoch 1** — `y11x_pano`, +`y11l_tiles`, `y26_tiles`, mtimes 2026-07-26 21:42 / 23:35 / 23:58, each roughly one epoch +after the 18:10 job start, while their `last.pt` files are current. For `y26_tiles` this is +confirmed exactly: the checkpoint's `best_fitness` is 0.35579, which is its epoch-1 mAP50-95 +to five decimals. **This split moves — re-check mtimes at eval time rather than trusting this +table.** ### The instability: collapse tracks the warmup LR peak @@ -165,18 +172,28 @@ WARNING ⚠️ train: Slow image access detected (ping: 0.6±0.2 ms, read: 2.1± …across 557,413 train tiles. So tiles epoch time is **node- and contention-dependent, not a fixed property of the config**: ~5.2 h on a good ckpt landing (epochs 1–3, 07-26/27), ~9.5 h -on a contended one (measured 07-29: 54% of an epoch in 5.15 h), and **~4.1 h on a dedicated -`gpu-l40s` L40S** with no allocation-mates competing for the filesystem. That last figure is -**projected, not yet measured** — extrapolated from 6.2 it/s at 12,123/92,903 iterations (13% of -the fork's first epoch); replace it with the `time` column of `y26_tiles_l40s/results.csv` once an -epoch completes. Note this also means +on a contended one (measured 07-29: 54% of an epoch in 5.15 h), and **4.94 h on a dedicated +`gpu-l40s` L40S** with no allocation-mates competing for the filesystem — now **measured**, from +the `time` column of the fork's completed epoch 4 (17,774.6 s). Its epoch 5 is tracking a little +faster (~4.0 h at 6.4 it/s), so treat **~4–5 h** as the dedicated-node range rather than a single +figure. An earlier draft of this file projected 4.1 h from 13% of an in-flight epoch; the real +value came in ~20% higher, which is the usual direction for an early-epoch extrapolation. Note this also means a faster GPU alone would not help; and `gpu-l40s` nodes report `TMP_DISK=0`, so the dataset cannot be staged to node-local disk to fix it. -**Consequence for the record.** Left on ckpt, the three pano arms plausibly reach the -60-epoch budget or their `patience` stop, while the two tiles arms stay at epoch 1 -indefinitely. That is a **structural gap, not a slow one**, and reporting it as "undertrained" -would misdescribe it. +**Refinement (2026-07-29 20:45 PT): the binding constraint is the *effective* preemption +interval, not the 8.24 h nominal cap.** `y11x_pano` has now also stopped advancing — no completed +epoch in ~15 h — despite an epoch of only ~3.4 h, which fits the nominal ceiling with room to +spare. Its segments that evening ran 0:58, 2:30, 0:09, 0:18, 0:02, 0:27. So the rule is not +"epoch < 8.24 h is safe"; it is **epoch < whatever contiguous run the partition currently +hands out**, and that collapsed to minutes. The arm is *not* wedged — its progress counter is +live (ep10, 12%, 7.2 it/s) — it simply keeps losing partial epochs. Any future capacity plan on +`ckpt` should budget against the *observed* segment distribution, not the slice cap. + +**Consequence for the record.** Left on ckpt, `y26_pano` and `y11l_pano` (epochs ~2.6–2.8 h) +plausibly reach the 60-epoch budget or their `patience` stop, while the two tiles arms — and now +`y11x_pano` — stay frozen at their current `best.pt`. That is a **structural gap, not a slow +one**, and reporting it as "undertrained" would misdescribe it. #### The `y26_tiles_l40s` fork (job `37889646`) @@ -202,9 +219,14 @@ Provenance, because a fork is easy to get silently wrong: came from. In practice they may never diverge, since the ckpt original has completed no epoch since 07-28. - **What this does and does not buy.** Per-epoch time is unchanged by the move (same shared - filesystem); what changes is the uninterrupted window. At ~4.1 h/epoch, 72 h yields roughly - 17 epochs — enough to show whether tiles turns the corner (`y26_pano` turned by ep9), but - **not** a converged 60-epoch tiles run. + filesystem); what changes is the uninterrupted window. At the measured ~4–5 h/epoch, 72 h yields + roughly **14 epochs**, reaching ~ep18 — enough to show whether tiles turns the corner + (`y26_pano` turned by ep9), but **not** a converged 60-epoch tiles run. +- **First fork epoch is in: ep4 = mAP50 0.268 / mAP50-95 0.111**, marginally *below* ep3's + 0.280 / 0.119, with precision 0.497 and recall 0.201. This is the expected place in the + schedule, not a bad sign: ep3 is the warmup LR peak and the pano arms bottomed at ep4–ep6 + before turning. The same "stops firing" signature (recall collapses, precision holds) is + present. **The recovery question stays open until ~ep9–10**, i.e. ~20–25 h of fork time. ### What is reportable today @@ -222,14 +244,16 @@ applied it to the whole grid, which now understates one arm and *overstates* the | arm | what `best.pt` actually is | how to report it | |---|---|---| -| `y26_pano` | epoch 18 of 60, still improving | undertrained relative to a 60-epoch stable schedule, but a genuinely trained model. A normal "lower bound" caveat. | -| `y11l_pano`, `y11x_pano`, `y11l_tiles`, `y26_tiles` | **epoch 1 of 60** | "lower bound" is too soft. These are *one-epoch models*, and the write-up must say so in those words — a reader will otherwise assume `best.pt` came from a converged run. | - -The distinction matters in both directions. Discounting `y26_pano` as "a 1-epoch model" would -understate the supervised baseline; reporting the four frozen arms as merely "undertrained" -would overstate it. Whether `y11l_pano` clears its ep1 fitness before its wall (it is ~1–2 -epochs away) decides which row it belongs in, so **re-check `best.pt`'s mtime at eval time** -rather than trusting this table. +| `y26_pano` (ep20), `y11l_pano` (ep14) | genuinely trained, still improving | undertrained relative to a 60-epoch stable schedule, but real models. A normal "lower bound" caveat. | +| `y11x_pano`, `y11l_tiles`, `y26_tiles` | **epoch 1 of 60** | "lower bound" is too soft. These are *one-epoch models*, and the write-up must say so in those words — a reader will otherwise assume `best.pt` came from a converged run. | + +The distinction matters in both directions. Discounting `y26_pano` or `y11l_pano` as "1-epoch +models" would understate the supervised baseline; reporting the three frozen arms as merely +"undertrained" would overstate it. + +**This table has already moved once** — `y11l_pano` was in the bottom row at the 14:45 PT check +and moved up by 20:45 PT, six hours later. So **re-check each `best.pt`'s mtime against its +`results.csv` at eval time** and place the arms then; do not inherit this split. ## Pre-registered evaluation & checkpoint-selection protocol (issue #71)