diff --git a/.gitignore b/.gitignore index f87b334..68972f9 100644 --- a/.gitignore +++ b/.gitignore @@ -30,3 +30,8 @@ analysis_out/ # the auto-labeler. Belongs with the imagery, which is intentionally not in git # (see benchmark/README.md) -- nothing in this repo reads it. benchmark/*/index.csv + +# Model weights never go in git -- durable homes are HF / lab storage (see +# scripts/model_comparison/yolo_baseline/README.md, "Where the weights live"). +*.pt +*.pth diff --git a/docs/model_comparison.md b/docs/model_comparison.md index 1047766..2822b46 100644 --- a/docs/model_comparison.md +++ b/docs/model_comparison.md @@ -26,6 +26,26 @@ Grounding DINO. Every split now carries the full roster; the remaining ❌s are GT-completeness correction (#55, done on two cities) and the null-recall pass on `manual_gold` (O(n²) in panos — see that section). +**Supervised baseline in progress (issue #51).** The roster above is all zero-shot except +RampNet. A supervised **YOLO** baseline — the architecture-vs-data ablation (does a generic +detector trained on the RampNet dataset *also* beat the zero-shot field, or is RampNet's +keypoint architecture doing the work?) — is training on Hyak but is **not yet in the results +tables**: benchmark eval is pending. Live status is in #51; the training record (per-epoch +curves, configs, provenance) is committed under +[`scripts/model_comparison/yolo_baseline/`](../scripts/model_comparison/yolo_baseline/). +Known gaps and caveats so far: the `y11x_tiles` config was dropped (GPU-saturated — epoch time +exceeded the ckpt scheduling slice, so it never checkpointed), and **all five remaining configs +hit the same training instability** — validation mAP peaks at epoch 1, collapses at epoch 3, and +recovers as the learning rate decays, while training loss falls straight through with no NaN or +AMP error. The collapse coincides exactly with the peak of the warmup LR ramp (`lr/pg0` 0.010 → +0.029 over `warmup_epochs=3`), and **not** with batch size, input resolution, architecture, or the +ckpt preemptions — `y11l_pano` and `y26_pano` share batch 4 / imgsz 1280 and both collapsed and +recovered. Figures and the full ruling-out are in the training record; the stabilized rerun is +tracked in #70 (whose original small-batch hypothesis this evidence refutes) and the caveat +write-up in #72. The reported baseline will be the best-val checkpoint, selected on val, per the +protocol in #71 — with the caveat that today's checkpoints are undertrained relative to a stable +schedule and so give a **lower bound** on supervised-YOLO performance. + Three classes of challenger, which fail differently and are worth keeping distinct: | class | models | output | tunable? | diff --git a/scripts/model_comparison/yolo_baseline/README.md b/scripts/model_comparison/yolo_baseline/README.md new file mode 100644 index 0000000..861fe6c --- /dev/null +++ b/scripts/model_comparison/yolo_baseline/README.md @@ -0,0 +1,175 @@ +# Supervised YOLO baseline — training record (issue #51) + +This directory is the **preserved record** of the supervised-YOLO baseline runs +trained on the RampNet dataset on Hyak (klone). It exists so the experiment survives +`/gscratch/scrubbed`'s ~21-day auto-purge and is reproducible/defensible for the paper. + +> **Status:** the runs are in-flight (issue #51); live status is tracked there and the +> finding is summarized in `docs/model_comparison.md`. The training code lives at its +> canonical paths — `scripts/model_comparison/run_yolo_train.slurm` and the repo-root +> `hyak_yolo_runbook.sh` — merged in PR #76. (This record briefly carried as-run +> snapshots of both; the only drift from the canonical copies was a display-only +> pano-count bug, documented in #76's description, and the snapshots remain in this +> branch's history.) + +**What's here (small, text, committed):** + +- `runs//results.csv` — per-epoch training curves (the primary evidence). +- `runs//args.yaml` — the exact resolved config for each run. +- `plot_training_curves.py` — regenerates every figure below from those CSVs alone + (CPU-only, no network, no checkpoints): `python plot_training_curves.py`. +- `figures/*.png` — the generated diagnostics, committed so the finding is legible + without re-running anything. + +**What's deliberately NOT here** (see the repo `.gitignore` philosophy — curated record +in git, bulk/binary/regenerable out): + +- **Model weights** (`best.pt`/`last.pt`) — binaries; they belong in durable external + storage, not git history. See "Where the weights live" below. +- **Slurm logs** (`*.out`/`*.err`) — 30 MB of progress-bar spam, already gitignored; + the few relevant lines are quoted below. +- **Benchmark-city eval numbers** — these `results.csv` values are the **internal YOLO + val-split proxy** used to judge *training health*, **not** the reported baseline. The + reported numbers come from local `compare.py` benchmark eval on each `best.pt`. + +## Config grid + +Six configs = {YOLO11l, YOLO11x, YOLO26l} × {tiles @1024, pano @1280}. One dropped. + +| Config | Model | Input | Base ckpt | Batch | +|--------------|----------|--------------|---------------|-------| +| `y11l_tiles` | YOLO11l | tiles @1024 | `yolo11l.pt` | 6 | +| `y11x_tiles` | YOLO11x | tiles @1024 | `yolo11x.pt` | 3→12 | +| `y26_tiles` | YOLO26l | tiles @1024 | `yolo26l.pt` | 6 | +| `y11l_pano` | YOLO11l | pano @1280 | `yolo11l.pt` | 4 | +| `y11x_pano` | YOLO11x | pano @1280 | `yolo11x.pt` | 2 | +| `y26_pano` | YOLO26l | pano @1280 | `yolo26l.pt` | 4 | + +Batches are the as-run values from each run's committed `args.yaml`. `y11x_tiles` was +submitted at batch 3, then resubmitted at batch 12 trying to finish an epoch inside the +ckpt scheduling slice (its `args.yaml` is the batch-12 attempt); neither completed one. + +## Status & findings (snapshot: 2026-07-28, training in progress) + +Val-split proxy mAP50 from `results.csv`. **Every config peaks at epoch 1, collapses at +epoch 3, and recovers as the learning rate decays.** The instability is universal, not +per-config — see the figures. + +| Config | Epochs | best (ep) | Current mAP50 | Assessment | +|--------------|--------|-----------|---------------|------------| +| `y26_pano` | 14 | 0.738 (1) | **0.659 ↑** | ✅ fully round-tripped: 0.738 → 0.125 @ep6 → 0.659 @ep14, climbing | +| `y11l_pano` | 10 | 0.779 (1) | **0.183 ↑** | 🟡 recovering — 7 epochs near 0, then 0.005 → 0.183 @ep10 | +| `y26_tiles` | 3 | 0.647 (1) | 0.280 ↓ | 🟡 in the dip, too early to call | +| `y11l_tiles` | 3 | 0.655 (1) | 0.042 ↓ | 🟡 in the dip, too early to call | +| `y11x_pano` | 7 | 0.777 (1) | **0.000** | ❌ five straight epochs at literal 0; no recovery yet | +| `y11x_tiles` | 0 | — | — | ⏹ dropped 2026-07-27 (GPU-saturated: epoch ~10 h > ckpt slice) | + +### The instability: collapse tracks the warmup LR peak + +![learning curves](figures/fig1_learning_curves.png) + +The `lr/pg0` column ramps **~3× during warmup** — 0.0100 (ep1) → 0.0197 (ep2) → **0.0290 +(ep3)** — then decays linearly. `warmup_epochs=3.0`. Every config's validation collapse +begins at that peak, and every recovery so far tracks the decay. + +What the evidence rules **in** and **out**: + +- **Not a crash.** No NaN/Inf, no AMP failure anywhere in the logs; training loss keeps + falling straight through the collapse while val `cls_loss` explodes 1.2 → 8.2 + (`figures/fig3_loss_divergence.png`). A textbook optimization instability. +- **Not the preemptions.** ckpt requeues are scattered across epochs 2–13 and do not line + up with the collapses (`figures/fig4_per_config.png`, purple vs grey lines). `resume=True` + restores the full training state; `y26_pano` climbed straight through five of them. +- **Not small physical batch.** `y11l_pano` (batch 4, imgsz 1280) and `y26_pano` + (**batch 4, imgsz 1280** — identical) both collapsed, and both recovered. Batch spans + 2/4/6 and imgsz 1024/1280 across the grid with no separation. This **refutes** the + BatchNorm/small-batch hypothesis this record previously carried. +- **Not architecture.** Both YOLO11 and YOLO26 collapse; both have now begun recovering. + Only the *rate* differs (`y26_pano` recovered by ep9; `y11l_pano` took until ep10). +- **Not a data fault.** A broken dataset would depress train loss too, and would not + recover on an LR schedule. + +**Hypothesis (untested):** the effective peak LR at the end of warmup is too high for this +task — 150k single-class, small-object images, fine-tuned from COCO weights. `optimizer=auto` +resolves to **`MuSGD(lr=0.01, momentum=0.9)`** in every job (from the Slurm logs), Ultralytics' +Muon-family optimizer. The clean test is a rerun with a lower peak LR / longer warmup, **not** +a larger batch. + +### Failure signature: recall collapse, not false-positive flood + +![precision and recall](figures/fig2_precision_recall.png) + +The model **stops firing** rather than starting to guess: `y11l_pano` holds precision at +0.94–1.00 while recall sits at 0.007–0.03, and `y11x_pano` emits no boxes at all (its +precision reads 0 by the 0/0 convention, not from false positives). This matters for +interpretation — the instability suppresses detections, so a checkpoint caught inside the +dip understates recall catastrophically and would badly misrepresent the baseline. + +### What is reportable today + +Epoch 1–2 are **not** a "pretrained artifact" — an earlier version of this record said so +and that was wrong. By epoch 1 the model has trained on all 150k images; the score is real, +and it is high because the LR was still in the low part of the warmup ramp. Each run's +`best.pt` therefore holds a **genuine, selectable checkpoint** under the best-val protocol. + +The honest caveat: these checkpoints are **undertrained** relative to a stable schedule, so +any benchmark number from them is a **lower bound** on supervised-YOLO performance, and must +be reported as such. + +## Provenance + +- **Training code:** `scripts/model_comparison/run_yolo_train.slurm` + the repo-root + `hyak_yolo_runbook.sh` (PR #76). (The Hyak runs used an rsync of the working tree; + the committed launcher is byte-identical to the as-run copy.) +- **Dataset:** `projectsidewalk/rampnet-dataset` (HF), pulled onto the cluster via + `download_dataset.py`. Prep: `prepare_yolo_dataset.py` via `run_yolo_prep.slurm` — + `--box-size pitch --ramp-size-m 1.8 --bg-keep-frac 0.15` per that launcher's as-run + defaults (its header records that `fixed:0.03` was rejected as too small and that + `gps` fell back to pitch on ~85% of panos). Tiles rendered at 1024 (train imgsz + 1024); pano at 2048×1024 (train imgsz 1280). + **Confirmed** from the run notes (2026-07-25, prep job 37677130, ~2h48m): box-size + `pitch` + ramp 1.8 m was chosen after eyeballing the overlay QA — `fixed:0.03` was + too small and `gps` fell back to pitch on 169/200 smoke panos. Output verified on + scratch: tiles 557,413 train / 161,002 val images; pano 150,063 / 42,875. (An + earlier draft of this record said `fixed:0.03` — that was the runbook's old + default, not what ran.) +- **Toolchain:** Ultralytics 8.4.105 · Python 3.11.15 · torch 2.13.0+cu126. +- **Hardware:** NVIDIA L40 (45 GB), 1 GPU/job, Hyak `ckpt-g2` (preemptable/requeue). +- **Hyperparameters (resolved):** `epochs=60`, `patience=20`, `optimizer=auto`, + `lr0=0.01`, `lrf=0.01`, `momentum=0.937`, `weight_decay=0.0005`, `warmup_epochs=3.0`, + `close_mosaic=10`, `amp=true`, `seed=0`. Per-run detail in each `runs//args.yaml`. + **`optimizer=auto` resolved to `MuSGD(lr=0.01, momentum=0.9)`** in all six jobs, which + overrides the passed `lr0`/`momentum` (the Slurm logs say so explicitly). The realized + schedule peaks at `lr/pg0 = 0.029` at the end of warmup — recorded per-epoch in the + `lr/pg*` columns of every `results.csv`, and the subject of the instability above. +- **Slurm job IDs:** y11l_tiles 37745358 · y11x_tiles 37745359 (batch-3 original; + batch-12 resubmit 37809205, scancel'd 2026-07-27) · y26_tiles 37745360 · y11l_pano + 37745361 · y11x_pano 37745362 · y26_pano 37745363. The job↔config map was read off + the `yolo_train_.out` log headers during the 2026-07-27 preemption check. + Dataset-download job 37649635; prep job 37677130. +- **Run dates:** 2026-07-26 → in progress as of 2026-07-28. + +## Where the weights live + +`best.pt` files are **not** in git. Durable homes: + +_TODO — stage to Hugging Face (`projectsidewalk/…`) or lab storage and record the URL here._ + +Keep **every** run's `best.pt`, including the ones that collapsed. An earlier version of +this record called those non-reportable epoch-1 artifacts and proposed letting them expire; +that was wrong on both counts (see "What is reportable today"). Each is a real best-val +checkpoint and the only supervised-YOLO baseline available until a stable schedule lands. + +## Reproducing + +On klone, via the repo-root runbook (each stage is idempotent; the multi-hour `data` / +`prep` stages have compute-node sbatch wrappers, `run_yolo_data.slurm` / +`run_yolo_prep.slurm` — login nodes reap heavy processes): + +```bash +bash hyak_yolo_runbook.sh env # lean venv on scratch (ultralytics + cu126 torch) +bash hyak_yolo_runbook.sh data # pull RampNet dataset from HF (hours) +bash hyak_yolo_runbook.sh prep # build tiles/ and pano/ YOLO datasets +bash hyak_yolo_runbook.sh train # sbatch the 6 configs concurrently +bash hyak_yolo_runbook.sh collect # rsync best.pt back; eval locally with compare.py +``` diff --git a/scripts/model_comparison/yolo_baseline/figures/fig1_learning_curves.png b/scripts/model_comparison/yolo_baseline/figures/fig1_learning_curves.png new file mode 100644 index 0000000..1501a33 Binary files /dev/null and b/scripts/model_comparison/yolo_baseline/figures/fig1_learning_curves.png differ diff --git a/scripts/model_comparison/yolo_baseline/figures/fig2_precision_recall.png b/scripts/model_comparison/yolo_baseline/figures/fig2_precision_recall.png new file mode 100644 index 0000000..2e13c2a Binary files /dev/null and b/scripts/model_comparison/yolo_baseline/figures/fig2_precision_recall.png differ diff --git a/scripts/model_comparison/yolo_baseline/figures/fig3_loss_divergence.png b/scripts/model_comparison/yolo_baseline/figures/fig3_loss_divergence.png new file mode 100644 index 0000000..b774cf4 Binary files /dev/null and b/scripts/model_comparison/yolo_baseline/figures/fig3_loss_divergence.png differ diff --git a/scripts/model_comparison/yolo_baseline/figures/fig4_per_config.png b/scripts/model_comparison/yolo_baseline/figures/fig4_per_config.png new file mode 100644 index 0000000..51e94b4 Binary files /dev/null and b/scripts/model_comparison/yolo_baseline/figures/fig4_per_config.png differ diff --git a/scripts/model_comparison/yolo_baseline/plot_training_curves.py b/scripts/model_comparison/yolo_baseline/plot_training_curves.py new file mode 100644 index 0000000..ad3c541 --- /dev/null +++ b/scripts/model_comparison/yolo_baseline/plot_training_curves.py @@ -0,0 +1,310 @@ +#!/usr/bin/env python3 +"""Training-diagnostic figures for the supervised-YOLO baseline (issue #51). + +Reads the committed per-epoch records in ``runs//results.csv`` (pulled from +the Hyak run directories, which live on ``/gscratch/scrubbed`` and auto-purge) and +writes the standard set of figures used to diagnose a detection training run: + + fig1_learning_curves val mAP50 / mAP50-95 vs epoch, with the LR schedule overlaid + fig2_precision_recall val precision and recall vs epoch, separately + fig3_loss_divergence train vs val loss per component, one panel per config + fig4_per_config per-config mAP50 with LR and preemption/resume markers + +Everything here is CPU-only and reads nothing but the committed CSVs, so the +figures are reproducible from a clean checkout: + + python scripts/model_comparison/yolo_baseline/plot_training_curves.py + +Why the LR overlay is on the headline figure: every config's validation collapse +begins at the epoch where the warmup ramp peaks, and recovery tracks the decay. +See README.md for the reading. +""" + +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +import matplotlib + +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import pandas as pd + +HERE = Path(__file__).resolve().parent + +# Display order and styling. Geometry picks the linestyle, model family the colour, +# so "is this a pano or a tiles run?" and "is this v11 or v26?" are both readable +# without consulting the legend. +CONFIGS = [ + # name, label, colour, linestyle + ("y26_pano", "YOLO26 pano (b4, 1280)", "#1b6ca8", "-"), + ("y11l_pano", "YOLO11l pano (b4, 1280)", "#c1121f", "-"), + ("y11x_pano", "YOLO11x pano (b2, 1280)", "#e07a5f", "-"), + ("y26_tiles", "YOLO26 tiles (b6, 1024)", "#457b9d", "--"), + ("y11l_tiles", "YOLO11l tiles (b6, 1024)", "#8b1e3f", "--"), +] + +MAP50 = "metrics/mAP50(B)" +MAP5095 = "metrics/mAP50-95(B)" +PRECISION = "metrics/precision(B)" +RECALL = "metrics/recall(B)" +LR = "lr/pg0" + + +def load_runs(runs_dir: Path) -> dict[str, pd.DataFrame]: + """Load every config that has a results.csv, preserving CONFIGS order. + + A config with no results.csv is skipped with a note rather than an error -- + y11x_tiles was dropped before it finished epoch 1 and has args.yaml only, and + that absence is itself part of the record. + """ + runs: dict[str, pd.DataFrame] = {} + for name, *_ in CONFIGS: + path = runs_dir / name / "results.csv" + if not path.exists(): + print(f" skip {name}: no results.csv (dropped before epoch 1?)") + continue + df = pd.read_csv(path) + df.columns = [c.strip() for c in df.columns] + runs[name] = df + return runs + + +def resume_epochs(df: pd.DataFrame) -> list[int]: + """Epochs at which the training process restarted. + + Ultralytics writes ``time`` as seconds elapsed in the *current* process, so it + resets on every requeue. A decrease therefore marks a ckpt preemption + resume. + """ + t = df["time"].to_numpy() + return [int(df["epoch"].iloc[i]) for i in range(1, len(t)) if t[i] < t[i - 1]] + + +def warmup_peak_epoch(df: pd.DataFrame) -> int | None: + """Epoch where the LR schedule peaks (end of the warmup ramp).""" + if LR not in df.columns or df[LR].isna().all(): + return None + return int(df["epoch"].iloc[int(df[LR].to_numpy().argmax())]) + + +def _style(ax, xlabel="epoch", ylabel=None, title=None): + ax.grid(alpha=0.25, linewidth=0.6) + ax.set_xlabel(xlabel) + if ylabel: + ax.set_ylabel(ylabel) + if title: + ax.set_title(title, fontsize=11) + ax.spines[["top", "right"]].set_visible(False) + + +def fig_learning_curves(runs, out: Path) -> None: + fig, axes = plt.subplots(1, 2, figsize=(13, 4.8)) + + for metric, ax, label in ((MAP50, axes[0], "val mAP@50"), + (MAP5095, axes[1], "val mAP@50-95")): + for name, lbl, colour, ls in CONFIGS: + if name not in runs: + continue + df = runs[name] + ax.plot(df["epoch"], df[metric], color=colour, linestyle=ls, + marker="o", markersize=3.5, linewidth=1.8, label=lbl, zorder=3) + _style(ax, ylabel=label, title=label + " vs epoch") + ax.set_ylim(bottom=-0.02) + + # LR overlay on the headline panel only -- the point is the alignment, and + # repeating it on both panels just adds ink. + ref = next(iter(runs.values())) + twin = axes[0].twinx() + twin.plot(ref["epoch"], ref[LR], color="#666666", linestyle=":", linewidth=1.6, + label="learning rate", zorder=1) + twin.set_ylabel("learning rate (lr/pg0)", color="#666666") + twin.tick_params(axis="y", colors="#666666") + twin.spines[["top"]].set_visible(False) + + peak = warmup_peak_epoch(ref) + if peak is not None: + for ax in axes: + ax.axvline(peak, color="#666666", linestyle="-", linewidth=1.0, alpha=0.5, zorder=1) + axes[0].annotate( + f"warmup peak\n(epoch {peak}, lr={ref[LR].max():.3f})", + xy=(peak, axes[0].get_ylim()[1] * 0.92), xytext=(peak + 0.5, axes[0].get_ylim()[1] * 0.80), + fontsize=8.5, color="#444444", + arrowprops=dict(arrowstyle="->", color="#888888", linewidth=0.9), + ) + + axes[0].legend(fontsize=8, loc="center right", framealpha=0.92) + fig.suptitle( + "Supervised YOLO baseline (#51): every config peaks before the warmup LR peak, " + "collapses at it, and recovers as the LR decays", + fontsize=11.5, y=1.0, + ) + fig.tight_layout() + fig.savefig(out, dpi=200, bbox_inches="tight") + plt.close(fig) + print(f" wrote {out.name}") + + +def fig_precision_recall(runs, out: Path) -> None: + fig, axes = plt.subplots(1, 2, figsize=(13, 4.8)) + + for metric, ax, label in ((PRECISION, axes[0], "val precision"), + (RECALL, axes[1], "val recall")): + for name, lbl, colour, ls in CONFIGS: + if name not in runs: + continue + df = runs[name] + ax.plot(df["epoch"], df[metric], color=colour, linestyle=ls, + marker="o", markersize=3.5, linewidth=1.8, label=lbl) + _style(ax, ylabel=label, title=label + " vs epoch") + ax.set_ylim(-0.03, 1.05) + + axes[0].legend(fontsize=8, loc="lower left", framealpha=0.92) + # Precision reads 0 where a model emits *no* boxes at all (y11x_pano from epoch 3), + # which is the 0/0 convention, not a flood of false positives. y11l_pano is the + # legible case: precision holds at 0.94-1.00 while recall sits near 0.01. + fig.suptitle( + "The failure signature is a RECALL collapse: the model stops firing rather than " + "starting to guess\n(precision holds high, or reads 0 where nothing is emitted at all)", + fontsize=11.5, y=1.02, + ) + fig.tight_layout() + fig.savefig(out, dpi=200, bbox_inches="tight") + plt.close(fig) + print(f" wrote {out.name}") + + +def fig_loss_divergence(runs, out: Path) -> None: + """Train vs val loss per config -- the generalization-gap view. + + YOLO26 logs an l1_loss where YOLO11 logs dfl_loss, so the third component is + resolved per config rather than assumed. + """ + names = [n for n, *_ in CONFIGS if n in runs] + fig, axes = plt.subplots(2, len(names), figsize=(3.3 * len(names), 6.6), squeeze=False) + + for col, name in enumerate(names): + df = runs[name] + lbl = next(c[1] for c in CONFIGS if c[0] == name) + for row, comp in enumerate(("box_loss", "cls_loss")): + ax = axes[row][col] + ax.plot(df["epoch"], df[f"train/{comp}"], color="#2a9d8f", linewidth=1.8, + marker="o", markersize=3, label="train") + ax.plot(df["epoch"], df[f"val/{comp}"], color="#c1121f", linewidth=1.8, + linestyle="--", marker="s", markersize=3, label="val") + peak = warmup_peak_epoch(df) + if peak is not None: + ax.axvline(peak, color="#888888", linewidth=1.0, alpha=0.5) + _style(ax, ylabel=comp if col == 0 else None, + title=lbl.split("(")[0].strip() if row == 0 else None) + if row == 0 and col == 0: + ax.legend(fontsize=8, loc="upper left") + + fig.suptitle( + "Train vs validation loss: training loss keeps falling while validation " + "cls_loss explodes -- an optimization instability, not a crash or a data fault", + fontsize=11.5, y=1.0, + ) + fig.tight_layout() + fig.savefig(out, dpi=200, bbox_inches="tight") + plt.close(fig) + print(f" wrote {out.name}") + + +def fig_per_config(runs, out: Path) -> None: + """One panel per config: mAP50 + LR + ckpt preemption markers.""" + names = [n for n, *_ in CONFIGS if n in runs] + ncol = min(3, len(names)) + nrow = (len(names) + ncol - 1) // ncol + fig, axes = plt.subplots(nrow, ncol, figsize=(4.6 * ncol, 3.6 * nrow), squeeze=False) + + for i, name in enumerate(names): + ax = axes[i // ncol][i % ncol] + df = runs[name] + lbl, colour, ls = next((c[1], c[2], c[3]) for c in CONFIGS if c[0] == name) + + ax.plot(df["epoch"], df[MAP50], color=colour, linestyle=ls, marker="o", + markersize=4, linewidth=2, label="val mAP@50", zorder=3) + + twin = ax.twinx() + twin.plot(df["epoch"], df[LR], color="#666666", linestyle=":", linewidth=1.4, zorder=1) + twin.set_ylabel("lr", color="#666666", fontsize=9) + twin.tick_params(axis="y", colors="#666666", labelsize=8) + twin.spines[["top"]].set_visible(False) + + for j, ep in enumerate(resume_epochs(df)): + ax.axvline(ep, color="#7209b7", linewidth=1.2, alpha=0.55, linestyle="-.", + zorder=2, label="ckpt resume" if j == 0 else None) + peak = warmup_peak_epoch(df) + if peak is not None: + ax.axvline(peak, color="#888888", linewidth=1.2, alpha=0.6, zorder=2) + + best = df.loc[df[MAP50].idxmax()] + ax.annotate(f"best ep{int(best['epoch'])}: {best[MAP50]:.3f}", + xy=(best["epoch"], best[MAP50]), xytext=(4, -12), + textcoords="offset points", fontsize=8, color="#333333") + + _style(ax, ylabel="val mAP@50", title=lbl) + ax.set_ylim(-0.03, 0.9) + ax.legend(fontsize=7.5, loc="upper right", framealpha=0.9) + + for k in range(len(names), nrow * ncol): + axes[k // ncol][k % ncol].axis("off") + + fig.suptitle( + "Per-config detail. Grey line = warmup LR peak, purple = ckpt preemption/resume. " + "Collapse tracks the LR peak, not the resumes.", + fontsize=11.5, y=1.0, + ) + fig.tight_layout() + fig.savefig(out, dpi=200, bbox_inches="tight") + plt.close(fig) + print(f" wrote {out.name}") + + +def print_summary(runs) -> None: + print("\nPer-config summary (internal YOLO val-split proxy, NOT benchmark eval)\n") + hdr = f"{'config':<12} {'eps':>4} {'best ep':>8} {'best mAP50':>11} {'last mAP50':>11} {'last P':>7} {'last R':>7} {'resumes':>8}" + print(hdr) + print("-" * len(hdr)) + for name, *_ in CONFIGS: + if name not in runs: + print(f"{name:<12} {'--':>4} (dropped before epoch 1 -- args.yaml only)") + continue + df = runs[name] + best = df.loc[df[MAP50].idxmax()] + last = df.iloc[-1] + print(f"{name:<12} {len(df):>4} {int(best['epoch']):>8} {best[MAP50]:>11.3f} " + f"{last[MAP50]:>11.3f} {last[PRECISION]:>7.3f} {last[RECALL]:>7.3f} " + f"{len(resume_epochs(df)):>8}") + print() + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--runs-dir", type=Path, default=HERE / "runs", + help="directory of /results.csv (default: ./runs)") + ap.add_argument("--out-dir", type=Path, default=HERE / "figures", + help="where to write the PNGs (default: ./figures)") + args = ap.parse_args() + + print(f"reading {args.runs_dir}") + runs = load_runs(args.runs_dir) + if not runs: + print("no results.csv found -- nothing to plot", file=sys.stderr) + return 1 + + args.out_dir.mkdir(parents=True, exist_ok=True) + print(f"writing {args.out_dir}") + fig_learning_curves(runs, args.out_dir / "fig1_learning_curves.png") + fig_precision_recall(runs, args.out_dir / "fig2_precision_recall.png") + fig_loss_divergence(runs, args.out_dir / "fig3_loss_divergence.png") + fig_per_config(runs, args.out_dir / "fig4_per_config.png") + print_summary(runs) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/model_comparison/yolo_baseline/runs/y11l_pano/args.yaml b/scripts/model_comparison/yolo_baseline/runs/y11l_pano/args.yaml new file mode 100644 index 0000000..87ae55d --- /dev/null +++ b/scripts/model_comparison/yolo_baseline/runs/y11l_pano/args.yaml @@ -0,0 +1,116 @@ +task: detect +mode: train +model: /gscratch/scrubbed/jfroehli/yolo_runs/y11l_pano/weights/last.pt +data: /gscratch/scrubbed/jfroehli/yolo/pano/data.yaml +epochs: 60 +time: null +patience: 20 +batch: 4 +imgsz: 1280 +save: true +save_period: -1 +cache: false +device: '0' +workers: 8 +project: /gscratch/scrubbed/jfroehli/yolo_runs +name: y11l_pano +exist_ok: true +pretrained: true +cls_remap: true +optimizer: auto +verbose: true +seed: 0 +deterministic: true +single_cls: false +rect: false +cos_lr: false +close_mosaic: 10 +resume: /gscratch/scrubbed/jfroehli/yolo_runs/y11l_pano/weights/last.pt +amp: true +fraction: 1.0 +profile: false +freeze: null +multi_scale: 0.0 +compile: false +channels_last: false +overlap_mask: true +mask_ratio: 4 +dropout: 0.0 +val: true +split: val +save_json: false +conf: null +iou: 0.7 +max_det: 300 +quantize: null +dnn: false +plots: true +end2end: null +source: null +vid_stride: 1 +stream_buffer: false +visualize: false +augment: false +agnostic_nms: false +classes: null +retina_masks: false +embed: null +show: false +save_frames: false +save_txt: false +save_conf: false +save_crop: false +show_labels: true +show_conf: true +show_boxes: true +line_width: null +format: torchscript +keras: false +optimize: false +dynamic: false +simplify: true +opset: null +workspace: null +nms: false +lr0: 0.01 +lrf: 0.01 +momentum: 0.937 +weight_decay: 0.0005 +warmup_epochs: 3.0 +warmup_momentum: 0.8 +warmup_bias_lr: 0.0 +distill_model: null +dis: 6.0 +box: 7.5 +cls: 0.5 +cls_pw: 0.0 +dfl: 1.5 +pose: 12.0 +kobj: 1.0 +rle: 1.0 +angle: 1.0 +dlog: 1.0 +dgrad: 0.5 +dlam: 1.0 +nbs: 64 +hsv_h: 0.015 +hsv_s: 0.7 +hsv_v: 0.4 +degrees: 0.0 +translate: 0.1 +scale: 0.5 +shear: 0.0 +perspective: 0.0 +flipud: 0.0 +fliplr: 0.5 +bgr: 0.0 +mosaic: 1.0 +mixup: 0.0 +cutmix: 0.0 +copy_paste: 0.0 +copy_paste_mode: flip +auto_augment: randaugment +erasing: 0.4 +cfg: null +tracker: tracktrack.yaml +save_dir: /mmfs1/gscratch/scrubbed/jfroehli/yolo_runs/y11l_pano diff --git a/scripts/model_comparison/yolo_baseline/runs/y11l_pano/results.csv b/scripts/model_comparison/yolo_baseline/runs/y11l_pano/results.csv new file mode 100644 index 0000000..1c49fc7 --- /dev/null +++ b/scripts/model_comparison/yolo_baseline/runs/y11l_pano/results.csv @@ -0,0 +1,11 @@ +epoch,time,train/box_loss,train/cls_loss,train/dfl_loss,metrics/precision(B),metrics/recall(B),metrics/mAP50(B),metrics/mAP50-95(B),val/box_loss,val/cls_loss,val/dfl_loss,lr/pg0,lr/pg1,lr/pg2,lr/pg3,lr/pg4,lr/pg5,lr/pg6,lr/pg7 +1,7352.39,1.58582,1.33396,1.31376,0.87308,0.67332,0.77945,0.42263,1.39892,1.13264,1.26075,0.00999973,0.00333324,0.00999973,0.00333324,0.00999973,0.00333324,0.00999973,0.00333324 +2,14020.1,1.50486,1.23682,1.26008,0.92014,0.61629,0.7638,0.41958,1.37778,1.26102,1.26183,0.0196697,0.00655658,0.0196697,0.00655658,0.0196697,0.00655658,0.0196697,0.00655658 +3,21144.2,1.47727,1.23784,1.2464,0.82445,0.03805,0.0316,0.0164,2.49774,6.40598,2.03689,0.0290097,0.00966991,0.0290097,0.00966991,0.0290097,0.00966991,0.0290097,0.00966991 +4,7167.47,1.49289,1.26434,1.25522,0,0,0,0,3.29462,8.22706,2.57656,0.028515,0.009505,0.028515,0.009505,0.028515,0.009505,0.028515,0.009505 +5,8237.51,1.46674,1.24697,1.2402,0,0,0,0,3.22697,8.22256,2.53027,0.02802,0.00934,0.02802,0.00934,0.02802,0.00934,0.02802,0.00934 +6,11551.6,1.43966,1.23225,1.22371,1,6e-05,0.005,0.00368,2.98339,7.89658,2.3733,0.027525,0.009175,0.027525,0.009175,0.027525,0.009175,0.027525,0.009175 +7,8063.86,1.40326,1.21359,1.20978,0.944,0.02833,0.02429,0.01449,2.61371,6.89773,2.06625,0.02703,0.00901,0.02703,0.00901,0.02703,0.00901,0.02703,0.00901 +8,16122.1,1.26461,1.17284,1.15046,0.97138,0.01066,0.01471,0.00775,2.85682,7.28249,2.19974,0.026535,0.008845,0.026535,0.008845,0.026535,0.008845,0.026535,0.008845 +9,26165.8,1.21197,1.14843,1.1307,0.9728,0.00728,0.005,0.00379,2.98835,7.42767,2.26476,0.02604,0.00868,0.02604,0.00868,0.02604,0.00868,0.02604,0.00868 +10,9855.4,1.26067,1.16203,1.15104,0.75276,0.20365,0.1834,0.08434,2.42545,5.31002,1.88035,0.025545,0.008515,0.025545,0.008515,0.025545,0.008515,0.025545,0.008515 diff --git a/scripts/model_comparison/yolo_baseline/runs/y11l_tiles/args.yaml b/scripts/model_comparison/yolo_baseline/runs/y11l_tiles/args.yaml new file mode 100644 index 0000000..8c94545 --- /dev/null +++ b/scripts/model_comparison/yolo_baseline/runs/y11l_tiles/args.yaml @@ -0,0 +1,116 @@ +task: detect +mode: train +model: /gscratch/scrubbed/jfroehli/yolo_runs/y11l_tiles/weights/last.pt +data: /gscratch/scrubbed/jfroehli/yolo/tiles/data.yaml +epochs: 60 +time: null +patience: 20 +batch: 6 +imgsz: 1024 +save: true +save_period: -1 +cache: false +device: '0' +workers: 8 +project: /gscratch/scrubbed/jfroehli/yolo_runs +name: y11l_tiles +exist_ok: true +pretrained: true +cls_remap: true +optimizer: auto +verbose: true +seed: 0 +deterministic: true +single_cls: false +rect: false +cos_lr: false +close_mosaic: 10 +resume: /gscratch/scrubbed/jfroehli/yolo_runs/y11l_tiles/weights/last.pt +amp: true +fraction: 1.0 +profile: false +freeze: null +multi_scale: 0.0 +compile: false +channels_last: false +overlap_mask: true +mask_ratio: 4 +dropout: 0.0 +val: true +split: val +save_json: false +conf: null +iou: 0.7 +max_det: 300 +quantize: null +dnn: false +plots: true +end2end: null +source: null +vid_stride: 1 +stream_buffer: false +visualize: false +augment: false +agnostic_nms: false +classes: null +retina_masks: false +embed: null +show: false +save_frames: false +save_txt: false +save_conf: false +save_crop: false +show_labels: true +show_conf: true +show_boxes: true +line_width: null +format: torchscript +keras: false +optimize: false +dynamic: false +simplify: true +opset: null +workspace: null +nms: false +lr0: 0.01 +lrf: 0.01 +momentum: 0.937 +weight_decay: 0.0005 +warmup_epochs: 3.0 +warmup_momentum: 0.8 +warmup_bias_lr: 0.0 +distill_model: null +dis: 6.0 +box: 7.5 +cls: 0.5 +cls_pw: 0.0 +dfl: 1.5 +pose: 12.0 +kobj: 1.0 +rle: 1.0 +angle: 1.0 +dlog: 1.0 +dgrad: 0.5 +dlam: 1.0 +nbs: 64 +hsv_h: 0.015 +hsv_s: 0.7 +hsv_v: 0.4 +degrees: 0.0 +translate: 0.1 +scale: 0.5 +shear: 0.0 +perspective: 0.0 +flipud: 0.0 +fliplr: 0.5 +bgr: 0.0 +mosaic: 1.0 +mixup: 0.0 +cutmix: 0.0 +copy_paste: 0.0 +copy_paste_mode: flip +auto_augment: randaugment +erasing: 0.4 +cfg: null +tracker: tracktrack.yaml +save_dir: /mmfs1/gscratch/scrubbed/jfroehli/yolo_runs/y11l_tiles diff --git a/scripts/model_comparison/yolo_baseline/runs/y11l_tiles/results.csv b/scripts/model_comparison/yolo_baseline/runs/y11l_tiles/results.csv new file mode 100644 index 0000000..4a4a2b9 --- /dev/null +++ b/scripts/model_comparison/yolo_baseline/runs/y11l_tiles/results.csv @@ -0,0 +1,4 @@ +epoch,time,train/box_loss,train/cls_loss,train/dfl_loss,metrics/precision(B),metrics/recall(B),metrics/mAP50(B),metrics/mAP50-95(B),val/box_loss,val/cls_loss,val/dfl_loss,lr/pg0,lr/pg1,lr/pg2,lr/pg3,lr/pg4,lr/pg5,lr/pg6,lr/pg7 +1,18691.1,1.42439,1.78812,1.5495,0.87911,0.46461,0.65494,0.34407,1.29443,2.15475,1.60019,0.00999989,0.0033333,0.00999989,0.0033333,0.00999989,0.0033333,0.00999989,0.0033333 +2,18270.2,1.23796,1.67682,1.38557,0.15692,0.59146,0.30866,0.10046,2.10103,3.69541,2.39119,0.0196699,0.00655663,0.0196699,0.00655663,0.0196699,0.00655663,0.0196699,0.00655663 +3,26482.8,1.20646,1.67449,1.36324,0.26701,0.08767,0.04223,0.00424,3.13441,5.81774,3.60727,0.0290099,0.00966997,0.0290099,0.00966997,0.0290099,0.00966997,0.0290099,0.00966997 diff --git a/scripts/model_comparison/yolo_baseline/runs/y11x_pano/args.yaml b/scripts/model_comparison/yolo_baseline/runs/y11x_pano/args.yaml new file mode 100644 index 0000000..861695f --- /dev/null +++ b/scripts/model_comparison/yolo_baseline/runs/y11x_pano/args.yaml @@ -0,0 +1,116 @@ +task: detect +mode: train +model: /gscratch/scrubbed/jfroehli/yolo_runs/y11x_pano/weights/last.pt +data: /gscratch/scrubbed/jfroehli/yolo/pano/data.yaml +epochs: 60 +time: null +patience: 20 +batch: 2 +imgsz: 1280 +save: true +save_period: -1 +cache: false +device: '0' +workers: 8 +project: /gscratch/scrubbed/jfroehli/yolo_runs +name: y11x_pano +exist_ok: true +pretrained: true +cls_remap: true +optimizer: auto +verbose: true +seed: 0 +deterministic: true +single_cls: false +rect: false +cos_lr: false +close_mosaic: 10 +resume: /gscratch/scrubbed/jfroehli/yolo_runs/y11x_pano/weights/last.pt +amp: true +fraction: 1.0 +profile: false +freeze: null +multi_scale: 0.0 +compile: false +channels_last: false +overlap_mask: true +mask_ratio: 4 +dropout: 0.0 +val: true +split: val +save_json: false +conf: null +iou: 0.7 +max_det: 300 +quantize: null +dnn: false +plots: true +end2end: null +source: null +vid_stride: 1 +stream_buffer: false +visualize: false +augment: false +agnostic_nms: false +classes: null +retina_masks: false +embed: null +show: false +save_frames: false +save_txt: false +save_conf: false +save_crop: false +show_labels: true +show_conf: true +show_boxes: true +line_width: null +format: torchscript +keras: false +optimize: false +dynamic: false +simplify: true +opset: null +workspace: null +nms: false +lr0: 0.01 +lrf: 0.01 +momentum: 0.937 +weight_decay: 0.0005 +warmup_epochs: 3.0 +warmup_momentum: 0.8 +warmup_bias_lr: 0.0 +distill_model: null +dis: 6.0 +box: 7.5 +cls: 0.5 +cls_pw: 0.0 +dfl: 1.5 +pose: 12.0 +kobj: 1.0 +rle: 1.0 +angle: 1.0 +dlog: 1.0 +dgrad: 0.5 +dlam: 1.0 +nbs: 64 +hsv_h: 0.015 +hsv_s: 0.7 +hsv_v: 0.4 +degrees: 0.0 +translate: 0.1 +scale: 0.5 +shear: 0.0 +perspective: 0.0 +flipud: 0.0 +fliplr: 0.5 +bgr: 0.0 +mosaic: 1.0 +mixup: 0.0 +cutmix: 0.0 +copy_paste: 0.0 +copy_paste_mode: flip +auto_augment: randaugment +erasing: 0.4 +cfg: null +tracker: tracktrack.yaml +save_dir: /mmfs1/gscratch/scrubbed/jfroehli/yolo_runs/y11x_pano diff --git a/scripts/model_comparison/yolo_baseline/runs/y11x_pano/results.csv b/scripts/model_comparison/yolo_baseline/runs/y11x_pano/results.csv new file mode 100644 index 0000000..d3be6ee --- /dev/null +++ b/scripts/model_comparison/yolo_baseline/runs/y11x_pano/results.csv @@ -0,0 +1,8 @@ +epoch,time,train/box_loss,train/cls_loss,train/dfl_loss,metrics/precision(B),metrics/recall(B),metrics/mAP50(B),metrics/mAP50-95(B),val/box_loss,val/cls_loss,val/dfl_loss,lr/pg0,lr/pg1,lr/pg2,lr/pg3,lr/pg4,lr/pg5,lr/pg6,lr/pg7 +1,12202.8,1.58839,1.40362,1.32786,0.89616,0.65338,0.77746,0.42107,1.40861,1.21531,1.26997,0.00999987,0.00333329,0.00999987,0.00333329,0.00999987,0.00333329,0.00999987,0.00333329 +2,23829.7,1.51142,1.29776,1.27756,0.91411,0.31562,0.5522,0.2944,1.5679,2.31553,1.49041,0.0196699,0.00655662,0.0196699,0.00655662,0.0196699,0.00655662,0.0196699,0.00655662 +3,12495.7,1.54439,1.35191,1.29484,0,0,0,0,3.35169,7.97593,2.71784,0.0290099,0.00966996,0.0290099,0.00966996,0.0290099,0.00966996,0.0290099,0.00966996 +4,10320.1,1.54659,1.3661,1.29763,0,0,0,0,4.12017,7.12724,2.49091,0.028515,0.009505,0.028515,0.009505,0.028515,0.009505,0.028515,0.009505 +5,12549.5,1.52107,1.3395,1.28476,0,0,0,0,3.99518,7.35629,2.42091,0.02802,0.00934,0.02802,0.00934,0.02802,0.00934,0.02802,0.00934 +6,12306.2,1.50048,1.31663,1.2736,0,0,0,0,3.88304,7.61658,2.40362,0.027525,0.009175,0.027525,0.009175,0.027525,0.009175,0.027525,0.009175 +7,13165.8,1.47356,1.29413,1.26027,0,0,0,0,3.51034,7.88276,2.40115,0.02703,0.00901,0.02703,0.00901,0.02703,0.00901,0.02703,0.00901 diff --git a/scripts/model_comparison/yolo_baseline/runs/y11x_tiles/args.yaml b/scripts/model_comparison/yolo_baseline/runs/y11x_tiles/args.yaml new file mode 100644 index 0000000..d1320af --- /dev/null +++ b/scripts/model_comparison/yolo_baseline/runs/y11x_tiles/args.yaml @@ -0,0 +1,116 @@ +task: detect +mode: train +model: /mmfs1/home/jfroehli/RampNet/yolo11x.pt +data: /gscratch/scrubbed/jfroehli/yolo/tiles/data.yaml +epochs: 60 +time: null +patience: 20 +batch: 12 +imgsz: 1024 +save: true +save_period: -1 +cache: false +device: '0' +workers: 8 +project: /gscratch/scrubbed/jfroehli/yolo_runs +name: y11x_tiles +exist_ok: true +pretrained: true +cls_remap: true +optimizer: auto +verbose: true +seed: 0 +deterministic: true +single_cls: false +rect: false +cos_lr: false +close_mosaic: 10 +resume: false +amp: true +fraction: 1.0 +profile: false +freeze: null +multi_scale: 0.0 +compile: false +channels_last: false +overlap_mask: true +mask_ratio: 4 +dropout: 0.0 +val: true +split: val +save_json: false +conf: null +iou: 0.7 +max_det: 300 +quantize: null +dnn: false +plots: true +end2end: null +source: null +vid_stride: 1 +stream_buffer: false +visualize: false +augment: false +agnostic_nms: false +classes: null +retina_masks: false +embed: null +show: false +save_frames: false +save_txt: false +save_conf: false +save_crop: false +show_labels: true +show_conf: true +show_boxes: true +line_width: null +format: torchscript +keras: false +optimize: false +dynamic: false +simplify: true +opset: null +workspace: null +nms: false +lr0: 0.01 +lrf: 0.01 +momentum: 0.937 +weight_decay: 0.0005 +warmup_epochs: 3.0 +warmup_momentum: 0.8 +warmup_bias_lr: 0.1 +distill_model: null +dis: 6.0 +box: 7.5 +cls: 0.5 +cls_pw: 0.0 +dfl: 1.5 +pose: 12.0 +kobj: 1.0 +rle: 1.0 +angle: 1.0 +dlog: 1.0 +dgrad: 0.5 +dlam: 1.0 +nbs: 64 +hsv_h: 0.015 +hsv_s: 0.7 +hsv_v: 0.4 +degrees: 0.0 +translate: 0.1 +scale: 0.5 +shear: 0.0 +perspective: 0.0 +flipud: 0.0 +fliplr: 0.5 +bgr: 0.0 +mosaic: 1.0 +mixup: 0.0 +cutmix: 0.0 +copy_paste: 0.0 +copy_paste_mode: flip +auto_augment: randaugment +erasing: 0.4 +cfg: null +tracker: tracktrack.yaml +save_dir: /mmfs1/gscratch/scrubbed/jfroehli/yolo_runs/y11x_tiles diff --git a/scripts/model_comparison/yolo_baseline/runs/y26_pano/args.yaml b/scripts/model_comparison/yolo_baseline/runs/y26_pano/args.yaml new file mode 100644 index 0000000..c7e6545 --- /dev/null +++ b/scripts/model_comparison/yolo_baseline/runs/y26_pano/args.yaml @@ -0,0 +1,116 @@ +task: detect +mode: train +model: /gscratch/scrubbed/jfroehli/yolo_runs/y26_pano/weights/last.pt +data: /gscratch/scrubbed/jfroehli/yolo/pano/data.yaml +epochs: 60 +time: null +patience: 20 +batch: 4 +imgsz: 1280 +save: true +save_period: -1 +cache: false +device: '0' +workers: 8 +project: /gscratch/scrubbed/jfroehli/yolo_runs +name: y26_pano +exist_ok: true +pretrained: true +cls_remap: true +optimizer: auto +verbose: true +seed: 0 +deterministic: true +single_cls: false +rect: false +cos_lr: false +close_mosaic: 10 +resume: /gscratch/scrubbed/jfroehli/yolo_runs/y26_pano/weights/last.pt +amp: true +fraction: 1.0 +profile: false +freeze: null +multi_scale: 0.0 +compile: false +channels_last: false +overlap_mask: true +mask_ratio: 4 +dropout: 0.0 +val: true +split: val +save_json: false +conf: null +iou: 0.7 +max_det: 300 +quantize: null +dnn: false +plots: true +end2end: null +source: null +vid_stride: 1 +stream_buffer: false +visualize: false +augment: false +agnostic_nms: false +classes: null +retina_masks: false +embed: null +show: false +save_frames: false +save_txt: false +save_conf: false +save_crop: false +show_labels: true +show_conf: true +show_boxes: true +line_width: null +format: torchscript +keras: false +optimize: false +dynamic: false +simplify: true +opset: null +workspace: null +nms: false +lr0: 0.01 +lrf: 0.01 +momentum: 0.937 +weight_decay: 0.0005 +warmup_epochs: 3.0 +warmup_momentum: 0.8 +warmup_bias_lr: 0.0 +distill_model: null +dis: 6.0 +box: 7.5 +cls: 0.5 +cls_pw: 0.0 +dfl: 1.5 +pose: 12.0 +kobj: 1.0 +rle: 1.0 +angle: 1.0 +dlog: 1.0 +dgrad: 0.5 +dlam: 1.0 +nbs: 64 +hsv_h: 0.015 +hsv_s: 0.7 +hsv_v: 0.4 +degrees: 0.0 +translate: 0.1 +scale: 0.5 +shear: 0.0 +perspective: 0.0 +flipud: 0.0 +fliplr: 0.5 +bgr: 0.0 +mosaic: 1.0 +mixup: 0.0 +cutmix: 0.0 +copy_paste: 0.0 +copy_paste_mode: flip +auto_augment: randaugment +erasing: 0.4 +cfg: null +tracker: tracktrack.yaml +save_dir: /mmfs1/gscratch/scrubbed/jfroehli/yolo_runs/y26_pano diff --git a/scripts/model_comparison/yolo_baseline/runs/y26_pano/results.csv b/scripts/model_comparison/yolo_baseline/runs/y26_pano/results.csv new file mode 100644 index 0000000..c8bf40a --- /dev/null +++ b/scripts/model_comparison/yolo_baseline/runs/y26_pano/results.csv @@ -0,0 +1,15 @@ +epoch,time,train/box_loss,train/cls_loss,train/l1_loss,metrics/precision(B),metrics/recall(B),metrics/mAP50(B),metrics/mAP50-95(B),val/box_loss,val/cls_loss,val/l1_loss,lr/pg0,lr/pg1,lr/pg2,lr/pg3,lr/pg4,lr/pg5,lr/pg6,lr/pg7 +1,8426.62,1.61195,1.77719,0.00269,0.81174,0.61329,0.73792,0.41315,1.36326,1.36852,0.00338,0.00999973,0.00333324,0.00999973,0.00333324,0.00999973,0.00333324,0.00999973,0.00333324 +2,16395.9,1.46926,1.51431,0.00235,0.82394,0.57935,0.72156,0.40969,1.32392,1.45054,0.00327,0.0196697,0.00655658,0.0196697,0.00655658,0.0196697,0.00655658,0.0196697,0.00655658 +3,24412.5,1.46689,1.53096,0.00235,0.85235,0.48441,0.67078,0.37666,1.35046,1.71333,0.00335,0.0290097,0.00966991,0.0290097,0.00966991,0.0290097,0.00966991,0.0290097,0.00966991 +4,7273.01,1.49076,1.56422,0.00239,0.82983,0.40175,0.59898,0.33015,1.39821,1.93435,0.00349,0.028515,0.009505,0.028515,0.009505,0.028515,0.009505,0.028515,0.009505 +5,14507,1.41274,1.48337,0.00225,0.61098,0.2659,0.3472,0.17936,1.64631,3.1187,0.00428,0.02802,0.00934,0.02802,0.00934,0.02802,0.00934,0.02802,0.00934 +6,21562.6,1.3595,1.42764,0.00216,0.21421,0.25785,0.12504,0.06099,1.92558,4.96187,0.00547,0.027525,0.009175,0.027525,0.009175,0.027525,0.009175,0.027525,0.009175 +7,8725.72,1.40228,1.46717,0.00224,0.6522,0.31318,0.40801,0.22175,1.57462,2.687,0.00404,0.02703,0.00901,0.02703,0.00901,0.02703,0.00901,0.02703,0.00901 +8,17262.2,1.31481,1.41433,0.00207,0.6173,0.41195,0.49115,0.27385,1.47443,2.14678,0.00371,0.026535,0.008845,0.026535,0.008845,0.026535,0.008845,0.026535,0.008845 +9,12033.1,1.3292,1.42895,0.0021,0.63274,0.48292,0.56741,0.33055,1.32759,1.73712,0.00327,0.02604,0.00868,0.02604,0.00868,0.02604,0.00868,0.02604,0.00868 +10,10365,1.31301,1.41394,0.00206,0.6415,0.52115,0.60416,0.36099,1.25303,1.59772,0.00306,0.025545,0.008515,0.025545,0.008515,0.025545,0.008515,0.025545,0.008515 +11,19393.6,1.2671,1.39006,0.00196,0.64587,0.53745,0.62133,0.37519,1.2237,1.54846,0.00297,0.02505,0.00835,0.02505,0.00835,0.02505,0.00835,0.02505,0.00835 +12,27905.8,1.2382,1.37647,0.00191,0.6407,0.55002,0.62359,0.3795,1.19926,1.53745,0.00288,0.024555,0.008185,0.024555,0.008185,0.024555,0.008185,0.024555,0.008185 +13,10029.7,1.25905,1.37167,0.00196,0.65355,0.5757,0.64884,0.3978,1.17371,1.48241,0.00279,0.02406,0.00802,0.02406,0.00802,0.02406,0.00802,0.02406,0.00802 +14,19929.1,1.23581,1.3696,0.0019,0.65777,0.59101,0.65936,0.40711,1.15456,1.44529,0.00272,0.023565,0.007855,0.023565,0.007855,0.023565,0.007855,0.023565,0.007855 diff --git a/scripts/model_comparison/yolo_baseline/runs/y26_tiles/args.yaml b/scripts/model_comparison/yolo_baseline/runs/y26_tiles/args.yaml new file mode 100644 index 0000000..8461e64 --- /dev/null +++ b/scripts/model_comparison/yolo_baseline/runs/y26_tiles/args.yaml @@ -0,0 +1,116 @@ +task: detect +mode: train +model: /gscratch/scrubbed/jfroehli/yolo_runs/y26_tiles/weights/last.pt +data: /gscratch/scrubbed/jfroehli/yolo/tiles/data.yaml +epochs: 60 +time: null +patience: 20 +batch: 6 +imgsz: 1024 +save: true +save_period: -1 +cache: false +device: '0' +workers: 8 +project: /gscratch/scrubbed/jfroehli/yolo_runs +name: y26_tiles +exist_ok: true +pretrained: true +cls_remap: true +optimizer: auto +verbose: true +seed: 0 +deterministic: true +single_cls: false +rect: false +cos_lr: false +close_mosaic: 10 +resume: /gscratch/scrubbed/jfroehli/yolo_runs/y26_tiles/weights/last.pt +amp: true +fraction: 1.0 +profile: false +freeze: null +multi_scale: 0.0 +compile: false +channels_last: false +overlap_mask: true +mask_ratio: 4 +dropout: 0.0 +val: true +split: val +save_json: false +conf: null +iou: 0.7 +max_det: 300 +quantize: null +dnn: false +plots: true +end2end: null +source: null +vid_stride: 1 +stream_buffer: false +visualize: false +augment: false +agnostic_nms: false +classes: null +retina_masks: false +embed: null +show: false +save_frames: false +save_txt: false +save_conf: false +save_crop: false +show_labels: true +show_conf: true +show_boxes: true +line_width: null +format: torchscript +keras: false +optimize: false +dynamic: false +simplify: true +opset: null +workspace: null +nms: false +lr0: 0.01 +lrf: 0.01 +momentum: 0.937 +weight_decay: 0.0005 +warmup_epochs: 3.0 +warmup_momentum: 0.8 +warmup_bias_lr: 0.0 +distill_model: null +dis: 6.0 +box: 7.5 +cls: 0.5 +cls_pw: 0.0 +dfl: 1.5 +pose: 12.0 +kobj: 1.0 +rle: 1.0 +angle: 1.0 +dlog: 1.0 +dgrad: 0.5 +dlam: 1.0 +nbs: 64 +hsv_h: 0.015 +hsv_s: 0.7 +hsv_v: 0.4 +degrees: 0.0 +translate: 0.1 +scale: 0.5 +shear: 0.0 +perspective: 0.0 +flipud: 0.0 +fliplr: 0.5 +bgr: 0.0 +mosaic: 1.0 +mixup: 0.0 +cutmix: 0.0 +copy_paste: 0.0 +copy_paste_mode: flip +auto_augment: randaugment +erasing: 0.4 +cfg: null +tracker: tracktrack.yaml +save_dir: /mmfs1/gscratch/scrubbed/jfroehli/yolo_runs/y26_tiles diff --git a/scripts/model_comparison/yolo_baseline/runs/y26_tiles/results.csv b/scripts/model_comparison/yolo_baseline/runs/y26_tiles/results.csv new file mode 100644 index 0000000..d5c093a --- /dev/null +++ b/scripts/model_comparison/yolo_baseline/runs/y26_tiles/results.csv @@ -0,0 +1,4 @@ +epoch,time,train/box_loss,train/cls_loss,train/l1_loss,metrics/precision(B),metrics/recall(B),metrics/mAP50(B),metrics/mAP50-95(B),val/box_loss,val/cls_loss,val/l1_loss,lr/pg0,lr/pg1,lr/pg2,lr/pg3,lr/pg4,lr/pg5,lr/pg6,lr/pg7 +1,20051.5,1.42301,2.03216,0.00647,0.66595,0.55011,0.64742,0.35579,1.20557,1.79447,0.00631,0.00999989,0.0033333,0.00999989,0.0033333,0.00999989,0.0033333,0.00999989,0.0033333 +2,19265.3,1.2233,1.72429,0.00533,0.60321,0.53767,0.60144,0.35348,1.11243,1.81904,0.0057,0.0196699,0.00655663,0.0196699,0.00655663,0.0196699,0.00655663,0.0196699,0.00655663 +3,27505.8,1.20769,1.74667,0.00523,0.51781,0.18848,0.27961,0.11929,2.03582,2.99481,0.01242,0.0290099,0.00966997,0.0290099,0.00966997,0.0290099,0.00966997,0.0290099,0.00966997