Add Gainesville, FL — third GSV city (split #9)

[jonfroehlich]

Phase 0 — what this split is for

Ninth split, third GSV city — and the first far-domain clean GSV city: no Stage-2 training city is anywhere near Florida, and Gainesville is registry-clean per docs/data_provenance.md (grep-verified 2026-07-29). With conventional street fabric (unlike Paterson's paired-TSI corners), it tests whether Paterson's recall profile (R 0.650, ceiling 0.754) is fabric-specific or generic to out-of-domain GSV. Bonus context: Project Sidewalk deployment is fresh and complete (40 regions, 39/40 at 100%, 13.9k labels, Jul 2026); the sampled boundary is the dissolved PS regions polygon (~47.6 km² deployment footprint, not the whole ~160 km² municipality).

• Pooled recommendation: yes — US city, GSV imagery, verdict-grade GT (reviewer confidence HIGH); now in US_SPLITS.
• Imagery tier: gsv via tier_of() (records.jsonl carries source: "launch"); no fallback or new tier branch needed.

Results so far

GT (Phase 3): unbiased P 0.943 (CI 0.896–0.970) / R 0.647 (CI 0.583–0.705); all-panos 0.945/0.695; reviewer confidence HIGH (a25e65f).

Answer to the Phase-0 question (Phase 4, f024570): the recall number replicates paterson's (0.647 vs 0.650) — far-domain GSV, not fabric — but the mechanism differs: gainesville's ceiling is 0.890 (normal US band) vs paterson's 0.757. Paterson's misses never fire; gainesville's fire sub-threshold, so 0.30 buys +9.9 R here vs +3.8 there. Details in the PR comments.

Checklist (docs/adding_a_benchmark_city.md)

Phase 1 — bundle (auto-labeler)

• City detected (main.py --source gsv)
• export_benchmark.py --bundle completed; index.csv written (verified post-hoc: 125/125 sha256 match, valid JPEGs); no decayed.txt (nothing decayed)
• benchmark/gainesville/records.jsonl committed; panos/ present locally and staged on klone gscratch (/gscratch/makelab/jfroehli/rampnet_benchmark_panos/gainesville); HF archive still Publish deployment validation ground-truth as a HuggingFace dataset (Bend GSV + Richmond Mapillary) #21 backlog
• Strata are 5 top / 95 random / 25 empty

Phase 2–3 — ground truth

• Reviewed with gt_gallery.py at model resolution (reviewer: jonf, 2026-07-30)
• review_notes written (confidence HIGH; novel ramp styles — large diagonal ramps at new/renovated intersections, wide intersections; FN causes by impression: debris, distance)
• verdicts.json committed (a25e65f)
• score_validation.py run; unbiased column: P 0.943 / R 0.647
• Camera provenance: no camera_make/camera_model, but source: "launch" → tier_of() = gsv (same as paterson)

Phase 4 — operating point

• Low-floor extraction run (job 37923168, ckpt-g2, 4m19s)
• low_floor_sweep.py parity PASSES — 95.4% within 0.5 R, max 0.439 R, count delta 3.9%
• sweep / hist / gtbias / floor / distance re-run (findings in PR comment)
• GT-completeness correction: spot-check the low-confidence incremental FPs from the operating-point curve #55 gallery tagged at op-threshold 0.25 — gallery built, 34 items, tagging pass in flight; incremental_fp_tags.json to follow
• corrected re-run + tagcheck (blocked on the tags)
• op_cache/gainesville.json and refreshed op/*.csv committed (f024570; corrected_at_*.csv refresh pending tags)
• Both figures regenerated and visually inspected — the 9th-series fold decision is documented in plot_operating_point.py (held-out splits → neutral ink + dashes; gainesville takes vacated slot-6 green; 7-hue set re-validated ALL PASS)

Phase 5 — code

• Added to US_SPLITS (⇒ CITY_SPLITS / ALL_SPLITS by construction)
• HELD_OUT n/a — gainesville is pooled
• tier_of() recognises the rig (gsv via source)
• SERIES handling decided and validated (see figures note above)
• pytest -q green (241 passed)

Phase 6 — docs

• benchmark/README.md: both tables + prose section
• docs/model_comparison.md: coverage matrix incl. GT-completeness correction: spot-check the low-confidence incremental FPs from the operating-point curve #55 A-rate + anchoring caveat list (challenger runs for gainesville: not yet run — will be stated explicitly)
• docs/operating_point.md: per-split, per-tier, pooled, corrected, storage-floor tables
• Pooled numbers and the recommendation re-checked (uncorrected pooled moved 0.848 → 0.843 F1 @0.30; corrected re-check pending tags)
• Anything deliberately not run stated explicitly, with the reason

🤖 Generated with Claude Code (claude-fable-5)

[jonfroehlich]

Phase 2–3 complete — GT verdicts committed (a25e65f), scored 2026-07-30.

	Precision	Recall
Unbiased (random+empty, n=120)	0.943 (CI 0.896–0.970)	0.647 (CI 0.583–0.705)
All panos (n=125)	0.945	0.695

Counts: 205 detections judged — 189 correct, 9 FP, 5 unsure (2.4% abstention), 2 duplicates (scored as FP, the default). 83 missed marks + 31 unsure-missed. Recall pool 125/125 (no pano lacked an attestation or a mark). Empty stratum 23/25 clean; the 2 exceptions hold 4 firm missed marks — real ramps the model was silent on, consistent with the annapolis pattern.

The Phase-0 question is answered early: paterson's recall was not fabric-specific. Gainesville has conventional fabric — none of paterson's paired TSIs — and lands at R 0.647 vs paterson's 0.650 (unbiased, same protocol). Two far-domain GSV cities now agree that out-of-domain GSV recall sits at ~0.65 at the 0.55 floor. Precision differs more (0.943 vs 0.987): gainesville produced 9 FPs + 2 duplicates against paterson's near-zero.

Reviewer (jonf, confidence HIGH) notes, committed in review_notes: ramp styles unlike prior splits — many wide intersections, and large diagonal curb ramps spanning both crossing directions at new/renovated intersections; false negatives by impression clustered on (1) significant debris on the ramp surface and (2) distance — both flagged for the Phase-4 distance analysis (and debris as a candidate qualitative-taxonomy bucket, issue #46).

Next: Phase-4 low-floor extraction on Hyak, then the #55 tagging pass.

🤖 Generated with Claude Code (claude-fable-5)

[jonfroehlich]

Phase-4 run status: low-floor extraction for gainesville submitted on Hyak as job 37923168 (partition ckpt-g2, account ckpt-makelab, 1 GPU — same shape as paterson's job 37895426, which took 5m21s for 125 panos). Panos staged to /gscratch/makelab/jfroehli/rampnet_benchmark_panos/gainesville (125 files, sha256-verified locally before upload); bundle metadata staged into the klone working copy. Expected output: analysis_out/op_cache/gainesville.json, after which the parity gate runs locally (GSV split: expect within 0.5 R, not bit-exact).

🤖 Generated with Claude Code (claude-fable-5)

[jonfroehlich]

Phase 4–5 swept (f024570). Extraction: Hyak job 37923168 (ckpt-g2, 4m19s). Parity gate PASS — 95.4% within 0.5 R, max 0.439 R (the familiar GSV-resample signature), count delta 3.9%.

The Phase-0 answer got sharper: same recall, different mechanism. Gainesville's recall ceiling at the 0.05 floor is 0.890 — squarely in the normal US band (0.86–0.91) — versus paterson's 0.757. So the two far-domain GSV cities arrive at the same ~0.65 deployed recall by different failure modes: paterson's misses produce no candidate at any threshold (never fires); gainesville's misses fire below threshold (weak activations — consistent with the reviewer's debris + distance impressions). Consequence: the 0.30 operating point buys gainesville +9.9 recall points (0.673 → 0.772), nearly 3× paterson's +3.8.

	ceiling R@0.05	@0.30	@0.55	F1-max
gainesville	0.890	P 0.857 / R 0.772	P 0.948 / R 0.673	0.822 @ 0.38
paterson	0.757	P 0.950 / R 0.719	P 0.971 / R 0.681	0.821 @ 0.26
POOLED (7 US, n=1,984)	0.871	P 0.900 / R 0.793	P 0.964 / R 0.722	0.844 @ 0.32

Other Phase-4 results: distance — far-field (>25 m) R@0.55 is 0.300, the worst far band of any US split (paterson 0.432); 0.55→0.30 recovers +12.2 mid / +12.0 far points. gtbias — canonical #55 signature present. floor — storage-floor cost 2.21%, recoverable headroom +0.195 (tied with clovis for largest). hist — sub-0.55 calibration noisy-low, lower bound until tagged. Pooled@0.30 moved 6-US → 7-US: P 0.907→0.900, R 0.796→0.793, F1 0.848→0.843 (uncorrected); recommendation re-check happens on the corrected re-run.

#55 gallery: 34 items in [0.25, 0.55) — above the 23–30 US precedent and 3× paterson's 10, consistent with gainesville's dense sub-floor band; 7 pre-flagged as likely duplicates. Awaiting the tagging pass at analysis_out/op/gainesville_incremental_fp/index.html, then corrected --op-threshold 0.30 + tagcheck + the corrected tables.

Figures: the 9th series forced the protocol's fold-or-facet decision. Resolution: budapest*/manual_gold (both held out of the recommendation) now wear neutral ink with distinct dash patterns — their visual weight now matches their scientific status — vacating validated slot 6 (green) for gainesville. The 7-hue set re-validates ALL PASS in slot order (validate_palette.js; the new green↔red adjacency ΔE 7.2 protan sits in the legal-with-secondary-encoding band, carried by legend + operating-point dots + the committed CSV). Both figures regenerated and visually inspected. pytest -q: 241 passed.

🤖 Generated with Claude Code (claude-fable-5)