Add Paterson, NJ — the benchmark's second GSV city (split #8) - #88
Merged
Conversation
Phase 1 deliverable for the paterson split, produced by sidewalk-auto-labeler (run log: ProjectSidewalk/sidewalk-auto-labeler#29, exporter run merged in ProjectSidewalk/sidewalk-auto-labeler#30). Phase-0 purpose, recorded verbatim: "second GSV city; out-of-training but metro-adjacent" -- NOT a geographic-transfer test. It exists to deconfound bend's GSV-vs-in-domain operating point (docs/operating_point.md). Clean per the contamination registry (docs/data_provenance.md): neither a Stage-2 open-data city nor a crop-model PS city; the NYC-metro adjacency (NYC Stage-2; Teaneck/Oradell/Cliffside Park crop cities next door) is the recorded caveat. Boundary: dissolved from the live PS deployment's regions API (https://sidewalk-paterson.cs.washington.edu/v3/api/regions, fetched 2026-07-29; 68 neighborhood polygons -> one ~22.5 km^2 polygon, essentially the whole municipality). Bundle facts (re-verified locally before this commit): - 125 records, strata exactly 5 top / 95 random / 25 empty, 125 unique pano ids; the empty group carries zero detections by construction. 284 detections total at the >=0.55 export floor, sampled from 34,427 whole-city source records (seed 0, min spacing 30 m). - Exporter exit 0; 125/125 panos fetched at native res (1.9 GB), 0 decayed, reconcile 1:1. panos/ is git-ignored here; the full-city native-res archive lives on makelab2 (#21 still lags the repo for durable HF archiving). - camera_make/camera_model absent in all 125 -- structural for GSV; Phase 5 needs a SPLIT_IMAGERY_FALLBACK entry, like bend. - Capture dates skew fresh (85/125 from 2024-2025, mirroring ~72% 2024-2026 city-wide), but two gen-1 3328x1664 panos (2007/2008) made the sample; representative -- review at model resolution like everything else. - Run-level operational detection rate 31.7%, the highest of any city; unquotable until GT lands (annapolis precedent, 27.96%). - Parity expectation: within 0.5 R, not bit-exact -- GSV production used the 4096x2048 intermediate, same as bend. Next: Phase 2 GT review (scripts/gt_gallery.py benchmark/paterson). Reviewer expectation: more degraded/legacy ramps than any existing split; use the review-notes field liberally (budapest precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nfidence HIGH Phase 2-3 of the paterson split (protocol: docs/adding_a_benchmark_city.md). Reviewed 2026-07-29 by jonf with scripts/gt_gallery.py at model resolution, all 125 panos fully judged. Headline (unbiased subset, random+empty, the between-city-comparable column): Precision 0.987 (95% CI 0.963-0.996) -- highest of any split Recall 0.650 (95% CI 0.599-0.698) All-panos: P 0.975 / R 0.684. Duplicates scored as FPs (default). Verdict counts: 270 correct / 6 FP / 1 duplicate / 7 unsure (2.5%, normal band); 161 missed-ramp marks (125 definite + 36 unsure-abstained). Recall pool covers all 125 panos (every pano attested no_missed or carries marks). Phase-3 gates, all pass: - Empty stratum: 24/25 attested clean, 1 missed mark total -- the strongest negative check of any split (annapolis: 10 marks across 4 panos). - Duplicates: 1 (normal 0-3; budapest's rubric ambiguity showed 7). - Recall-gate coverage: 125/125. The review_notes block (printed by score_validation.py before the numbers) records the split's two findings, both reviewer impressions to be tested by the Phase-4 distance analysis: - Paired tactile surface indicators: many corners carry two TSIs offset ~60-90 degrees, one per crossing; each marked as its own ramp. The model usually fired once per corner, so the partner lands as a miss -- a recall effect (only 1 duplicate), plausibly a major driver of the 161 marks. This corner style appears in no other split. - Distant ramps: substantial far-field misses, echoing annapolis. Reviewer-rated confidence HIGH: rubric fit, judgment calls minor, a second rater expected to land within the CIs. Unlike budapest (LOW, held out), nothing here argues against pooling; the pool/hold decision lands with the Phase-5 US_SPLITS change, not this commit. The run-level operational detection rate (31.7%, flagged unquotable at Phase 1) can now be read against GT: precision 0.987 says those detections were real, and recall 0.650 says the true ramp density is higher still. Next: Phase 4 -- low-floor extraction (parity gate: within 0.5 R, GSV intermediate like bend), sweep/hist/gtbias/floor/distance, then the #55 gallery for the second human pass at op-threshold 0.25. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extraction: klone job 37895426 (ckpt-g2, L40, 5m21s, qwenvl env python). The launcher's `source activate sidewalkcv2` fallback fails from a non-interactive submission shell -- pass PYTHON= explicitly; noted in scripts/analysis/README.md. Cache committed (analysis_out/op_cache/paterson.json). PARITY GATE: PASS. 96.8% of 284 committed detections reproduce within 0.5 R (gate >=95%), max displacement 0.439 R -- identical to bend's max, the expected GSV-intermediate signature. Count delta 3/284 (~1%, gate <=5%). paterson joins US_SPLITS -> the pooled recommendation now runs over six splits (n=734 panos). Pooled at the recommended 0.30: P 0.907 / R 0.796 / F1 0.848; F1-max at 0.32; the 0.55->0.30 move buys +6.3 pooled recall points for -5.2 precision. The 0.30 recommendation stands. paterson sweep (all-panos, deployment-faithful): 0.55: P 0.971 / R 0.681; 0.30: P 0.950 / R 0.719; F1-max at 0.26 (0.821). Its threshold response is the shallowest of any split: 0.55->0.30 buys only +3.8 recall points (pooled: +6.3). Three findings, recorded before the #55 correction (which will raise sub-0.55 precision -- gtbias shows the canonical signature: below the review floor 22 of 27 TPs come from missed marks): - The far-field impression is now a measurement: recall@0.55 by distance band is 0.794 near (<12.5 m) / 0.707 mid / 0.432 far (>25 m), and dropping to 0.30 gains most in the far band (+0.091). Echoes annapolis. - The recall CEILING is the real story: 0.754 at the 0.05 extraction floor, the lowest of any US split (others 0.86-0.91), with only +0.073 recoverable headroom above the deployed threshold and just 1 GT ramp lost to the 0.10 storage floor (others lose 6-9). paterson's misses are not under-confident detections waiting below the threshold -- they produce no candidate at all. Hypothesis worth testing (#62-adjacent): the reviewer-documented paired TSIs sit close enough that peak_local_max min_distance=10 may suppress the partner peak outright; a min_distance=3 re-extraction of this one split would separate "NMS-suppressed" from "never fired". - GSV tier now pools bend + paterson (n=235) -- the deconfounding this split was added for. Figure/code fixes that rode along: the pooled label ("5 US") and the footer pano count in both plot scripts are now computed, not hardcoded -- the committed "1,625 panos" was already stale before paterson (true pre-paterson count: 1,734; now 1,859). Both figure titles reworded to stay accurate with paterson in (its +3.8-point gain broke "~7 points on every split"; pooled discard share is 2.18%, was written as ~2.7%). paterson takes categorical slot 8 (#e34948) from the validated palette; the 8-slot sequence passes the palette validator (CVD worst adjacent pair 9.1, normal-vision floor 19.6). pytest -q: 241 passed. Remaining for the split: the #55 incremental-FP tagging pass (human), corrected tables, and the Phase-6 documentation sweep. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The #55 tagging pass (jonf, 2026-07-29): 10 incremental FPs in [0.25, 0.55) -- by far the fewest of any US city (others 23-30) -- tagged 2 A / 5 B / 3 unsure, A-rate 20.0%, inside the 13-30% band. tagcheck resolves 10/10. The low count is the split's shallow threshold response measured a second way, and the runbook now says so. Corrected results with paterson pooled (six US splits, n=734): op raw P corr P band-hi raw R corr R corr F1 0.25 0.884 0.903 0.914 0.806 0.810 0.854 0.30 0.907 0.924 0.934 0.796 0.799 0.857 <- still the optimum 0.35 0.922 0.935 0.941 0.782 0.784 0.853 The 0.30 recommendation survives its sixth split: corrected F1 still peaks at 0.30, corrected precision there (0.924) still exceeds raw precision at 0.35 (0.922). New headline trade: +6.9 corrected recall points for -4.2 precision at 1.91 -> 2.27 dets/pano (was +7.7/-4.8 on five splits). Documentation updated everywhere the runbook's Phase 6 names: - docs/operating_point.md: headline, parity table (+paterson, max 0.439 R = bend's), gtbias section (paterson footnote: 5 sub-0.5 TPs trace to reviewed detections via GSV resample jitter -- softens the anchoring cliff +0.345 -> +0.301 without touching its cause), A/B table, pooled tables with CIs, per-split rows, per-tier table, distance, density, corrected tables, storage floor (+11.6 pts headroom, cost now ~2.2%), recommendation, crossover bins. Flip-TTA section gets a scope note: its arm predates paterson and was deliberately not extended (#78 closed). - **The GSV-tier confound is resolved and documented**: with paterson, the GSV tier optimum moved 0.50 -> 0.26, into the cluster (0.26-0.35 across all five tiers). bend's outlier was in-domain-ness, not GSV. The "second GSV split" item in "What would change this" is struck through with the answer; a new item logs the paterson min_distance=3 re-check (#62-adjacent, untested). - benchmark/README.md: both split tables, the recall-vs-camera prose (paterson = sharpest counterexample), a full paterson prose section (purpose, boundary provenance, paired-TSI + far-field findings, 0.757 recall ceiling, negative check 24/25), and the #21 lag note now lists all five unpublished splits. - docs/model_comparison.md: coverage matrix gains the paterson row with its gaps stated -- challengers and null-recall NOT run; no ranking table includes paterson until they are. Anchoring caveat now quotes seven measured cities. - docs/adding_a_benchmark_city.md: PYTHON= submission requirement and the low-#55-count expectation, both learned this split. pytest -q: 241 passed. Remaining before this branch PRs: challenger + null-recall runs on paterson (or an explicit decision to defer them), and the HF benchmark dataset lag (#21) now five splits behind. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The three corrected_at_*.csv tables now include paterson and the six-split pooled rows; they are the committed source for docs/operating_point.md's corrected tables and re-derive with: python scripts/analysis/low_floor_sweep.py corrected --op-threshold <op> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The paterson coverage-matrix gaps are closed: 8-model challenger pass (klone jobs 37897737-40 for the five open-weight models, ~14-30 min each on ckpt-g2; the two Geminis locally via Vertex ADC) and the null-recall pass, all from the shared .model_cache (detections rsync'd back, 626 entries). Results (125 panos, 395 GT ramps, all-panos): rampnet P 0.975 R 0.686 F1 0.805 (AP 0.681) gemini-3.1-pro-preview P 0.852 R 0.567 F1 0.681 gemini-3.6-flash P 0.744 R 0.514 F1 0.608 molmo2-8B P 0.585 R 0.453 F1 0.511 Qwen3-VL-8B P 0.460 R 0.362 F1 0.405 Qwen3-VL-32B P 0.813 R 0.220 F1 0.347 owlv2 (0.05 floor) P 0.040 R 0.891 F1 0.077 (sweep-best 0.216) gdino (0.05 floor) P 0.036 R 0.800 F1 0.068 (sweep-best 0.100) Three findings, written into docs/model_comparison.md: 1. RampNet wins its seventh city, but by the NARROWEST lead measured anywhere: 0.124 F1 (prior range 0.19-0.34). Decomposition: RampNet's structural recall ceiling meets precision-friendly fresh GSV imagery (gemini-pro posts its best-anywhere precision 0.852) -- not a challenger closing the gap. Headline range updated to 0.12-0.34. 2. The Qwen inversion RECURS: 32B below 8B (0.347 vs 0.405), the same swap budapest showed, now on a HIGH-confidence US rubric. This removes "rubric disagreement" as a necessary condition -- 32B hyper-caution on unusual infrastructure is the mechanism with evidence on two splits. Budapest's second-rater ask stands for its magnitude, but the inversion itself no longer hangs on it. Budapest's section updated. 3. The misses are model-agnostic: complementarity (gemini-pro) recovers 36 of RampNet's 124 misses, but 88 ramps -- 22% of the split's GT -- are found by NO model tested; oracle-union recall 0.777 is the lowest in the benchmark. The structural population (paired TSIs, far-field) is invisible to everything, which is the challenger-side confirmation of the recall-ceiling finding. Fusion note: gemini's union lift here costs only 39 FPs (richmond: 127), so paterson is where a fusion experiment would look best if ever run. Null-recall reproduces the density story: owlv2 null 0.712 at 70.3 boxes/pano (within the established 0.56-0.77 band); rampnet null 0.067 at 2.3/pano, 0.664 above chance -- the most real signal in the roster. Also: scripts/analysis/complementarity.py now takes the split as argv[2] (was hardcoded to richmond); richmond default output verified unchanged. pytest -q: 241 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
Author
|
Challenger results are in (commit 0704244): all 8 models × 125 panos, plus null-recall and the #35 complementarity check.
Three takeaways, written into
Null-recall reproduces the density story: owlv2's 0.891 nominal recall carries a 0.712 null at 70 boxes/pano; RampNet's null is 0.067 at 2.3/pano. Run provenance: klone ckpt-g2 jobs 37897737–40 (open-weight models, 14–30 min each), Geminis local via Vertex ADC; detections merged into 🤖 Generated with Claude Code (claude-fable-5) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds paterson to the validation benchmark, end-to-end per
docs/adding_a_benchmark_city.md: bundle → GT review → scoring → operating-point integration → #55 correction → documentation. Auto-labeler side: ProjectSidewalk/sidewalk-auto-labeler#29 (bundle export merged there as ProjectSidewalk/sidewalk-auto-labeler#30).Why this city
Phase-0 purpose, recorded verbatim: "second GSV city; out-of-training but metro-adjacent" — not a geographic-transfer test. bend, previously the only GSV split, is also a Stage-2 training city, so "GSV" and "in-domain" were confounded in every GSV number (
docs/operating_point.mdflagged this explicitly). Paterson is clean per the contamination registry (docs/data_provenance.md), and is the first split that is a live Project Sidewalk deployment — the boundary is the deployment's own regions API footprint (68 polygons → ~22.5 km² ≈ the whole municipality), so a future crowd agree-rate comparison shares the benchmark's footprint by construction.Headline numbers
Reviewer-rated confidence HIGH (rubric fit; second rater expected within the CIs). Recall pool covers all 125 panos; empty stratum 24/25 attested clean (strongest negative check of any split); 1 duplicate; 2.5% unsure.
What the split found
min_distance=3re-extraction to separate "NMS-suppressed" from "never fired" (Peak extraction's min_distance (10) is under half the matcher's radius (22.5 px), so it can emit guaranteed-FP duplicates #62-adjacent).tagcheck10/10.Runbook checklist
Phase 1 — bundle (auto-labeler)
main.py --source gsv; 34,427 panos; run-level detection rate 31.7%, densest city yet)export_benchmark.py --bundleexited zero;index.csvwritten;decayed.txtreviewed (0 decayed; reconcile 1:1)records.jsonlcommitted;panos/local + archived (makelab2 full-city; klone gscratch bundle, sha256-verified 125/125)Phase 2–3 — ground truth
gt_gallery.pyat model resolution (jonf, 2026-07-29)review_noteswritten (confidence HIGH + paired-TSI and far-field caveats)verdicts.jsoncommittedscore_validation.pyrun; unbiased column recordedrecords.jsonl— absent in all 125, structural for the GSV path; the split'ssource: launchfield drivestier_of()instead, so noSPLIT_IMAGERY_FALLBACKentry was needed (better than bend, which needs the fallback)Phase 4 — operating point
PYTHON=requirement its first failed submission exposed)sweep/hist/gtbias/floor/distancere-run; caches + CSVs committedincremental_fp_tags.jsoncommitted;correctedre-run at 0.25/0.30/0.35;tagcheck100%Phase 5 — code
US_SPLITS(pooled: US deployment city, verdict-grade GT at HIGH confidence — default per the runbook; noHELD_OUTentry needed)tier_of()resolves via per-panosource(gsv tier)SERIEScolour = validated palette slot 8 (#e34948); 8-slot sequence passes the palette validatorpytest -qgreen (241)Phase 6 — docs
benchmark/README.md: both tables + paterson prose sectiondocs/model_comparison.md: coverage matrix incl. A-rate; anchoring caveat lists seven citiesdocs/operating_point.md: every table (parity, pooled, per-split, per-tier, distance, corrected, storage-floor) + prose; flip-TTA section carries a scope note (its arm predates paterson, deliberately not extended — Experiment: flip-TTA at deployment operating points — the evaluated config used it, deployed inference doesn't #78 closed)min_distance=3re-check (logged as future work); HF dataset archiving (Publish deployment validation ground-truth as a HuggingFace dataset (Bend GSV + Richmond Mapillary) #21, now five splits behind)🤖 Generated with Claude Code (claude-fable-5)