Flip-TTA at the deployment operating points: measured, and not worth 2× GPU (#78)

[jonfroehlich]

Closes #78.

What this measures

The evaluated configuration (stage_two/evaluate.py, horizontal-flip TTA) and the deployed detector (single-pass) have differed by an unmeasured delta since sidewalk-auto-labeler#3. This rides #54's low-floor extraction exactly as the issue prescribed: a second extract --tta arm (mirrored heatmap un-flipped and maxed with the original — numerically identical to evaluate.py's composition, with tests asserting the clip-order equivalence), swept post-hoc on identical GT alongside the single-pass arm. manual_gold needed no GPU pass: its committed detections are a TTA export at the 0.05 floor, so the records supplied that arm.

What flip-TTA actually is

For anyone reading this cold: it is the obvious thing. Run the panorama through the model; mirror the panorama and run it again; un-mirror that second heatmap back into pano orientation; combine the two. Nothing is retrained — it is purely an inference-time trick that costs a second forward pass.

The combine rule is where the entire result comes from. stage_two/evaluate.py:191 composes with elementwise np.maximum, not an average.

Two reasons this was worth running even though the outcome is negative:

1. The accounting reason, which was the real driver. --tta defaults to True in evaluate.py, commented "as in the paper" — so every committed gold-set number is a TTA number, while the deployed labeler is single-pass. Sizing that gap was mandatory regardless of whether TTA turned out to be worth deploying.
2. The ensembling hope. Two views, partly uncorrelated errors, and RampNet is recall-first (a false negative is a permanently missed ramp; a false positive is cheap to reject downstream). A cheap recall lever deserved a measurement.

The answer

Pooled US (609 panos): TTA at the deployed 0.55 buys +2.4 recall points for −1.1 precision — but after the #54 threshold drop to 0.30, the marginal value collapses to +0.9 R / −2.4 P, with F1 down on four of the five US splits. TTA is a mild threshold-lowering in disguise (its recall curve is the single-pass curve shifted left — see the committed figure), so the free lever absorbs almost everything the 2×-GPU lever would buy. AP spans −0.012 (annapolis) to +0.014 (morgantown), +0.004 pooled.

lever (from single@0.55, pooled US)	ΔR	ΔP	ΔF1
threshold drop alone (0.55→0.30)	+0.074	−0.068	+0.016
TTA alone (at 0.55)	+0.024	−0.011	+0.011
both	+0.083	−0.092	+0.009
TTA after the drop (at 0.30)	+0.009	−0.024	−0.007

manual_gold (in-domain control) is the one exception — marginal +1.9 R / −1.5 P, AP 0.904 → 0.917 — so the paper's TTA evaluation was a reasonable in-domain choice; it just doesn't transfer to deployment imagery. And the configuration mismatch is now quantified and closed: TTA-evaluated numbers read +2.4 recall points above what single-pass deployment delivers at 0.55.

Negative result, recorded deliberately: no production TTA knob gets filed in sidewalk-auto-labeler. The #78 plan filed it only if TTA won at the chosen operating point; it loses there. The 0.30 single-pass recommendation from #54 is unchanged.

Why the mechanism produces exactly this result

Both halves of the finding fall out of two properties of the setup, neither of them specific to this benchmark. This is now written into docs/operating_point.md so the reasoning survives with the numbers.

Max-combine can only add detections. Because the two passes combine pixelwise by max, the TTA heatmap is pointwise ≥ the single-pass heatmap everywhere. At a fixed threshold, TTA can promote a detection but can never suppress one. That is the mechanical reason it behaves as a threshold change rather than merely correlating with one: the two levers are the same kind of operation — both admit weaker evidence, trading precision for recall. Hence "threshold-lowering in disguise" is a description of the operator, not an analogy.

(An averaging TTA would be a different animal: a variance reducer, able to remove false positives as well as add true ones. That is not what the paper used and not what these numbers measure. It remains untested.)

The model was already trained to be flip-invariant. stage_two/train.py:202-207 applies random horizontal flip at p = 0.5, correctly relabelling x → 1 − x; the crop models do the same (stage_one/crop_model/*/train.py:71). Test-time flipping can therefore only exploit whatever asymmetry the training-time flip failed to remove. A small gain is the expected outcome, not an anomaly — training aug and test aug are the same transform pointed at the same invariance, and the better the former worked, the less the latter can find.

Read the other way, this makes the marginal TTA gain a measurement: it says how much the two views still disagree in a way that matters at the operating point. At ≤1.3 recall points per US split, the answer is "barely" — the training augmentation did its job.

What this does not say — follow-up filed as #82

That second point also bounds the scope of the negative result. The horizontal flip is the only augmentation in the training recipe: the input transform is Resize(2048×4096) → ToTensor → Normalize (train.py:224-229), with no photometric, scale, or other geometric variation, on either the pano model or the crop models.

So #78 establishes that the one invariance that was trained for is saturated. It says nothing about the ones that were never trained for — and the benchmark's live failure mode is cross-city transfer (GT-measured recall at 0.55 spans richmond 0.895 down to clovis 0.650). #82 takes that up: yaw-roll and photometric jitter, which are the two augmentations that are simultaneously legal on an equirectangular projection and untried. Worth noting for anyone reaching for the standard list — in-plane rotation and perspective warp are not legal here (the top and bottom image rows are the poles; an image-plane rotation mixes pole with horizon and destroys the projection), whereas horizontal roll is exactly a camera yaw, wraps seamlessly, and is exactly label-preserving.

What's in the PR

• operating_point_curve.py extract --tta + arm guard (TTA and single-pass caches cannot share a directory — extract's resume-skip would otherwise silently score one arm as the other), arm-aware parity gate, low_floor_sweep.py tta with the four-lever decomposition, plot_tta.py + committed figure, the docs/operating_point.md section (including the caveat direction: the GT-completeness correction: spot-check the low-confidence incremental FPs from the operating-point curve #55 GT bias overstates TTA's marginal precision cost if anything, and the exact marginal recall gain is too small to price 2× GPU against), and the reproduce/README updates.
• The mechanism subsection above, and a #82 entry in "What would change this" — better transfer would move the 0.30 threshold itself, since part of what 0.30 is compensating for is out-of-domain under-confidence.
• 241 tests pass. Run log and interim results are in the Experiment: flip-TTA at deployment operating points — the evaluated config used it, deployed inference doesn't #78 comments.

🤖 Generated with Claude Code (claude-fable-5)
🤖 "What flip-TTA actually is" / "Why the mechanism produces exactly this result" sections added with Claude Code (claude-opus-5[1m])

[jonfroehlich]

Deep Code Review: PR #81

PR Title: Flip-TTA at the deployment operating points: measured, and not worth 2× GPU (#78)
Branch: feat/flip-tta-78
Status: All 241 unit tests pass (pytest -q verified)

---

1. Executive Summary

PR #81 evaluates horizontal-flip Test-Time Augmentation (TTA) against single-pass inference across all 7 benchmark splits (5 US city splits, 1 Budapest split, and 1 in-distribution manual_gold split).

The PR addresses a longstanding configuration mismatch identified in sidewalk-auto-labeler#3 (where evaluate.py used flip-TTA while production deployment ran single-pass). It demonstrates empirically and mathematically that TTA is not worth the 2× GPU inference cost at deployment.

Headline Findings

1. TTA acts as an implicit threshold-lowering mechanism: Because composition uses pointwise elementwise maximum $\max(H_{\text{orig}}, \text{fliplr}(H_{\text{flip}}))$, heatmaps satisfy $H_{\text{TTA}} \ge H_{\text{single}}$. TTA exclusively promotes predictions and cannot suppress false positives.
2. Diminishing returns after threshold drop: While TTA appears attractive at the legacy $0.55$ threshold (+2.4 recall points on pooled US splits), its marginal benefit collapses to +0.9 recall points at a cost of -2.4 precision points once the operating threshold is lowered to $0.30$ (as established in PR #54). F1 scores actually drop on 4 of the 5 US splits.
3. Training augmentation saturation: Because training already incorporates random horizontal flipping ($p=0.5$), the model's learned flip invariance is already near saturation ($\le 1.3$ recall points gained per split).
4. Decision: Explicitly records a negative result in documentation and declines to add a TTA knob to production auto-labeling pipelines.

---

2. Technical Code Analysis & Architecture

A. Heatmap Composition & Numerical Equivalence

• operating_point_curve.py: compose_tta(h_orig, h_flip) implements $\max(H_{\text{orig}}, \text{fliplr}(H_{\text{flip}}))$.
  • Commutativity with Clipping: The PR correctly observes and proves (via unit test test_compose_tta_commutes_with_clip) that composing raw logit heatmaps and clipping at detection time is numerically identical to stage_two/evaluate.py's clip-then-max order.
  • Mirroring Order: ImageOps.mirror(img) mirrors the input image before standard pre-processing, matching stage_two/evaluate.py.

B. Lever Decomposition & Analysis Framework

• low_floor_sweep.py: tta_levers(...) cleanly decomposes the recall gains into a 4-row ledger:
  1. threshold drop alone ($0.55 \to 0.30$ single-pass)
  2. TTA alone ($0.55$ single-pass $\to 0.55$ TTA)
  3. both ($0.55$ single-pass $\to 0.30$ TTA)
  4. TTA after the drop ($0.30$ single-pass $\to 0.30$ TTA) — the key 2× GPU decision metric.
• Added unit test test_tta_levers_ledger_adds_up verifies arithmetic consistency (both == drop + marginal).

C. Cache Safety & Arm Isolation

• operating_point_curve.py: _assert_cache_arm(cache_dir, tta) checks metadata of existing JSON files in cache_dir and refuses to mix TTA (meta["tta"] = True) and single-pass caches in the same directory.
• Prevents subtle evaluation bugs where downstream CPU scripts could score a single-pass cache as TTA or vice versa.

D. Zero-GPU Fallback for manual_gold

• low_floor_sweep.py: tta_panos_from_records(...) reuses committed manual_gold TTA export records directly without requiring a GPU pass, maintaining fast CPU-only test suite execution.

E. Visualization & Artifact Generation

• plot_tta.py: Generates docs/figures/tta_operating_point.png showing the dual PR-curve and Threshold-vs-Metrics panels.
• Uses the project-standard 2-slot color palette (#2a78d6 single-pass, #eb6834 flip-TTA).

---

3. Code Quality & Standards Compliance

Criterion	Assessment	Details
Scientific Protocol (CLAUDE.md)	EXCELLENT	Negative result is fully documented in docs/operating_point.md with theoretical justification and reproduction commands.
Test Coverage	EXCELLENT	125 new lines of tests in test_low_floor_sweep.py and test_operating_point_curve.py covering composition, clipping commutativity, cache guardrails, and ledger math. All 241 repo tests pass.
Reproducibility	EXCELLENT	analysis_out/op_cache_tta/*.json and analysis_out/op/tta_compare.csv are committed; numbers and figures re-derive on CPU without GPU or image dependencies.
Documentation	EXCELLENT	Comprehensive updates to docs/operating_point.md explaining why max-composition TTA behaves like a threshold drop and bounding future augmentation experiments (#82).

---

4. Strengths & Key Observations

1. Theoretical Clarity: The explanation of why TTA fails to add value at $0.30$ threshold (max-composition elementwise inequality $H_{\text{TTA}} \ge H_{\text{single}}$ + saturation of trained horizontal flip augmentation) elevates this PR from a simple empirical benchmark to a generalizable architectural insight.
2. Defensive Guardrails: _assert_cache_arm actively prevents accidental cache contamination across arms, saving future maintainers from silent evaluation errors.
3. No Production Complexity: By documenting the negative result, the author explicitly avoids adding dead configuration flags or unnecessary runtime complexity to the production auto-labeler.

---

5. Conclusion & Recommendation

Recommendation: MERGE AS IS

This PR is an exemplary piece of scientific research engineering. The code is well-tested, robust, thoroughly documented, and adheres strictly to repository guidelines.

🤖 Generated with Google Antigravity (Gemini 3.6 Flash)