fix(linalg): hc2 / unweighted hc2_bm fail closed at leverage-one observations (family-wide with hc3)

[igerber]

Summary

Resolves the TODO.md backlog row "hc2/hc2_bm floor 1 - h_ii at 1e-10 in the shared leverage meat" (origin #588). At a leverage-one observation, vcov_type="hc2" (every weight type) and unweighted, unclustered vcov_type="hc2_bm" now return an all-NaN covariance and DOF vector with a UserWarning, the same contract hc3 already had. Point estimates are unchanged; only the undefined inference is suppressed.

Why. A perfectly fitted row (the single treated unit under [1, D], a singleton cohort x period cell in a full-dummy design) has hat-matrix leverage 1 and residual exactly 0, so the leverage-corrected meat term u_i^2 / (1 - h_ii) is 0/0. The former floor max(1 - h, 1e-10) did not inflate that term; it silently dropped the row's outcome noise and reported an understated finite variance. On the probe design (X = [1, D], n = 6, one treated row, y = numpy.random.default_rng(0).normal(size=6)):

vcov_type	SE(D) before	SE(D) after
classical	0.469	0.469
hc1	0.210	0.210
hc2	0.191	NaN + warning
hc2_bm	0.191 (with a NaN Satterthwaite DOF)	NaN + warning
hc3	NaN + warning	NaN + warning

Source evidence. R sandwich::vcovHC(type = "HC2") and "HC3" return NaN for every coefficient at exact leverage one on this probe and warn that the covariances are undefined there. The new hc2 behavior agrees on this probe; the library retains its existing hc3 threshold of h_ii >= 1 - 1e-8, rather than claiming identical behavior to R throughout the near-one region. R clubSandwich::vcovCR(cluster = 1:n, type = "CR2") returns a finite variance on the same design (SE 0.1915, Satterthwaite df 4) because its Moore-Penrose generalized inverse zeroes that row's adjustment; that is the same understated value the floor produced. Unweighted one-way hc2_bm therefore deviates from clubSandwich and is recorded with the **Deviation from R:** label in docs/methodology/REGISTRY.md.

Final behavior.

• The guard fires when any positive-weight row has h_ii >= 1 - 1e-8. The whole vcov is NaN (as in R and in hc3), not per coefficient. The warning names the offending rows, states that point estimates are unaffected, and recommends vcov_type="classical" (exact inference under homoskedastic normal errors, which needs positive residual df) or adding observations to the perfectly fitted cell. hc1 is deliberately not offered as a remedy because at such a row it also omits the zero-residual observation and returns the understated number.
• The over-one-leverage "fall back to HC1 with a warning" branch is retired; that case is inside the guard.
• Under fweight, a zero-count row's leverage (the unweighted quadratic form against the weighted bread) is unbounded and meaningless. Zero-count rows are excluded from the guard and from the meat, so compressed HC2/HC3 equal the literal np.repeat expansion at any leverage. This also fixes a pre-existing case where an inert zero-count row could NaN the hc3 vcov or push hc2 into the HC1 fallback.
• The one-way Bell-McCaffrey DOF helper drops its sibling floor: at leverage one every contrast's DOF is NaN because the variance it belongs to is undefined; below leverage one its per-contrast noise-floor guard is unchanged.
• Weighted (pweight) one-way hc2_bm routes through the clubSandwich WLS-CR2 port and keeps its finite generalized-inverse result at leverage one, including under a no-op weights=np.ones(n); clustered CR2 keeps the Moore-Penrose absorbed-cluster-FE convention. Both are documented as unchanged, and the resulting hc2_bm versus hc2_bm + weights=ones discontinuity is stated in the registry and changelog.
• The Rust HC2 kernel drops its floor and raises a "Hat-matrix leverage ~1" sentinel. The Python dispatcher re-dispatches that sentinel, and the legacy over-one sentinel of a kernel built before this change, to the NumPy branch, which owns the single warning; attribution is to the caller on both backends. A stale kernel cannot signal h == 1 exactly, so the extension must be rebuilt (maturin develop --release); a direct-kernel test fails loudly on a stale build.
• LWDiD no longer emits a second, differently worded warning per cell; it already failed closed on these designs and its behavior is otherwise unchanged.

Affected estimator paths (verified by tracing the vcov call on fixtures with maximum leverage 1.0): DifferenceInDifferences, MultiPeriodDiD and LinearRegression on both families; TwoWayFixedEffects on explicit hc2 and on hc2_bm in event-study spec="pooled" without unit= (no auto-cluster); SunAbraham and WooldridgeDiD (OLS) on one-way hc2 whenever the full-dummy design has a singleton cohort x period cell. Such fits previously returned finite, understated SEs and now return NaN inference with preserved point estimates. StackedDiD cannot reach the branch (hc2 rejected at __init__; hc2_bm always clustered).

Files. diff_diff/linalg.py (guard, floor removal, fweight handling, DOF helper, Rust dispatch, docstrings), rust/src/linalg.rs, diff_diff/lwdid.py, class docstrings in estimators.py / twfe.py / sun_abraham.py / wooldridge.py, docs/api/lwdid.rst, tutorial 31 (markdown cell only), docs/methodology/variance-conventions.md, docs/methodology/REGISTRY.md (new family-wide Note, Deviation from R, weighted/clustered Note; amended one-way-scores, fweight, guard-ordering and LWDiD notes), TODO.md row removed, changelog.d/20260906-hc2-leverage-one-fail-closed.md.

Methodology references (required if estimator / math changes)

• Method name(s): HC2 / HC3 leverage-corrected heteroskedasticity-robust variance (MacKinnon & White 1985); Imbens & Kolesar (2016) Bell-McCaffrey Satterthwaite DOF; Pustejovsky & Tipton (2018) CR2.
• Paper / source link(s): R sandwich::vcovHC (HC2/HC3 NaN at exact leverage one on the probe) and R clubSandwich::vcovCR(cluster = 1:n, type = "CR2") (finite generalized-inverse variance), both run locally on the probe design.
• Any intentional deviations from the source (and why): unweighted, unclustered one-way hc2_bm fails closed where clubSandwich's singleton CR2 returns a finite understated variance (the observation's outcome noise is dropped); recorded as **Deviation from R:** in REGISTRY.md. Weighted hc2_bm and clustered CR2 keep the clubSandwich convention (recorded as a **Note:**).

Validation

• Tests added/updated:
  • tests/test_linalg_hc2_bm.py::TestLeverageOneFailsClosed: both entry points and both families with and without DOF; message count and five-row preview on a six-dummy design; aweight/pweight; fweight count-1 vs count-2 singleton; zero-count fweight expansion parity for hc2 and hc3 at leverage 1.0 and 2.7; mixed design proving the zero-count mask does not disable the guard; weighted hc2_bm + ones stays at the clubSandwich value; over-one leverage; legacy Rust sentinel re-dispatch; no-floor denominator on a defined high-leverage design; DOF helper; warning attribution.
  • tests/test_estimators_vcov_type.py::TestLeverageOneEstimators: DiD (hc2, hc2_bm), MultiPeriodDiD unclustered hc2_bm (per-period and post-average), TWFE pooled event study without unit= plus its clustered control, TWFE static explicit hc2, SunAbraham singleton cohort, WooldridgeDiD singleton cohort; each checks the warning, point estimates against a classical/hc1 fit of the same panel, and all-NaN inference.
  • tests/test_rust_backend.py::TestRustHC2Vcov: fail-closed parity on an exact-unit-leverage design, direct-kernel sentinel test, legacy-sentinel re-dispatch, warning attribution; class docstring updated.
  • tests/test_linalg.py::TestOneWayBMScoresDOF::test_leverage_one_nans_every_contrast (replaces the per-contrast pin), tests/test_lwdid.py leverage-one hc2 test (kernel-shaped warnings only), tests/test_methodology_wls_cr2.py equivalence test scoped to non-degenerate designs.
• Runs executed in the frozen local environment (Python 3.14, NumPy 2.5.2, Rust extension rebuilt with maturin develop --release):
  • pytest tests/test_linalg_hc2_bm.py tests/test_linalg.py tests/test_rust_backend.py tests/test_methodology_wls_cr2.py tests/test_estimators_vcov_type.py tests/test_lwdid.py: 881 passed, 28 deselected.
  • DIFF_DIFF_BACKEND=python pytest tests/test_linalg_hc2_bm.py tests/test_rust_backend.py tests/test_linalg.py: 290 passed, 150 skipped.
  • Pre-merge scan resolved run-list (26 test files): 1838 passed, 43 skipped.
  • Estimator and docs sweep (24 files including test_tracking_files.py, test_changelog_fragments.py, test_docs_ia.py, test_variance_conventions.py): 2602 passed, 45 skipped.
  • cargo test --release: 57 passed; cargo fmt --check: clean.
  • ruff check diff_diff tests, black --check diff_diff tests, .claude/scripts/changelog_compile.py check: clean.
• Backtest / simulation / notebook evidence (if applicable): probe script and R anchors above; tutorial 31 change is markdown only, no re-execution.
• Material limitations: mypy diff_diff with the project configuration (Python 3.10 target) fails on unchanged pinned source because the frozen environment's NumPy 2.5.2 stubs use Python 3.12 type statements (numpy/__init__.pyi:737); the changed modules type-check cleanly with mypy --python-version 3.14, and the environment and project configuration were left unchanged, so the CI Lint Gate with its NumPy 2.4.5 pin is the authoritative type check. The pre-merge scan reports one Check-D pattern hit at diff_diff/linalg.py:5593 (_compute_confidence_interval definition); it is pre-existing and outside this diff. The full test suite and slow-marked tests were not run locally.

Security / privacy

• Confirm no secrets/PII in this PR: confirmed. Code, tests, docs and a changelog fragment only.

Changelog

• changelog.d/ fragment added (or N/A - no user-visible change): changelog.d/20260906-hc2-leverage-one-fail-closed.md (Fixed + Behavioral Changes).

[github-actions]

PR Review

Overall assessment

✅ Looks good — no unmitigated P0 or P1 findings.

Executive summary

• HC2, HC3, and unweighted one-way HC2-BM behavior correctly fails closed when leverage reaches one.
• NaN covariance propagates consistently through SE, t-statistic, p-value, CI, and aggregate inference.
• The clubSandwich deviations and weighted/unweighted discontinuity are explicitly documented.
• Zero-count frequency-weight rows correctly preserve literal-expansion parity.
• A changelog fragment and extensive regression tests are included.
• Tests were not executable here because pytest is unavailable.

Methodology

M1 — Documented HC2-BM deviation

• Severity: P3 — informational
• Impact: Unweighted, unclustered HC2-BM returns NaN at leverage one, while clubSandwich uses a generalized inverse and returns finite inference. Supplying all-one pweights consequently changes this behavior.
• Evidence: docs/methodology/REGISTRY.md:L736-L738, diff_diff/linalg.py:L3649-L3675
• Concrete fix: None required; both the deviation and discontinuity are explicitly labeled in the Registry.

The HC2/HC3 denominator behavior agrees with MacKinnon–White at exact leverage one, and suppressing Bell–McCaffrey DOF when its covariance is undefined is internally consistent.

Code Quality

No findings. The warning, NaN return shape, and affected-family labels are centralized in helpers at diff_diff/linalg.py:L3430-L3463.

Performance

No findings. The guard occurs before meat construction, and Rust-to-NumPy redispatch is limited to the exceptional leverage-one path.

Maintainability

No findings. Python and Rust use the same threshold and sentinel contract, with legacy-sentinel compatibility.

Tech Debt

No findings. The resolved backlog row is removed from TODO.md, with the resulting behavior captured in the Methodology Registry.

Security

No findings. No secrets, unsafe input handling, or security-sensitive changes were identified.

Documentation/Tests

No findings. User-visible behavior has the required fragment at changelog.d/20260906-hc2-leverage-one-fail-closed.md, and tests cover NaN propagation, estimator aggregation, weighting interactions, Rust dispatch, and warning attribution.