feat(dml): Caetano, Callaway, Payne & Sant'Anna (2026) bad-control lane on DMLDiD (PR-B)

[igerber]

Summary

• PR-B of the bad-controls initiative (PR-A docs(methodology): Caetano, Callaway, Payne & Sant'Anna (2026) bad-controls paper review (PR-A) #819 put the paper review on file). Implements the covariate-unconfoundedness doubly-robust / DML estimator of Caetano, Callaway, Payne & Sant'Anna (2026), "Difference-in-differences with 'bad controls'" (arXiv:2608.03881; Eq. 10-11, Algorithm 1, Propositions 5-7; staggered per Proposition 2 / SB.3), as fit-time arguments on DMLDiD: fit(..., bad_control=, bad_control_covariates=). No new estimator class; bad_control=None is the untouched pre-existing code path (bit-identity pinned on every payload).
• _dr_scores.py: ccps_panel_score / ccps_panel_score_augmented (the paper's four-term orthogonal score; reduces to chang_panel_score with nu = m, omega = ps/(1-ps), pinned at 1e-14).
• _crossfit.py: per-fold generator iter_fold_fits / FoldFit extracted under cross_fit_predict (behavior-preserving; the deep-copy fallback warning prefix becomes "_crossfit: could not deep-copy ...").
• dml_did.py: _compute_ccps_gt + _ccps_nuisances (m, p, nested nu / omega: parametric linear/logit use in-sample fold-k targets = the paper's Assumption-8 plug-in; ridge / sieve / user learners split the training complement in half, swap and average, footnote 9), omega clipped to [0, (1-trim)/trim] with a warning, one complete-case rule incl. treated X_t, per-cell ATT_X(g,t) diagnostic (Remark 6; AIPW on levels with its own IF payload through the shared SE / df branch), the plain lane's inference tail factored into _finish_panel_cell / _cell_se_from_payload; fail-closed gates (RCS, survey_design, anticipation != 0, universal base -> NotImplementedError; bare cluster= supported), name-level ValueErrors.
• dml_did_results.py: bad_control / bad_control_covariates / bad_control_diagnostics fields, bad_control_summary(), summary() header lines (banner, W, ATT_X cell counts, analytical-SE note on bootstrapped fits, CS-simple weighting note), att_x / se_x joined in to_dataframe() at level="group_time", conditional to_dict() keys, pre-upgrade-pickle safe reads.
• Reporting: target-parameter clause + reference string, BusinessReport identification branch + second citation, practitioner learner-sensitivity snippet carries the bad-control args (and control_group=).
• R oracle (black box only, GPL-3 never read): benchmarks/R/generate_badcontrols_golden.R + commit-pinned installer in benchmarks/R/requirements.R (badcontrols 1.0.0 at 651ccc92, ptetools 1.0.0), benchmarks/data/badcontrols_{panel.csv,golden.json}, benchmarks/R/README.md section.
• Docs: REGISTRY DMLDiD "Bad-control extension (CCPS 2026)" block (equations as implemented + Notes: nested-stage convention, omega clip, W default, base-period semantics incl. pre-period Y_{t-1}, anticipation, pi_hat, fail-closed lanes, complete cases, time-invariant bad control, aggregation weights vs Remark 4, ATT_X inference, validation scope), CS Approach-1 Note, infra section (fourth score family, iter_fold_fits contract); docs/api/dml_did.rst methodology / restrictions / runnable snippet; docs/api/staggered.rst "Covariates and bad controls"; paper-review checklist flipped with library annotations; guides (llms.txt signature + Diagnostics entry, llms-full.txt, practitioner pitfall 3 rewrite, autonomous matrix); choosing / decision tree / index / README; survey roadmap + theory carve-outs; REPORTING.md Note; references; doc-deps; regenerated variance-conventions table; TODO / DEFERRED rows; changelog fragment.

Methodology references (required if estimator / math changes)

• Method name(s): Caetano-Callaway-Payne-Sant'Anna (2026) bad-control doubly-robust / DML DiD (Eq. 10-11, Algorithm 1) on the DMLDiD CS cell architecture; ATT_X(g,t) pre-test (Remark 6); Approach 1 (Theorem 1 / Proposition 3) documented on CallawaySantAnna / DMLDiD base-period covariates.
• Paper / source link(s): https://arxiv.org/abs/2608.03881 (v2 + Supplementary Appendix v1; review on file at docs/methodology/papers/caetano-2026-review.md, a deliberate exception to the published-source rule). R reference hugosantanna/badcontrols (GPL-3) executed only as a black-box oracle.
• Any intentional deviations from the source (and why): all recorded as REGISTRY DMLDiD Notes - nested-stage targets (in-sample plug-in for the parametric built-ins, split-half otherwise; the paper leaves the construction to the practitioner); omega_hat clipped to [0, (1-trim)/trim] (S2(iv) boundedness; the paper gives no rule); W default = none (R parity; Remark 5's Y_{g-1} is the recommended explicit choice); pi_hat = the cell's treated share; varying-base pre-period cells read W = Y_{t-1}; headline att keeps the CS "simple" weighting rather than Remark 4's overall (TODO row); RCS / anticipation / universal base / survey_design fail closed (DEFERRED rows); ATT_X is an analytical mean-effect diagnostic, never aggregated or bootstrapped.

Validation

• Tests added/updated: tests/test_methodology_dr_scores.py (TestCCPSScore: oracle recovery, paired double robustness, four-direction Neyman orthogonality vs purpose-built non-orthogonal comparators, reduction to Chang, augmented identity, validation), tests/test_methodology_dml_did.py (SA DGP frame + moment pins, numpy Eq. 11 / Algorithm 1 oracle at 1e-12 for ATT / SE / ATT_X, gapped time grid, Assumption-8 plug-in identity, oracle user learners on the split-half branch, DGP 1 / 4 recovery with a discriminating naive include-X_t contrast, slow MC coverage), tests/test_dml_did.py::TestBadControlAPI (43 tests: reduction bit-identity, validation matrix, complete cases, S/R column order, results surface, diagnostics schema, omega clip, ATT_X NaN contract, split-half hand replication + PSU-cohesive halves + degeneracy, cluster / single-PSU NaN, bootstrap + aggregations incl. the total guard, time-invariant bad control, reporting surfaces), tests/test_crossfit.py::TestIterFoldFits, tests/test_variance_conventions.py (dml_did_bad_control row), tests/test_guides.py, tests/test_dml_did_bad_controls_parity.py (new; R goldens, 10-seed means both sides, |dATT| < 0.5 SE_R + SE ratio 0.3 on the W=[W] runs and W=[Y] post cells; characterizations at 1.0 SE otherwise).
• Backtest / simulation / notebook evidence (if applicable): SA DGP 1 / 4 recover ATT = 1.00 and ATT_X = 0.50 within 3 SE (seed-pinned) with ~95% MC coverage; R badcontrols single-seed gaps <= 0.63 SE with 10-seed means agreeing within 0.1 SE, SE ratios within 1.25. Sphinx -W build passes. Tutorial section deferred to PR-C (TODO row).

Security / privacy

• Confirm no secrets/PII in this PR: Yes

Changelog

• changelog.d/ fragment added (or N/A - no user-visible change): Yes (changelog.d/20260905-bad-controls-dml.md)

[github-actions]

Overall assessment: ⚠️ Needs changes

Executive summary

• CCPS Equation 10 scoring and Proposition 7 influence-function centering match the cited methodology.
• Control composition, complete cases, degeneracy handling, parameter gates, and NaN inference are handled safely.
• P1: Practitioner-facing guidance reverses the meaning of a nonzero pre-period ATT_X.
• Intentional deviations are clearly documented and are informational only.
• A changelog fragment and extensive tests are included; runtime tests could not run because dependencies are unavailable.

Methodology

M1 — Incorrect interpretation of pre-period ATT_X

• Severity: P1
• Location: diff_diff/guides/llms-practitioner.txt:L663-L675, diff_diff/guides/llms.txt:L87-L89, README.md:L127-L129
• Impact: The guides say a nonzero pre-period ATT_X is evidence that the covariate is a bad control. Remark 6 says the opposite role: pre-period estimates assess MP-5/MP-8, so a nonzero value is evidence against the identifying assumptions. Only a nonzero post-period value supports treatment affecting the control. An automated agent could therefore treat a failed identification pre-test as support for using the estimator.
• Concrete fix: State explicitly: “Nonzero pre-period ATT_X indicates a possible MP-5/MP-8 violation; nonzero post-period ATT_X is evidence treatment affects X.” Qualify the similar generic statement in docs/methodology/REGISTRY.md:L3293-L3304.

M2 — Documented implementation choices

• Severity: P3 informational
• Impact: Split-half nested nuisances, omega clipping, unsupported-lane gates, CS-simple headline weighting, and analytical-only ATT_X are documented deviations or limitations.
• Concrete fix: None required. See docs/methodology/REGISTRY.md:L3306-L3417.

Code Quality

No unmitigated findings. Both ATT and ATT_X use safe_inference(), and degenerate scores fail closed.

Performance

No findings.

Maintainability

No findings. The shared cross-fit and inference helpers avoid duplicating statistical logic.

Tech Debt

No unmitigated findings. Deferred variants are tracked in TODO.md:L60-L65 and DEFERRED.md:L48-L51.

Security

No findings. The R oracle is commit-pinned, and no secrets or PII were identified.

Documentation/Tests

Finding M1 applies. Otherwise, the PR includes methodology oracles, DGP recovery, parity fixtures, edge-case tests, and a changelog fragment. Static syntax/AST checks passed; runtime pytest execution was unavailable because NumPy and pytest are not installed.

Path to Approval

1. Correct every pre/post ATT_X interpretation so pre-period nonzero values flag MP-5/MP-8 failure, while post-period nonzero values support treatment affecting the bad control.

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: 65c6651e635c8ddd6a20adaed06c333c8d318850

---

Overall assessment: ✅ Looks good

Executive summary

• The previous P1 is resolved: pre-period ATT_X now correctly signals possible MP-5/MP-8 violations; post-period ATT_X checks Condition 2.
• Equation 10 scoring and Proposition 7 variance centering match the cited paper.
• Control composition, empty cells, and NaN inference fail safely.
• Intentional deviations and deferred extensions are documented and tracked.
• A changelog fragment and extensive methodology/edge-case tests are present.

Methodology

No unmitigated findings. The score and influence function match Equations 10–11 and Proposition 7 (diff_diff/_dr_scores.py:L341-L396, diff_diff/dml_did.py:L1725-L2024).

The prior guidance issue is corrected consistently in diff_diff/guides/llms-practitioner.txt:L663-L679, diff_diff/guides/llms.txt:L89, and docs/methodology/REGISTRY.md:L3293-L3309.

M1 — Documented implementation choices

• Severity: P3 informational
• Impact: Nested-stage construction, omega clipping, unsupported-lane gates, CS-simple headline weighting, and analytical-only ATT_X inference differ from optional alternatives but are explicitly documented.
• Concrete fix: None required. See docs/methodology/REGISTRY.md:L3311-L3422.

Code Quality

No findings. Both ATT and ATT_X use safe_inference() with joint NaN propagation.

Performance

No findings.

Maintainability

No findings. Shared cross-fitting and cell-finalization helpers limit duplicated statistical logic.

Tech Debt

T1 — Deferred extensions are tracked

• Severity: P3 informational
• Impact: Remark-4 aggregation, ATT_X aggregation/bootstrap, empty-Z support, RCS, survey, anticipation, and universal-base variants remain unavailable but fail closed.
• Concrete fix: None required for this PR; tracked in TODO.md:L60-L65 and DEFERRED.md:L48-L51.

Security

No findings. The R oracle is commit-pinned, and no secrets or PII were detected.

Documentation/Tests

No findings. The behavior change has changelog.d/20260905-bad-controls-dml.md, and tests cover methodology oracles, inference degeneracy, clustering, bootstrap, aggregation, and R parity.

Runtime tests could not be collected because pytest is unavailable in the review environment. Static parsing of all 324 Python files succeeded.