From 68c1caa3d2958003a382389c6bcc4101e0328c03 Mon Sep 17 00:00:00 2001 From: igerber Date: Mon, 7 Sep 2026 08:24:22 -0400 Subject: [PATCH 1/2] feat(dml): Caetano et al. (2026) bad-control lane on DMLDiD (PR-B) Implements the covariate-unconfoundedness doubly-robust / DML estimator of Caetano, Callaway, Payne & Sant'Anna (2026), "Difference-in-differences with 'bad controls'" (arXiv:2608.03881; Eq. 10-11, Algorithm 1), as fit-time arguments on DMLDiD: fit(..., bad_control=, bad_control_covariates=). - _dr_scores.py: ccps_panel_score / ccps_panel_score_augmented (reduce to chang_panel_score with nu = m, omega = ps/(1-ps)). - _crossfit.py: per-fold generator iter_fold_fits / FoldFit under cross_fit_predict (behavior-preserving; deep-copy warning prefix now "_crossfit: could not deep-copy ..."). - dml_did.py: _compute_ccps_gt + _ccps_nuisances (m, p, nested nu / omega; parametric in-sample plug-in vs split-half swap-and-average), omega clip with warning, complete-case policy incl. treated X_t, per-cell ATT_X(g,t) diagnostic with its own IF payload; the inference tail factored into _finish_panel_cell / _cell_se_from_payload; fail-closed gates (RCS, survey_design, anticipation, universal base); name-level validation. - dml_did_results.py: bad_control / bad_control_covariates / bad_control_diagnostics fields, bad_control_summary(), summary() header lines, att_x / se_x in to_dataframe(), conditional to_dict keys. - Reporting: target-parameter clause + reference, BusinessReport identification branch + Caetano citation, practitioner refit snippet. - Tests: score-level DR / orthogonality / reduction; numpy Eq. 11 oracle; split-half oracle learners; plug-in identity; SA DGP 1 / 4 recovery + slow MC coverage; TestBadControlAPI (reduction bit-identity, validation matrix, complete cases, column order, results surface, cluster / single PSU, bootstrap + aggregations, split-half replication, reporting); iter_fold_fits contract; variance-conventions row; guides pin; R badcontrols 1.0.0 (commit 651ccc92) black-box goldens with a tolerance-based parity test (10-seed means both sides). - Docs: REGISTRY DMLDiD "Bad-control extension" block + CS Approach-1 Note + infra section; API pages; paper-review checklist flips; guides; choosing / decision tree / index / README; survey docs; references; doc-deps; variance-conventions table; TODO / DEFERRED rows; changelog fragment. --- DEFERRED.md | 6 +- README.md | 3 +- TODO.md | 7 + benchmarks/R/README.md | 61 + benchmarks/R/generate_badcontrols_golden.R | 175 + benchmarks/R/requirements.R | 44 +- benchmarks/data/badcontrols_golden.json | 3402 ++++++++++++++ benchmarks/data/badcontrols_panel.csv | 4001 +++++++++++++++++ changelog.d/20260905-bad-controls-dml.md | 55 + diff_diff/_crossfit.py | 457 +- diff_diff/_dr_scores.py | 140 +- diff_diff/_reporting_helpers.py | 22 +- diff_diff/business_report.py | 55 +- diff_diff/dml_did.py | 1045 ++++- diff_diff/dml_did_results.py | 110 +- diff_diff/guides/llms-autonomous.txt | 15 +- diff_diff/guides/llms-full.txt | 8 +- diff_diff/guides/llms-practitioner.txt | 17 +- diff_diff/guides/llms.txt | 3 +- diff_diff/practitioner.py | 17 + ...iff_diff.dml_did_results.DMLDiDResults.rst | 4 + docs/api/dml_did.rst | 80 +- docs/api/staggered.rst | 15 + docs/choosing_estimator.rst | 9 +- docs/doc-deps.yaml | 23 +- docs/index.rst | 2 +- docs/methodology/REGISTRY.md | 242 +- docs/methodology/REPORTING.md | 17 + .../methodology/papers/caetano-2026-review.md | 59 +- docs/methodology/survey-theory.md | 8 +- docs/methodology/variance-conventions.md | 1 + docs/practitioner_decision_tree.rst | 5 +- docs/references.rst | 4 + docs/survey-roadmap.md | 1 + tests/test_crossfit.py | 156 + tests/test_dml_did.py | 679 +++ tests/test_dml_did_bad_controls_parity.py | 214 + tests/test_guides.py | 4 + tests/test_methodology_dml_did.py | 359 ++ tests/test_methodology_dr_scores.py | 174 + tests/test_variance_conventions.py | 34 + 41 files changed, 11490 insertions(+), 243 deletions(-) create mode 100644 benchmarks/R/generate_badcontrols_golden.R create mode 100644 benchmarks/data/badcontrols_golden.json create mode 100644 benchmarks/data/badcontrols_panel.csv create mode 100644 changelog.d/20260905-bad-controls-dml.md create mode 100644 tests/test_dml_did_bad_controls_parity.py diff --git a/DEFERRED.md b/DEFERRED.md index 795c8e253..2adfbfa86 100644 --- a/DEFERRED.md +++ b/DEFERRED.md @@ -45,6 +45,10 @@ provenance and AI-review deviation-documentation: a row here (or in | **`LPDiD` non-absorbing exit-event dynamics** (Dube et al. 2025 online Appendix C `eta_h^{g,n}`): the shipped `non_absorbing` modes estimate the **entry-effect** estimands (Eq. 12/13) only; separate dynamic event-studies for treatment switch-*offs* are not implemented. Needs the exit-event clean-sample derivation + estimand contract. | `lpdid.py`, REGISTRY | PR-C follow-up | Low | | MMM interop, survey/RC totals for the five `aggregate('total')` adopters (CS/DMLDiD/EfficientDiD/ImputationDiD/TwoStageDiD; the shipped v1 is panel non-survey only and fails closed with `NotImplementedError` on repeated-cross-section-routed fits, fits declaring a `survey_design=`, and the CS/DMLDiD bare-`cluster=` divergent corner where the cohort-mass weighting disagrees with the complete-case count): needs (a) DESIGN-AWARE raw-mass recovery - analytic pweight/aweight resolved weights are normalized to sum = n and CS retains no raw record (fit-time RAW retention required there; CS accepts pweight only), while EDiD's analytic fweight fits and all replicate-design fits already retain raw scale in the kits and Imputation/TwoStage retain raw `SurveyMetadata.sum_weights` - and (b) the mass-uncertainty `att*dC` variance derivation (the shipped relay conditions on the realized mass and omits it). Never ship a survey/RC total from normalized masses. | `diff_diff/staggered_results.py`, `diff_diff/survey.py` | mmm-interop | Low | | **`LPDiD` non-absorbing interior-gap support**: non-absorbing modes require a gap-free panel within each unit's observed span and raise on interior time gaps (the `[t-L, t+h]` window conditions can't be verified across a gap). The absorbing path already reindexes interior gaps to the calendar grid; extending that fail-closed handling (per-window gap masking) to non-absorbing is deferred. | `lpdid.py::_prepare_panel` | PR-C follow-up | Low | +| Bad-control lane on repeated cross sections: Caetano et al. (2026) Remark 1 calls the approach unavailable "to a large extent" with repeated cross sections (`X_{g-1}` and `X_t` must be observed for the same unit); an RCS variant needs its own derivation. `DMLDiD(panel=False).fit(bad_control=)` fails closed. | `dml_did.py` | bad-controls PR-B | Low | +| Limited-anticipation extension of the bad-control lane: MP-2 assumes no anticipation for `Y` AND `X`; the base-period convention for `X` and `W` under `anticipation > 0` is not developed by the paper. `DMLDiD(anticipation=k).fit(bad_control=)` fails closed. | `dml_did.py` | bad-controls PR-B | Low | +| Universal-base bad-control pre-test: MP-5 conditions on the immediately preceding period, and under `base_period="universal"` a pre cell with `t < g - 1` would read `X_t` earlier than `X_{g-1}` / `W`, inverting the ordering; a universal-base version needs its own derivation. `DMLDiD(base_period="universal").fit(bad_control=)` fails closed. | `dml_did.py` | bad-controls PR-B | Low | +| Bad-control lane under `survey_design=`: the nested second-stage nuisances under design weights (Hajek moments, `sample_weight` into the nested fits) and the replicate-weight variance of the nested scores are unvalidated; fails closed (bare `cluster=` is supported). | `dml_did.py` | bad-controls PR-B | Low | ## Needs external reference (R / Stata / Julia) @@ -111,7 +115,7 @@ For survey-specific limitations (`NotImplementedError` paths), see the | Rust local-method `estimate_model` → unify to `solve_wls_svd` (the global-method's SVD helper) for sub-1e-14 bootstrap-SE parity. The local-method bootstrap parity test passes at `atol=1e-5`; the residual ~1e-7 is roundoff, not a user-visible correctness bug. | `rust/src/trop.rs`, `rust/src/linalg.rs` | follow-up | Low | | Validate the `.txt` AI guides (`llms-full.txt`, `llms-practitioner.txt`) as executable snippets — **not low-lift** (re-scoped 2026-06-01): only ~20% of ~112 fenced blocks are standalone-runnable; the rest are signature pseudo-code, context fragments, or data-shape-specific. Needs signature-block detection + a context/data skip-allowlist + per-snippet fixtures. | `tests/test_doc_snippets.py` | #239 | Low | | `TestWorkflowDoesNotExecutePRHeadCode` (CodeQL #14 guard) doesn't model `bash/sh/./source