docs(tutorials): bad-controls tutorial notebook 33 (Caetano et al. 2026, PR-C) - #831
Conversation
Overall assessment: ✅ Looks goodExecutive summary
Methodology
No undocumented mismatch, missing assumption check, or variance/SE defect found. Code QualityNo findings. PerformanceNo findings. MaintainabilityNo findings. Tech DebtNo findings. Unsupported bad-control paths and aggregation limitations remain tracked in SecurityNo findings; no secrets, PII, tracebacks, or local paths found. Documentation/TestsNo findings. Numerical re-derivation, notebook drift guards, methodology wording pins, and label tests are comprehensive (test_t33_bad_controls_drift.py:L186, test_dml_did.py:L2815). The changelog fragment passes its validator. |
Adds docs/tutorials/33_bad_controls.ipynb, the Caetano, Callaway, Payne & Sant'Anna (2026) bad-controls walkthrough on a staggered version of the paper's DGP 1: the naive TWFE regression with the bad control at t (the paper's Estimator 1) misses by the full treatment effect on the covariate; Approach 1 through base-period covariates with and without W; the DMLDiD bad-control lane and its W choices (Remark 5's lagged outcome, none); reading bad_control_summary() (pre rows pre-test MP-5/MP-8 and should be zero, post rows check that treatment moves the covariate); the event study; a ridge refit through the split-half nested stage; the lane's restrictions. No "exclude X" arm: in the paper's linear designs every route that conditions on W coincides, so the notebook makes no claim that the lane beats a CS fit that conditions on W. - Registered in docs/tutorials/index.rst (card + toctree; description count 34), docs/tutorials/README.md, diff_diff/guides/llms.txt (lines for 32 and 33), docs/doc-deps.yaml (dml_did, dml_did_results, staggered, _learners, linalg, utils). - tests/test_t33_bad_controls_drift.py: code-cell hashes, quoted-number and pre/post ATT_X wording pins, DGP re-derivation at 5e-4, source fragment pins, |t| < 2 narrative guard on the correctly specified pre-period rows. - TODO.md: PR-C row removed; changelog fragment added. - diff_diff/practitioner.py: practitioner_next_steps() names a bad-control fit "DMLDiD (CCPS 2026 bad-control score)" (was the Chang 2020 label); test in tests/test_dml_did.py; changelog "Changed" bullet.
c909dab to
2ccf90f
Compare
|
🔁 AI review rerun (requested by @igerber) Head SHA: Overall assessment: ✅ Looks goodExecutive summary
Methodology
The Approach 1/2 assumption maps, comparison groups, DGP, and pre/post Code QualityNo findings. The per-instance practitioner label is narrowly implemented and preserves the Chang label for ordinary fits ( PerformanceNo findings. MaintainabilityNo findings. Tech Debt
SecurityNo findings; no secrets, PII, tracebacks, or local paths were found. Documentation/TestsNo findings. Tests pin notebook code, outputs, assumptions, DGP re-derivation, estimates, SEs, and practitioner labels ( |
1 participant
Summary
docs/tutorials/33_bad_controls.ipynb, "Bad Controls - Covariates That Treatment Can Affect": a staggered version of Caetano, Callaway, Payne & Sant'Anna (2026) Supplementary Appendix DGP 1 (the paper's coefficients, 4 periods, cohorts 3/4 plus never-treated,default_rng(5), n = 2000), on which the naive TWFE regression with the bad control att(the paper's Estimator 1) lands at 0.4886 against a true 1.00; Approach 1 through base-period covariates withoutW(1.0540, the paper's inconsistent "ML (Pre-treatment)" arm) and withW(0.9754); theDMLDiDbad-control lane withW=["w"](0.9766), Remark 5'sW=["y"](1.0169, labelled a deliberately misspecified sensitivity fit) and noW(1.0591);bad_control_summary()with the pre/postATT_Xreading (pre rows pre-test MP-5/MP-8 and should be zero; post rows check that treatment moves the covariate: 0.51 / 0.51 / 0.48 against 0.50); the event study; a ridge refit through the split-half nested stage (0.9770); the lane's restrictions. No "exclude X" arm (a maintainer decision: in the paper's linear designs every route that conditions onWcoincides; the paper's discard bias is stated, not demonstrated). The tutorial names the full Approach-1 / Proposition-2/3 assumption maps (MP-4 maintained; MP-8 or MP-9; MP-6 on the never-treated side with trimming as regularization; MP-7 endpoint reduction) and makes no claim that the lane beats a CS fit that conditions onW.docs/tutorials/index.rst(card + toctree, description count 34),docs/tutorials/README.md,diff_diff/guides/llms.txt(lines for 32 and 33),docs/doc-deps.yaml(dml_did, dml_did_results, staggered, _learners, linalg, utils, _crossfit, _dr_scores, practitioner).tests/test_t33_bad_controls_drift.py: code-cell hashes, quoted-number pins, the pre/postATT_Xand identification-caveat wording pins, DGP re-derivation at 5e-4 (all seven estimates, the sixATT_Xrows, the four event-study rows), source-fragment pins, and a |t| < 2 narrative guard on the correctly specified pre-period rows.practitioner_next_steps()now labels a bad-control fit "DMLDiD (CCPS 2026 bad-control score)" instead of the Chang (2020) label (diff_diff/practitioner.py; asserted intests/test_dml_did.py; changelog "Changed" bullet).TODO.md: the PR-C row is removed.Methodology references (required if estimator / math changes)
docs/methodology/papers/caetano-2026-review.md); Chang (2020).ATT_X, fail-closed lanes).Validation
tests/test_t33_bad_controls_drift.py(new),tests/test_dml_did.py(practitioner banner assertion). Notebook executed once with the per-worktree kernel and validated withpytest --nbmake;test_docs_ia,test_doc_deps_integrity,test_tracking_files,test_changelog_fragments,test_guides,test_practitionerpass; Sphinx-Wbuild passes with the new page.Security / privacy
Changelog
changelog.d/20260907-bad-controls-tutorial.md)