diff --git a/TODO.md b/TODO.md index 18bfb90db..28de96f22 100644 --- a/TODO.md +++ b/TODO.md @@ -102,5 +102,4 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m | Clean-negative precision re-run: Campaign 1's `s3_negative` plans were base_sha-contaminated (real defects at the pinned base), so hallucination rate cleared (dual 3.4%) but trivia-flooding on a *genuinely* clean plan is unmeasured. Small future run (~30 s3-style reviews on 3-5 constructed-clean plans, not the full 120-matrix). | `tools/plan-review-eval/` | campaign-1 verdict | Mid | Low | | `trop-silent-drop` criteria regression: the rewritten `criteria.md` dropped a silent-failure catch the OLD criteria had (arm A caught it, B/C missed — campaign-1 A-vs-B contrast). Patch the criteria + re-validate (re-opens the criteria identity, so gated behind a re-validation run). | `.claude/skills/plan-review/criteria.md`, `tools/plan-review-eval/` | campaign-1 verdict | Mid | Medium | | The Stata `jwdid` parity arms cover BALANCED panels only. `jwdid_alltreated` pins the all-eventually-treated cell set, `N` and SE ratio on the `mpdta` subset (191 units, 955 rows, 764 estimated), but every arm is a balanced frame, so REGISTRY's parity claim is scoped to that case. The comparison-support predicate's UNBALANCED behavior -- where `G_max` may be unobserved at later periods so the closed form `t < G_max - anticipation` does not hold and support is set by whichever cohorts are actually observed -- is verified against the predicate directly, not against Stata. Add an unbalanced `jwdid` arm to close that gap. | `benchmarks/stata/generate_etwfe_cs_golden.do`, `tests/test_etwfe_cs_stata_parity.py` | #729-followup | Mid | Low | -| Bad-controls tutorial section for `docs/tutorials/32_dml_did.ipynb` (PR-C): the `fit(bad_control=)` workflow, the `ATT_X(g,t)` pre-test reading, and the Approach-1 comparison via CS base-period covariates. | `docs/tutorials/32_dml_did.ipynb` | bad-controls PR-B | Quick | Medium | | DurationDiD tutorial notebook (Appendix C-style simulation showing the sign reversal of standard DiD on an absorbing outcome, CD vs PH, the fitting window, and the pretest), registered in `docs/tutorials/index.rst` with a drift test; deferred from the estimator PR, which relies on the executed API-page examples | `docs/tutorials/` | #822 | Mid | Low | diff --git a/changelog.d/20260907-bad-controls-tutorial.md b/changelog.d/20260907-bad-controls-tutorial.md new file mode 100644 index 000000000..7139c3a7c --- /dev/null +++ b/changelog.d/20260907-bad-controls-tutorial.md @@ -0,0 +1,19 @@ +### Documentation +- **Tutorial 33, "Bad Controls - Covariates That Treatment Can Affect"** + (`docs/tutorials/33_bad_controls.ipynb`; PR-C of the Caetano, Callaway, Payne & + Sant'Anna 2026 bad-controls initiative). On a staggered version of the paper's DGP 1 + it shows the naive TWFE regression with the bad control at `t` missing by the full + treatment effect on the covariate, Approach 1 through base-period covariates with and + without the confounders `W`, the `DMLDiD` bad-control lane and the choice of `W` + (Remark 5's lagged outcome), how to read `bad_control_summary()` (pre-period rows + pre-test MP-5/MP-8 and should be zero; post-period rows check that treatment moves the + covariate), the event study, a ridge refit through the split-half nested stage, and + the lane's restrictions. Registered in the tutorials index, `docs/tutorials/README.md`, + `diff_diff/guides/llms.txt` (together with a line for tutorial 32) and + `docs/doc-deps.yaml`; pinned by `tests/test_t33_bad_controls_drift.py` (code-cell + hashes, quoted numbers, DGP re-derivation, pre-period narrative guard). + +### Changed +- **`practitioner_next_steps()` names the bad-control lane**: on a `DMLDiD` fit with + `bad_control` set the guidance banner reads "DMLDiD (CCPS 2026 bad-control score)" + instead of the Chang (2020) label, matching the results `summary()` banner. diff --git a/diff_diff/guides/llms.txt b/diff_diff/guides/llms.txt index 2b79c763b..a9df7dc97 100644 --- a/diff_diff/guides/llms.txt +++ b/diff_diff/guides/llms.txt @@ -119,6 +119,8 @@ The site is organized into 5 sections, each with a landing page: - [16 Wooldridge ETWFE](https://diff-diff.readthedocs.io/en/stable/tutorials/16_wooldridge_etwfe.html): Wooldridge (2023, 2025) ETWFE — saturated OLS, logit/Poisson (ASF-based ATT), aggregation types - [22 HAD Survey-Weighted Workflow](https://diff-diff.readthedocs.io/en/stable/tutorials/22_had_survey_design.html): HeterogeneousAdoptionDiD + did_had_pretest_workflow under SurveyDesign(strata, psu, weights, fpc) — BRFSS-shape panel, modest SE inflation explanation, Phase 4.5 C0 QUG-deferred verdict - [26 Composition Drift & Calibration](https://diff-diff.readthedocs.io/en/stable/tutorials/26_composition_drift_calibration.html): When differential non-response biases the DiD itself — per-state raking with Meta's balance package, `balance.interop.diff_diff` adapter, raking-granularity lesson (requires `pip install balance`) +- [32 Double ML DiD](https://diff-diff.readthedocs.io/en/stable/tutorials/32_dml_did.html): Chang (2020) DMLDiD — cross-fitted ML nuisances on a staggered panel with nonlinear confounding, learner comparison, cross-fitting diagnostics, event study + HonestDiD, the repeated-cross-section lane with a survey design +- [33 Bad Controls](https://diff-diff.readthedocs.io/en/stable/tutorials/33_bad_controls.html): Caetano, Callaway, Payne & Sant'Anna (2026) bad controls — the include-at-t bias, Approach 1 via base-period covariates (with and without W), the DMLDiD bad-control lane (`fit(bad_control=, bad_control_covariates=)`), reading `bad_control_summary()` (pre rows pre-test MP-5/MP-8, post rows check treatment moves the covariate), ridge refit via the split-half nested stage ## Survey Support diff --git a/diff_diff/practitioner.py b/diff_diff/practitioner.py index 82c17fba3..e91635857 100644 --- a/diff_diff/practitioner.py +++ b/diff_diff/practitioner.py @@ -76,8 +76,11 @@ def _estimator_display(type_name: str, results: Any) -> str: ``ChangesInChangesResults`` is shared by CiC and QDiD (``QDiDResults`` is an alias), so the static per-type map cannot distinguish them; the - ``method`` field ("cic"/"qdid") does. Defensive: mock results may - lack the field, in which case the static entry is the fallback. + ``method`` field ("cic"/"qdid") does. ``DMLDiDResults`` likewise + serves two lanes: a fit with ``bad_control`` set runs the Caetano, + Callaway, Payne & Sant'Anna (2026) bad-control score and is named for + it, the plain fit keeps the Chang (2020) label. Defensive: mock results + may lack the fields, in which case the static entry is the fallback. """ if type_name == "ChangesInChangesResults": kind = _distributional_kind(results) @@ -85,6 +88,10 @@ def _estimator_display(type_name: str, results: Any) -> str: return "ChangesInChanges (CiC)" if kind == "qdid": return "QDiD" + if type_name == "DMLDiDResults" and getattr(results, "bad_control", None) is not None: + # The bad-control lane runs the Caetano, Callaway, Payne & Sant'Anna + # (2026) score, not Chang's; the results banner says the same. + return "DMLDiD (CCPS 2026 bad-control score)" return _ESTIMATOR_NAMES.get(type_name, type_name) diff --git a/docs/doc-deps.yaml b/docs/doc-deps.yaml index 44cfb833e..eac062e9c 100644 --- a/docs/doc-deps.yaml +++ b/docs/doc-deps.yaml @@ -203,6 +203,9 @@ sources: - path: docs/tutorials/16_survey_did.ipynb type: tutorial note: "CallawaySantAnna survey examples" + - path: docs/tutorials/33_bad_controls.ipynb + type: tutorial + note: "Approach 1 (pre-treatment bad control read at the cell's base period) with and without W" - path: README.md section: "Estimators (one-line catalog entry)" type: user_guide @@ -1059,6 +1062,8 @@ sources: type: user_guide - path: docs/tutorials/32_dml_did.ipynb type: tutorial + - path: docs/tutorials/33_bad_controls.ipynb + type: tutorial diff_diff/dml_did_results.py: drift_risk: low @@ -1074,6 +1079,8 @@ sources: type: methodology - path: docs/tutorials/32_dml_did.ipynb type: tutorial + - path: docs/tutorials/33_bad_controls.ipynb + type: tutorial # ── TROP (trop group) ────────────────────────────────────────────── @@ -1387,6 +1394,9 @@ sources: - path: docs/methodology/variance-conventions.md type: methodology note: "Clustered-variance conventions inventory (CR1 k + tail-df map, D/L classification); table generated by tests/test_variance_conventions.py" + - path: docs/tutorials/33_bad_controls.ipynb + type: tutorial + note: "LinearRegression within-regression benchmark with cluster_k_adjustment (absorbed-FE CR1 convention)" - path: docs/methodology/REGISTRY.md section: "Variance Estimation, Cluster-Robust SE" type: methodology @@ -1419,6 +1429,9 @@ sources: - path: docs/methodology/variance-conventions.md type: methodology note: "Clustered-variance conventions inventory (CR1 k + tail-df map, D/L classification); table generated by tests/test_variance_conventions.py" + - path: docs/tutorials/33_bad_controls.ipynb + type: tutorial + note: "within_transform used for the TWFE benchmark" - path: docs/methodology/REGISTRY.md section: "Inference, safe_inference NaN gating" type: methodology @@ -1515,6 +1528,8 @@ sources: diff_diff/_crossfit.py: drift_risk: low docs: + - path: docs/tutorials/33_bad_controls.ipynb + type: tutorial - path: docs/methodology/REGISTRY.md section: "Cross-fitting, DR-score, and ridge infrastructure (DML)" type: methodology @@ -1532,10 +1547,14 @@ sources: note: "Duck-typed learner protocol (RegressorLearner/ClassifierLearner Protocols, validate_learner, _validate_predictions) + native learners (LinearLearner/RidgeLearner/LogitLearner/SieveLearner) wrapping linalg solvers. Contracts documented in REGISTRY: raw-X-no-intercept input, fit-reset semantics (documented limitation for stateful user learners), identified-columns prediction under rank deficiency." - path: docs/tutorials/32_dml_did.ipynb type: tutorial + - path: docs/tutorials/33_bad_controls.ipynb + type: tutorial diff_diff/_dr_scores.py: drift_risk: low docs: + - path: docs/tutorials/33_bad_controls.ipynb + type: tutorial - path: docs/methodology/REGISTRY.md section: "Cross-fitting, DR-score, and ridge infrastructure (DML)" type: methodology @@ -1599,6 +1618,8 @@ sources: diff_diff/practitioner.py: drift_risk: low docs: + - path: docs/tutorials/33_bad_controls.ipynb + type: tutorial - path: diff_diff/guides/llms-practitioner.txt type: user_guide - path: diff_diff/guides/llms-full.txt diff --git a/docs/tutorials/33_bad_controls.ipynb b/docs/tutorials/33_bad_controls.ipynb new file mode 100644 index 000000000..54d62e469 --- /dev/null +++ b/docs/tutorials/33_bad_controls.ipynb @@ -0,0 +1,1136 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "18af9f30", + "metadata": {}, + "source": [ + "# Bad Controls - Covariates That Treatment Can Affect\n", + "\n", + "A **bad control** is a covariate that treatment can change. Occupation, hours, health\n", + "insurance, firm size: anything measured after treatment starts and plausibly moved by\n", + "it. Caetano, Callaway, Payne and Sant'Anna (2026) formalize when such a covariate is\n", + "\"bad\" - it is *relevant* for the untreated outcome path (Condition 1) and treatment\n", + "*affects* it (Condition 2) - and show that the two conventions practitioners reach for\n", + "are both wrong in general:\n", + "\n", + "- **conditioning on the bad control at $t$** compares treated and untreated units at\n", + " different values of $X_t(0)$, so the estimate absorbs part of the treatment's own\n", + " effect on $X$ (their Section 3.1 `tau^use` bias);\n", + "- **dropping the bad control** ignores that its untreated path carries information\n", + " about the outcome trend (their Section 3.2 `tau^discard` bias); the paper's Table\n", + " S1 marks the \"TWFE, exclude BC\" arm inconsistent in every one of its designs. This\n", + " notebook states that result and does not run a \"drop X\" arm - it is out of scope\n", + " here, not harmless.\n", + "\n", + "The paper offers two identification routes, and both are available here:\n", + "\n", + "1. **Approach 1 - condition on the pre-treatment value.** The maintained\n", + " assumption is parallel trends conditional on the bad control's *untreated* path\n", + " (Assumption 2 / MP-4); Approach 1 adds either *simple covariate unconfoundedness*\n", + " of that path given $(X_{g-1}, Z)$ (Assumption 4 / MP-8) or *bad-control\n", + " redundancy* (Assumption 5 / MP-9), and the conclusion is that conditioning on the\n", + " pre-treatment value $X_{g-1}$ alone identifies the ATT (Theorem 1 / Proposition\n", + " 1; staggered: Proposition 3 under MP-1 to MP-4, MP-6, MP-7 and MP-8 or MP-9).\n", + " `CallawaySantAnna` and `DMLDiD` already read `covariates` at each cell's base\n", + " period on the panel lane, so passing the bad control there IS Approach 1.\n", + "2. **Covariate unconfoundedness for the bad control's path (Approach 2).** The same\n", + " maintained parallel trends (MP-4), but the bad control's untreated evolution is\n", + " now identified given its pre-treatment value, extra pre-treatment confounders\n", + " $W$ and the clean covariates $Z$ (Assumption 6 / MP-5), again with MP-7 so that\n", + " the untreated outcome change depends on the bad control's history only through\n", + " its two endpoints, $X_{\\text{base}}(0)$ and $X_t(0)$ (Proposition 2). Both\n", + " approaches allow treatment to affect $X$; what distinguishes this one is the\n", + " $W$-conditional path-unconfoundedness strategy and the nested doubly-robust\n", + " score that implements it. This is the `DMLDiD` bad-control lane:\n", + " `fit(..., bad_control=, bad_control_covariates=)`.\n", + " It also reports the paper's Remark 6 pre-test $ATT_X(g, t)$ - the effect of\n", + " treatment on the covariate itself - for every cell.\n", + "\n", + "This tutorial:\n", + "\n", + "1. Builds a staggered version of the paper's first Monte Carlo design (DGP 1).\n", + "2. Runs the naive TWFE regression with the bad control at $t$ and watches it miss by\n", + " the full treatment effect on $X$.\n", + "3. Runs Approach 1 with and without $W$.\n", + "4. Runs the bad-control lane and reads its `summary()`.\n", + "5. Compares the choices of $W$ (the paper's Remark 5 lagged outcome; none).\n", + "6. Reads `bad_control_summary()` - pre-period rows are a pre-test, post-period rows\n", + " are the Condition-2 check - and the event study.\n", + "7. Refits with a different learner (the nested stage switches to split-half) and lists\n", + " what the lane will not do." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "727429dd", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-07T18:15:19.565639Z", + "iopub.status.busy": "2026-09-07T18:15:19.565435Z", + "iopub.status.idle": "2026-09-07T18:15:20.182804Z", + "shell.execute_reply": "2026-09-07T18:15:20.182404Z" + } + }, + "outputs": [], + "source": [ + "import warnings\n", + "\n", + "import numpy as np\n", + "import pandas as pd\n", + "\n", + "from diff_diff import CallawaySantAnna, DMLDiD, LinearRegression, practitioner_next_steps\n", + "from diff_diff.utils import within_transform\n", + "\n", + "pd.set_option(\"display.precision\", 4)\n", + "\n", + "FIT_KW = dict(outcome=\"y\", unit=\"unit\", time=\"time\", first_treat=\"first_treat\")" + ] + }, + { + "cell_type": "markdown", + "id": "270a944b", + "metadata": {}, + "source": [ + "## 1. The Paper's DGP 1, Staggered\n", + "\n", + "The Supplementary Appendix's first design (SA pp. 14-15) has two periods; we keep every\n", + "coefficient and extend it to four periods with two treated cohorts ($g = 3$ and\n", + "$g = 4$) plus a never-treated pool. Per unit, with $Z_i, \\eta_i \\sim N(0, 1)$ and all\n", + "noise terms $\\varepsilon \\sim N(0, 1)$ mutually independent:\n", + "\n", + "$$\n", + "\\begin{aligned}\n", + "W_i &= 0.8\\,\\eta_i + 0.3\\,Z_i + 0.2\\,\\varepsilon^W_i, &\n", + "D_i &= \\mathbf{1}\\{0.2\\,Z_i + 0.4\\,W_i + 0.3\\,\\eta_i + \\varepsilon^D_i > 0\\}, \\\\\n", + "X_{i1}(0) &= 0.5\\,\\eta_i + 0.4\\,Z_i + 0.3\\,\\varepsilon^{X}_{i1}, &\n", + "X_{it}(0) &= 0.7\\,X_{i,t-1}(0) + 0.3\\,Z_i + 0.2\\,W_i + 0.15 + 0.3\\,\\varepsilon^{X}_{it}, \\\\\n", + "X_{it}(g) &= X_{it}(0) + \\lambda\\ \\text{for } t \\ge g, &\n", + "Y_{it}(0) &= 0.3\\,t + 0.5\\,\\eta_i + 0.3\\,Z_i + X_{it}(0) + 0.3\\,\\varepsilon^{Y}_{it}, \\\\\n", + "Y_{it}(g) &= Y_{it}(0) + \\lambda + \\delta\\ \\text{for } t \\ge g, & \\lambda &= \\delta = 0.5 .\n", + "\\end{aligned}\n", + "$$\n", + "\n", + "Treated units draw their cohort by a fair coin. The unobserved $\\eta_i$ drives\n", + "treatment, $W_i$ and the bad control's level, so the bad control's *path* carries the\n", + "confounding; $Z_i$ is a clean covariate; $W_i$ is the observed pre-treatment confounder\n", + "of $X$. Truth in every post-treatment cell: $ATT(g, t) = \\lambda + \\delta =\n", + "\\mathbf{1.00}$ and $ATT_X(g, t) = \\lambda = \\mathbf{0.50}$; both are $0$ before\n", + "treatment.\n", + "\n", + "A note on the seed. The structural assumptions - parallel trends given the bad\n", + "control's untreated path, no anticipation, and covariate unconfoundedness of that path\n", + "given $(X_{t-1}, W, Z)$ - hold by construction, so pre-period placebo estimates are\n", + "pure sampling noise, and a pre-test rejects on roughly 5% of draws even then. One\n", + "assumption does *not* hold uniformly: MP-6 requires the probability of staying\n", + "untreated given the covariate history to be bounded away from zero - equivalently,\n", + "the treatment propensity bounded away from one - and with a Gaussian latent index and\n", + "unbounded covariates no such bound exists; this design shares that feature with the\n", + "paper's own simulations. The lane's propensity trimming is numerical regularization\n", + "of the estimated scores, not a repair of MP-6 - the fits below report how often it\n", + "binds. We use `default_rng(5)`, a draw whose pre-period placebos all sit\n", + "within one standard error of zero; the first seed we tried produced a 3-SE pre-period\n", + "pseudo-ATT in one cell. Read pre-tests jointly across cells and against the size of\n", + "the post-treatment effects, never one cell at a time." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "43b5a63e", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-07T18:15:20.184406Z", + "iopub.status.busy": "2026-09-07T18:15:20.184291Z", + "iopub.status.idle": "2026-09-07T18:15:20.190248Z", + "shell.execute_reply": "2026-09-07T18:15:20.189928Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "first_treat\n", + "0 974\n", + "3 531\n", + "4 495\n", + "Name: count, dtype: int64" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "n, T = 2000, 4\n", + "rng = np.random.default_rng(5)\n", + "Z = rng.standard_normal(n)\n", + "eta = rng.standard_normal(n) # unobserved heterogeneity\n", + "W = 0.8 * eta + 0.3 * Z + 0.2 * rng.standard_normal(n)\n", + "D = (0.2 * Z + 0.4 * W + 0.3 * eta + rng.standard_normal(n)) > 0 # ever treated\n", + "g = np.where(D, rng.choice([3, 4], size=n), 0) # cohort (0 = never treated)\n", + "X0 = np.empty((n, T + 1)) # column t = X_t(0), 1-based\n", + "X0[:, 1] = 0.5 * eta + 0.4 * Z + 0.3 * rng.standard_normal(n)\n", + "for t in range(2, T + 1):\n", + " X0[:, t] = 0.7 * X0[:, t - 1] + 0.3 * Z + 0.2 * W + 0.15 + 0.3 * rng.standard_normal(n) # DGP 1\n", + "frames = []\n", + "for t in range(1, T + 1): # one outcome-noise draw per period, in period order\n", + " post = (g > 0) & (t >= g)\n", + " x_t = X0[:, t] + 0.5 * post # X_t(g) = X_t(0) + lambda\n", + " y_t = 0.3 * t + 0.5 * eta + 0.3 * Z + X0[:, t] + 0.3 * rng.standard_normal(n) + post * (0.5 + 0.5)\n", + " frames.append(\n", + " pd.DataFrame({\"unit\": np.arange(n), \"time\": t, \"first_treat\": g, \"y\": y_t, \"x\": x_t, \"z\": Z, \"w\": W})\n", + " )\n", + "df = pd.concat(frames, ignore_index=True)\n", + "df[\"post\"] = ((df[\"first_treat\"] > 0) & (df[\"time\"] >= df[\"first_treat\"])).astype(float)\n", + "\n", + "df.groupby(\"unit\")[\"first_treat\"].first().value_counts().sort_index()" + ] + }, + { + "cell_type": "markdown", + "id": "da295cf9", + "metadata": {}, + "source": [ + "**974** never-treated units against cohorts of **531** ($g = 3$) and **495** ($g = 4$).\n", + "\n", + "## 2. The Naive Regression\n", + "\n", + "The paper's first application estimator is a two-way fixed-effects regression of\n", + "$Y_{it}$ on the staggered treatment indicator $D_{it}$ *and the bad control* $X_{it}$\n", + "(\"TWFE: include BC\"). We run it as a within regression - unit and period means removed\n", + "with `within_transform`, then `LinearRegression` on the demeaned columns with unit\n", + "clusters. (The library's `TwoWayFixedEffects` is a $2 \\times 2$ estimator: with a\n", + "unit-specific `post` indicator its `treatment x post` column coincides with `post`\n", + "itself and the fit is rejected as collinear, which is why the two-line within\n", + "regression is used here.) The clustered standard error uses the library's absorbed\n", + "fixed-effects convention: `cluster_k_adjustment=4` counts the absorbed constant plus\n", + "the $T - 1 = 3$ period effects that are not nested in the unit clusters\n", + "(`docs/methodology/variance-conventions.md`, defect D2)." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "2ac8200c", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-07T18:15:20.191220Z", + "iopub.status.busy": "2026-09-07T18:15:20.191163Z", + "iopub.status.idle": "2026-09-07T18:15:20.198327Z", + "shell.execute_reply": "2026-09-07T18:15:20.197966Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
estimatorattsebias
0TWFE, include X_t0.48860.0142-0.5114
\n", + "
" + ], + "text/plain": [ + " estimator att se bias\n", + "0 TWFE, include X_t 0.4886 0.0142 -0.5114" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def twfe(cols):\n", + " # Within regression of y on [D_it, *cols] with unit and period effects removed.\n", + " d = within_transform(df, variables=[\"y\", \"post\", *cols], unit=\"unit\", time=\"time\")\n", + " X = d[[f\"{c}_demeaned\" for c in [\"post\", *cols]]].to_numpy()\n", + " y = d[\"y_demeaned\"].to_numpy()\n", + " lr = LinearRegression(include_intercept=False, cluster_ids=df[\"unit\"].to_numpy()).fit(\n", + " X, y, cluster_k_adjustment=4\n", + " )\n", + " return float(lr.coefficients_[0]), float(np.sqrt(lr.vcov_[0, 0]))\n", + "\n", + "\n", + "rows = []\n", + "att, se = twfe([\"x\"])\n", + "rows.append(dict(estimator=\"TWFE, include X_t\", att=att, se=se, bias=att - 1.0))\n", + "pd.DataFrame(rows)" + ] + }, + { + "cell_type": "markdown", + "id": "05344bfb", + "metadata": {}, + "source": [ + "**0.4886 ± 0.0142** against a true **1.00**: the regression nets out the part of the\n", + "treatment effect that runs through the covariate ($\\lambda = 0.5$), matching the\n", + "paper's Table S2 bias of $-0.501$ at $n = 2000$. Nothing about the sample size fixes\n", + "this.\n", + "\n", + "## 3. Approach 1 - Condition on the Pre-Treatment Value\n", + "\n", + "On the panel lane, `CallawaySantAnna` reads every covariate at the cell's base period\n", + "(the period before the cohort's first treatment for post-treatment cells), never at\n", + "$t$. Passing the bad control in `covariates` therefore conditions on $X_{g-1}$ - the\n", + "paper's Theorem 1 / Proposition 3 estimand. The estimator reads the supplied column at\n", + "the base row, but it cannot tell whether that value contains only information\n", + "available then: a lead, a copied post-treatment value or any other future-informed\n", + "column would silently recreate the Section 2 bias (the paper's Section 3.1 `tau^use` bias), so supplying a genuine\n", + "pre-treatment value is the user's responsibility. We use the not-yet-treated comparison group throughout,\n", + "the group the paper's staggered results are stated for.\n", + "\n", + "The assumption map for Approach 1 is worth stating precisely, because none of its\n", + "labels is a \"parallel trends given $X_{g-1}$\" assumption. Two periods (Theorem 1 and\n", + "Proposition 1): Assumptions 1-3 (absorbing treatment, no anticipation, parallel\n", + "trends conditional on the bad control's untreated path and $Z$) plus *either*\n", + "Assumption 4, simple covariate unconfoundedness $X_{t^*}(0) \\perp D \\mid X_{t^*-1},\n", + "Z$, *or* Assumption 5, bad-control redundancy. Staggered (Proposition 3): MP-1 to\n", + "MP-4, overlap MP-6, the endpoint-only reduction MP-7 - the untreated outcome change\n", + "may depend on the bad control's history only through $X_{g-1}(0)$ and $X_t(0)$, which\n", + "holds here because $Y_t(0) - Y_b(0) = 0.3(t - b) + X_t(0) - X_b(0) + \\varepsilon^Y_t\n", + "- \\varepsilon^Y_b$ - plus MP-8 or MP-9. What these deliver is that conditioning on\n", + "$X_{g-1}$ and $Z$ *alone* identifies the ATT. In this design that is\n", + "not enough: the bad control's untreated evolution depends on $W$, which is correlated\n", + "with treatment through $\\eta$, so the paper's own \"ML (Pre-treatment)\" arm is\n", + "inconsistent here (Table S1; bias $+0.042$ at $n = 2000$). Adding $W$ to the\n", + "conditioning set treats $(W, Z)$ as the expanded Approach-1 covariate set: MP-4 and\n", + "MP-8 must then hold relative to that set, they do here ($X_t(0)$ given $X_{t-1}, W, Z$\n", + "no longer depends on $\\eta$), and the estimate recovers the truth. In this linear design every route that conditions on $W$ agrees. Both approaches\n", + "allow treatment to move $X$; what the bad-control lane adds is Approach 2's\n", + "identification strategy - unconfoundedness of the bad control's untreated path given\n", + "$(X_{\\text{base}}, W, Z)$, implemented by the nested doubly-robust score - and the\n", + "$ATT_X$ surface." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "267df6f6", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-07T18:15:20.199222Z", + "iopub.status.busy": "2026-09-07T18:15:20.199159Z", + "iopub.status.idle": "2026-09-07T18:15:20.223703Z", + "shell.execute_reply": "2026-09-07T18:15:20.223425Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
estimatorattsebias
0TWFE, include X_t0.48860.0142-0.5114
1CS, Approach 1: X_{g-1}, Z1.05400.02350.0540
2CS, Approach 1: X_{g-1}, Z, W0.97540.0274-0.0246
\n", + "
" + ], + "text/plain": [ + " estimator att se bias\n", + "0 TWFE, include X_t 0.4886 0.0142 -0.5114\n", + "1 CS, Approach 1: X_{g-1}, Z 1.0540 0.0235 0.0540\n", + "2 CS, Approach 1: X_{g-1}, Z, W 0.9754 0.0274 -0.0246" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "with warnings.catch_warnings():\n", + " # (one trimmed propensity per small cell; the same warning is shown and\n", + " # explained in Section 4)\n", + " warnings.simplefilter(\"ignore\")\n", + " cs_a1 = CallawaySantAnna(control_group=\"not_yet_treated\").fit(df, **FIT_KW, covariates=[\"x\", \"z\"])\n", + " cs_a1w = CallawaySantAnna(control_group=\"not_yet_treated\").fit(\n", + " df, **FIT_KW, covariates=[\"x\", \"z\", \"w\"]\n", + " )\n", + "for label, r in ((\"CS, Approach 1: X_{g-1}, Z\", cs_a1), (\"CS, Approach 1: X_{g-1}, Z, W\", cs_a1w)):\n", + " rows.append(dict(estimator=label, att=r.att, se=r.se, bias=r.att - 1.0))\n", + "pd.DataFrame(rows)" + ] + }, + { + "cell_type": "markdown", + "id": "913482ff", + "metadata": {}, + "source": [ + "Approach 1 as the paper states it lands at **1.0540 ± 0.0235** (biased by two\n", + "standard errors); with $W$ in the conditioning set it lands at **0.9754 ± 0.0274**,\n", + "within one standard error of the truth.\n", + "\n", + "## 4. The Bad-Control Lane\n", + "\n", + "`DMLDiD` keeps its Callaway-Sant'Anna cell architecture and swaps the per-cell score\n", + "for the paper's Neyman-orthogonal doubly-robust score (their Equation 10) when a\n", + "`bad_control` is declared. The per-cell estimand is Proposition 2's: parallel trends\n", + "given the bad control's untreated path (MP-4), covariate unconfoundedness of that\n", + "path (MP-5), overlap (MP-6) and the endpoint-only reduction MP-7 that lets the\n", + "score condition on $(X_{\\text{base}}, X_t)$ instead of the full history. Four\n", + "nuisances are cross-fitted per cell:\n", + "\n", + "- the control outcome-change regression $m_0$ on $R = (X_t, X_{\\text{base}}, Z)$,\n", + "- the propensity $p$ on $S = (X_{\\text{base}}, W, Z)$,\n", + "- and two *nested* second stages fit on the training controls only: $\\nu_0$, the\n", + " first-stage predictions regressed on $S$, and $\\omega_0$, the propensity odds\n", + " regressed on $R$.\n", + "\n", + "The base period is $g - 1$ for post-treatment cells and the immediately preceding\n", + "period for the pre-treatment pseudo-cells (the library's varying-base convention).\n", + "`bad_control_covariates` is the paper's $W$; the outcome name is allowed there and\n", + "means the outcome at the base period (Remark 5). `covariates` stays required and is\n", + "the paper's $Z$; the bad control must not appear in `covariates`.\n", + "\n", + "Two kinds of warning are raised by this fit and are worth reading rather than\n", + "silencing; the cell below collects them and prints their messages. The\n", + "$\\omega_0$ stage projects propensity *odds* with a linear regression, which can go\n", + "negative; the lane clips $\\hat\\omega$ to $[0, (1 - \\text{trim}) / \\text{trim}]$ and\n", + "warns with the count per cell (the paper gives no rule; the choice is recorded as a\n", + "REGISTRY Note). The three cells with fewer than 2,000 units also trim one fitted\n", + "propensity to the `pscore_trim` bounds - the existing propensity diagnostic, which\n", + "is raised once per affected cell." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "d0f2568a", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-07T18:15:20.224886Z", + "iopub.status.busy": "2026-09-07T18:15:20.224814Z", + "iopub.status.idle": "2026-09-07T18:15:20.279874Z", + "shell.execute_reply": "2026-09-07T18:15:20.279498Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=====================================================================================\n", + " DML DiD (CCPS 2026 bad-control score) Staggered Difference-in-Differences Results \n", + "=====================================================================================\n", + "\n", + "Propensity learner: 'logit'\n", + "Outcome learner: 'linear'\n", + "Cross-fitting folds (K): 5\n", + "Seed: 0\n", + "Bad control: x\n", + "Bad-control covariates (W): w\n", + "ATT_X cells (post / pre): 3 / 3\n", + "Overall ATT weighting: CS simple (not Remark 4; REGISTRY DMLDiD Note).\n", + "Call results.bad_control_summary() for the ATT_X pre-test table.\n", + "\n", + "Total observations: 8000\n", + "Treated units: 1026\n", + "Never-treated units: 974\n", + "Treatment cohorts: 2\n", + "Time periods: 4\n", + "Control group: not_yet_treated\n", + "Base period: varying\n", + "\n", + "-------------------------------------------------------------------------------------\n", + " Overall Average Treatment Effect on the Treated \n", + "-------------------------------------------------------------------------------------\n", + "Parameter Estimate Std. Err. z-stat P>|z| Sig.\n", + "-------------------------------------------------------------------------------------\n", + "ATT 0.9766 0.0260 37.550 0.0000 ***\n", + "-------------------------------------------------------------------------------------\n", + "\n", + "95% Confidence Interval: [0.9256, 1.0276]\n", + "CV (SE/abs(ATT)): 0.0266\n", + "\n", + "Signif. codes: '***' 0.001, '**' 0.01, '*' 0.05, '.' 0.1\n", + "=====================================================================================\n", + "n_clipped_omega per cell: {(3, 2): 61, (3, 3): 48, (3, 4): 131, (4, 2): 57, (4, 3): 120, (4, 4): 127}\n", + "\n", + "9 warnings raised by the fit:\n", + " UserWarning: DMLDiD cell (g=3, t=2): 61 of 2000 nested odds-projection predictions (omega_hat) fell outside [0, 99] and were clipped (Assumption S2(iv) boundedness; the paper gives no rule - REGISTRY DMLDiD Note).\n", + " UserWarning: DMLDiD cell (g=3, t=3): 48 of 2000 nested odds-projection predictions (omega_hat) fell outside [0, 99] and were clipped (Assumption S2(iv) boundedness; the paper gives no rule - REGISTRY DMLDiD Note).\n", + " UserWarning: Propensity scores for 1 of 1505 observations (0.1%) were outside [0.01, 0.99] and will be trimmed. This may indicate near-separation in the propensity score model.\n", + " UserWarning: DMLDiD cell (g=3, t=4): 131 of 1505 nested odds-projection predictions (omega_hat) fell outside [0, 99] and were clipped (Assumption S2(iv) boundedness; the paper gives no rule - REGISTRY DMLDiD Note).\n", + " UserWarning: DMLDiD cell (g=4, t=2): 57 of 2000 nested odds-projection predictions (omega_hat) fell outside [0, 99] and were clipped (Assumption S2(iv) boundedness; the paper gives no rule - REGISTRY DMLDiD Note).\n", + " UserWarning: Propensity scores for 1 of 1469 observations (0.1%) were outside [0.01, 0.99] and will be trimmed. This may indicate near-separation in the propensity score model.\n", + " UserWarning: DMLDiD cell (g=4, t=3): 120 of 1469 nested odds-projection predictions (omega_hat) fell outside [0, 99] and were clipped (Assumption S2(iv) boundedness; the paper gives no rule - REGISTRY DMLDiD Note).\n", + " UserWarning: Propensity scores for 1 of 1469 observations (0.1%) were outside [0.01, 0.99] and will be trimmed. This may indicate near-separation in the propensity score model.\n", + " UserWarning: DMLDiD cell (g=4, t=4): 127 of 1469 nested odds-projection predictions (omega_hat) fell outside [0, 99] and were clipped (Assumption S2(iv) boundedness; the paper gives no rule - REGISTRY DMLDiD Note).\n" + ] + } + ], + "source": [ + "with warnings.catch_warnings(record=True) as caught:\n", + " warnings.simplefilter(\"always\")\n", + " res = DMLDiD(control_group=\"not_yet_treated\", seed=0).fit(\n", + " df, **FIT_KW, covariates=[\"z\"], bad_control=\"x\", bad_control_covariates=[\"w\"]\n", + " )\n", + "print(res.summary())\n", + "print(\n", + " \"n_clipped_omega per cell:\",\n", + " {(int(g_), int(t_)): d[\"n_clipped_omega\"] for (g_, t_), d in res.cross_fit_diagnostics.items()},\n", + ")\n", + "print(f\"\\n{len(caught)} warnings raised by the fit:\")\n", + "for w in caught:\n", + " print(f\" {w.category.__name__}: {w.message}\")\n", + "rows.append(dict(estimator=\"DMLDiD bad-control lane, W = [w]\", att=res.att, se=res.se, bias=res.att - 1.0))" + ] + }, + { + "cell_type": "markdown", + "id": "b49a5ce6", + "metadata": {}, + "source": [ + "The header records the lane (`CCPS 2026 bad-control score`), the bad control, $W$,\n", + "how many $ATT_X$ cells are post- and pre-treatment, and that the headline ATT uses the\n", + "Callaway-Sant'Anna simple weighting rather than the paper's Remark 4 overall. The\n", + "overall estimate is **0.9766 ± 0.0260**, within one standard error of the truth.\n", + "\n", + "## 5. Choosing $W$\n", + "\n", + "$W$ is whatever pre-treatment information makes the bad control's untreated evolution\n", + "unconfounded (MP-5). The paper's Remark 5 discusses the base-period outcome, $Y_{g-1}$,\n", + "as the natural candidate when covariate unconfoundedness is credible *given the lagged\n", + "outcome*; passing the outcome name in `bad_control_covariates` does exactly that. In\n", + "this design it is a deliberately misspecified sensitivity fit: $X_t(0)$ still depends\n", + "on the true $W$, treatment depends on $W$ and $\\eta$, and $Y_{g-1}$ is only a noisy\n", + "proxy for both, so MP-5 does **not** hold with `W=[\"y\"]` here. The estimate moves to\n", + "**1.0169 ± 0.0259**; that it stays within one standard error of the truth is a\n", + "property of this draw, not an identification diagnostic. Declaring no $W$ at all\n", + "(`bad_control_covariates=None`, the R package's default) is the analogue of Approach 1\n", + "without $W$ and lands at **1.0591 ± 0.0235**." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "f41228d6", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-07T18:15:20.280951Z", + "iopub.status.busy": "2026-09-07T18:15:20.280882Z", + "iopub.status.idle": "2026-09-07T18:15:20.381151Z", + "shell.execute_reply": "2026-09-07T18:15:20.380804Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
estimatorattsebias
0TWFE, include X_t0.48860.0142-0.5114
1CS, Approach 1: X_{g-1}, Z1.05400.02350.0540
2CS, Approach 1: X_{g-1}, Z, W0.97540.0274-0.0246
3DMLDiD bad-control lane, W = [w]0.97660.0260-0.0234
4DMLDiD bad-control lane, W = [y] (Remark 5)1.01690.02590.0169
5DMLDiD bad-control lane, no W1.05910.02350.0591
\n", + "
" + ], + "text/plain": [ + " estimator att se bias\n", + "0 TWFE, include X_t 0.4886 0.0142 -0.5114\n", + "1 CS, Approach 1: X_{g-1}, Z 1.0540 0.0235 0.0540\n", + "2 CS, Approach 1: X_{g-1}, Z, W 0.9754 0.0274 -0.0246\n", + "3 DMLDiD bad-control lane, W = [w] 0.9766 0.0260 -0.0234\n", + "4 DMLDiD bad-control lane, W = [y] (Remark 5) 1.0169 0.0259 0.0169\n", + "5 DMLDiD bad-control lane, no W 1.0591 0.0235 0.0591" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "with warnings.catch_warnings():\n", + " warnings.simplefilter(\"ignore\") # (same omega-clip / trim warnings as above)\n", + " res_y = DMLDiD(control_group=\"not_yet_treated\", seed=0).fit(\n", + " df, **FIT_KW, covariates=[\"z\"], bad_control=\"x\", bad_control_covariates=[\"y\"]\n", + " )\n", + " res_now = DMLDiD(control_group=\"not_yet_treated\", seed=0).fit(\n", + " df, **FIT_KW, covariates=[\"z\"], bad_control=\"x\"\n", + " )\n", + "rows.append(dict(estimator=\"DMLDiD bad-control lane, W = [y] (Remark 5)\", att=res_y.att, se=res_y.se, bias=res_y.att - 1.0))\n", + "rows.append(dict(estimator=\"DMLDiD bad-control lane, no W\", att=res_now.att, se=res_now.se, bias=res_now.att - 1.0))\n", + "pd.DataFrame(rows)" + ] + }, + { + "cell_type": "markdown", + "id": "55426426", + "metadata": {}, + "source": [ + "## 6. Reading `bad_control_summary()`\n", + "\n", + "Every retained cell carries $\\widehat{ATT}_X(g, t)$: an AIPW estimate of the effect of\n", + "treatment on the bad control itself, with its own analytical standard error. The\n", + "reading depends on the period:\n", + "\n", + "- **Pre-treatment rows** ($t < g$) are the paper's Remark 6 pre-test of the\n", + " identifying assumptions for the bad control's untreated path (MP-5 / MP-8). They\n", + " should be zero. A nonzero pre-period value flags a possible violation of those\n", + " assumptions - it is *not* evidence that the covariate is a bad control.\n", + "- **Post-treatment rows** ($t \\ge g$) are the Condition-2 check. A nonzero value is\n", + " evidence that treatment moves the covariate; a zero value does not establish the\n", + " converse (heterogeneous effects can cancel in the mean), and $ATT_X$ says nothing\n", + " about Condition 1.\n", + "\n", + "Here the three pre-period rows are **0.0019**, **0.0120** and **0.0061** (standard\n", + "errors 0.017-0.019) and the three post-period rows are **0.5137**, **0.5055** and\n", + "**0.4764** against the true $\\lambda = 0.50$. Remember the seed note from Section 1:\n", + "with the assumptions holding by construction, a pre-period row a few standard errors\n", + "from zero is a draw, not a diagnosis, on roughly one seed in twenty (the structural\n", + "assumptions hold by construction; only uniform overlap does not, see Section 1).\n", + "\n", + "The ATT itself aggregates exactly as on the plain lane. The event study below shows the\n", + "analytical confidence intervals; the simultaneous `cband_*` columns need\n", + "`n_bootstrap > 0` and are omitted." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "31e0b740", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-07T18:15:20.382228Z", + "iopub.status.busy": "2026-09-07T18:15:20.382166Z", + "iopub.status.idle": "2026-09-07T18:15:20.387098Z", + "shell.execute_reply": "2026-09-07T18:15:20.386758Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " group time post att_x se_x t_stat p_value conf_int_lower conf_int_upper\n", + " 3 2 False 0.0019 0.0166 0.1132 9.0990e-01 -0.0307 0.0345\n", + " 3 3 True 0.5137 0.0156 32.9304 8.0768e-238 0.4831 0.5442\n", + " 3 4 True 0.5055 0.0299 16.9082 3.9136e-64 0.4469 0.5641\n", + " 4 2 False 0.0120 0.0169 0.7131 4.7575e-01 -0.0210 0.0451\n", + " 4 3 False 0.0061 0.0190 0.3211 7.4810e-01 -0.0311 0.0433\n", + " 4 4 True 0.4764 0.0263 18.1414 1.5012e-73 0.4250 0.5279\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
event_timeattseconf_int_lowerconf_int_upper
0-20.01280.0281-0.04230.0680
1-1-0.00440.0208-0.04500.0363
200.98370.02260.93931.0281
310.96290.04120.88221.0437
\n", + "
" + ], + "text/plain": [ + " event_time att se conf_int_lower conf_int_upper\n", + "0 -2 0.0128 0.0281 -0.0423 0.0680\n", + "1 -1 -0.0044 0.0208 -0.0450 0.0363\n", + "2 0 0.9837 0.0226 0.9393 1.0281\n", + "3 1 0.9629 0.0412 0.8822 1.0437" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "print(res.bad_control_summary().to_string(index=False))\n", + "res.aggregate(\"event_study\").to_dataframe()[[\"event_time\", \"att\", \"se\", \"conf_int_lower\", \"conf_int_upper\"]]" + ] + }, + { + "cell_type": "markdown", + "id": "4c29c06b", + "metadata": {}, + "source": [ + "Event times $-2$ and $-1$ sit at **0.0128** and **-0.0044**; event times $0$ and $1$ at\n", + "**0.9837** and **0.9629**.\n", + "\n", + "## 7. Learner Sensitivity, and What the Lane Will Not Do\n", + "\n", + "Baker et al.'s step 8 is a refit under a different specification. With the parametric\n", + "built-in learners the nested second stages use the first stage's in-sample fitted\n", + "values as their targets - the paper's own plug-in for its linear working models. With\n", + "any other learner (`ridge`, `sieve`, or a user object) the lane instead splits each\n", + "training fold in half, fits the first stage on one half and the nested stage on the\n", + "other, swaps, and averages (the paper's footnote 9); `cross_fit_diagnostics` records\n", + "which branch ran as `nested_stage`.\n", + "\n", + "Restrictions worth knowing before reaching for the lane:\n", + "\n", + "- **Panel data only** (the paper's Remark 1): `panel=False` with a bad control raises.\n", + "- **`cluster=` yes, `survey_design=` no**; `anticipation=0` and the varying base\n", + " period only. Each unsupported combination raises `NotImplementedError` rather than\n", + " estimating something else.\n", + "- **The bad control must not appear in `covariates`** (the fit raises) - that is the\n", + " Section 2 bias (the paper's Section 3.1 `tau^use` bias). To run Approach 1 instead, drop `bad_control` and pass the column in\n", + " `covariates` alone.\n", + "- **$ATT_X$ is analytical only**: never bootstrapped, never aggregated.\n", + "- **The headline ATT keeps the Callaway-Sant'Anna simple weighting**, not the paper's\n", + " Remark 4 cohort-mass weighting; `summary()` says so.\n", + "- The lane is validated as a black box against the authors' R package `badcontrols`\n", + " (tolerance-based, because the R package cross-fits with its own fold draw) and\n", + " against the paper's Monte Carlo designs; the REGISTRY entry lists every convention." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "499dc03c", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-07T18:15:20.387967Z", + "iopub.status.busy": "2026-09-07T18:15:20.387904Z", + "iopub.status.idle": "2026-09-07T18:15:20.575139Z", + "shell.execute_reply": "2026-09-07T18:15:20.574754Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "nested_stage: {'split_half'}\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
estimatorattsebias
0TWFE, include X_t0.48860.0142-0.5114
1CS, Approach 1: X_{g-1}, Z1.05400.02350.0540
2CS, Approach 1: X_{g-1}, Z, W0.97540.0274-0.0246
3DMLDiD bad-control lane, W = [w]0.97660.0260-0.0234
4DMLDiD bad-control lane, W = [y] (Remark 5)1.01690.02590.0169
5DMLDiD bad-control lane, no W1.05910.02350.0591
6DMLDiD bad-control lane, W = [w], ridge0.97700.0261-0.0230
\n", + "
" + ], + "text/plain": [ + " estimator att se bias\n", + "0 TWFE, include X_t 0.4886 0.0142 -0.5114\n", + "1 CS, Approach 1: X_{g-1}, Z 1.0540 0.0235 0.0540\n", + "2 CS, Approach 1: X_{g-1}, Z, W 0.9754 0.0274 -0.0246\n", + "3 DMLDiD bad-control lane, W = [w] 0.9766 0.0260 -0.0234\n", + "4 DMLDiD bad-control lane, W = [y] (Remark 5) 1.0169 0.0259 0.0169\n", + "5 DMLDiD bad-control lane, no W 1.0591 0.0235 0.0591\n", + "6 DMLDiD bad-control lane, W = [w], ridge 0.9770 0.0261 -0.0230" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "with warnings.catch_warnings():\n", + " warnings.simplefilter(\"ignore\") # (same omega-clip / trim warnings as above)\n", + " res_ridge = DMLDiD(control_group=\"not_yet_treated\", seed=0, outcome_learner=\"ridge\").fit(\n", + " df, **FIT_KW, covariates=[\"z\"], bad_control=\"x\", bad_control_covariates=[\"w\"]\n", + " )\n", + "print(\"nested_stage:\", {d[\"nested_stage\"] for d in res_ridge.cross_fit_diagnostics.values()})\n", + "rows.append(dict(estimator=\"DMLDiD bad-control lane, W = [w], ridge\", att=res_ridge.att, se=res_ridge.se, bias=res_ridge.att - 1.0))\n", + "pd.DataFrame(rows)" + ] + }, + { + "cell_type": "markdown", + "id": "0c49011b", + "metadata": {}, + "source": [ + "The ridge refit lands at **0.9770 ± 0.0261** through the split-half branch: the\n", + "estimate is insensitive to this linear-to-ridge refit on this draw. (A materially\n", + "different learner class, and a refit of the propensity learner, would be needed\n", + "before claiming more.)\n", + "\n", + "## Summary\n", + "\n", + "| Situation | Reach for |\n", + "|-----------|-----------|\n", + "| A covariate that treatment can change, parallel trends given its untreated path (Assumption 2 / MP-4), and either simple covariate unconfoundedness (Assumption 4 / MP-8) or redundancy (Assumption 5 / MP-9) | `CallawaySantAnna` or `DMLDiD` with the column in `covariates` (read at the base period: Approach 1) |\n", + "| The same covariate, identifying its untreated path through covariate unconfoundedness given its pre-treatment value, named confounders $W$ and $Z$ (Assumption 6 / MP-5; Approach 2) | `DMLDiD(...).fit(..., bad_control=, bad_control_covariates=[...])` |\n", + "| You want to know whether treatment moved the covariate at all, or pre-test the path assumptions | `results.bad_control_summary()` (post rows / pre rows) |\n", + "| Conditioning on the covariate at $t$ | Never - that is the Section 2 bias (the paper's Section 3.1 `tau^use` bias). The panel API reads `covariates` at the base row (a future-informed column is not detectable, so keep it genuinely pre-treatment), and the bad-control lane rejects a column that is both `bad_control` and in `covariates` |\n", + "| Repeated cross-sections, survey designs, anticipation, universal base period | Not available on the bad-control lane (fails closed); see the tracked rows |\n", + "\n", + "**References**: Caetano, C., Callaway, B., Payne, S., & Sant'Anna, H. (2026).\n", + "Difference-in-differences with \"bad controls\". arXiv:2608.03881. | Chang, N.-C.\n", + "(2020). Double/debiased machine learning for difference-in-differences models. *The\n", + "Econometrics Journal*, 23(2), 177-191.\n", + "\n", + "## What next\n", + "\n", + "Fitted results know their own follow-up work: `practitioner_next_steps()` returns the\n", + "Baker, Callaway, Cunningham, Goodman-Bacon & Sant'Anna (2025) practitioner workflow\n", + "steps as runnable templates. On a bad-control fit the learner-sensitivity template\n", + "carries `bad_control=`, `bad_control_covariates=` and the comparison group forward, so\n", + "the refit targets the same estimand." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "e1669601", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-07T18:15:20.576181Z", + "iopub.status.busy": "2026-09-07T18:15:20.576108Z", + "iopub.status.idle": "2026-09-07T18:15:20.578032Z", + "shell.execute_reply": "2026-09-07T18:15:20.577688Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "============================================================\n", + "Practitioner Guidance — DMLDiD (CCPS 2026 bad-control score)\n", + "Baker et al. (2025) 8-Step Workflow\n", + "============================================================\n", + "\n", + "Recommended next steps (7 remaining):\n", + "\n", + " * [HIGH] Step 1: Define target parameter\n", + " Why: State explicitly what causal effect you are estimating (ATT, ATT(g,t), weighted/unweighted) and what policy question it answers.\n", + " >>> # What is the target parameter? ATT? Weighted or unweighted?\n", + "\n", + " * [HIGH] Step 2: State identification assumptions\n", + " Why: Name the parallel trends variant you are invoking (unconditional, conditional, PT-GT-NYT, etc.), the no-anticipation assumption, and any overlap conditions.\n", + " >>> # Which PT variant? No-anticipation? Overlap?\n", + "\n", + " * [HIGH] Step 3: Test parallel trends (event-study pre-periods)\n", + " Why: For staggered designs, inspect event-study pre-period coefficients rather than the generic check_parallel_trends() which assumes a single binary treatment with universal pre-periods. Pre-treatment ATTs should be near zero. Use CS post-fit results.aggregate('event_study') or check the estimator's event-study output directly.\n", + " >>> # Inspect pre-treatment event-study coefficients:\n", + " >>> # (available after fitting with event-study aggregation)\n", + " >>> # Pre-period effects should be near zero and insignificant.\n", + "\n", + " * [HIGH] Step 6: Run HonestDiD sensitivity analysis\n", + " Why: Bounds the treatment effect under plausible violations of (conditional) parallel trends. Aggregate the event study post-fit — no refit needed.\n", + " >>> from diff_diff import compute_honest_did\n", + " >>> # Aggregate post-fit; the container feeds HonestDiD directly:\n", + " >>> es = results.aggregate('event_study')\n", + " >>> honest = compute_honest_did(es, method='relative_magnitude', M=1.0)\n", + " >>> print(honest.summary())\n", + "\n", + " * [HIGH] Step 7: Examine group and event study effects\n", + " Why: Aggregate ATT may mask heterogeneity across cohorts or dynamic effects over time. Inspect group and event study aggregations.\n", + " >>> # Aggregate post-fit - no refit needed:\n", + " >>> print(results.aggregate('group').to_dataframe()) # Per-cohort ATTs\n", + " >>> print(results.aggregate('event_study').to_dataframe()) # Dynamic effects\n", + "\n", + " * [HIGH] Step 8: Compare with alternative estimators (CallawaySantAnna, SunAbraham, or ImputationDiD)\n", + " Why: Agreement across estimators with different assumptions strengthens conclusions. Disagreement reveals sensitivity.\n", + " >>> # Re-estimate with CallawaySantAnna, SunAbraham, or ImputationDiD and compare ATT, SE, CI\n", + " >>> # If results agree, confidence increases.\n", + " >>> # If they disagree, investigate which assumptions differ.\n", + "\n", + " * [HIGH] Step 8: Check learner sensitivity\n", + " Why: DML estimates plug in cross-fitted nuisance functions; a conclusion that survives a different outcome learner (e.g. the adaptive polynomial sieve, or a penalized ridge) is more credible than one that depends on the default linear specification.\n", + " >>> # Refit with alternative nuisance learners (carry the fit's\n", + " >>> # design forward so the comparison isolates the learner):\n", + " >>> alt = DMLDiD(outcome_learner='sieve', seed=0, control_group='not_yet_treated').fit(\n", + " >>> df, outcome=..., unit=..., time=..., first_treat=...,\n", + " >>> covariates=[...],\n", + " >>> bad_control='x',\n", + " >>> bad_control_covariates=['w'])\n", + " >>> print(alt.att, results.att) # should be close\n", + "\n", + "============================================================\n", + "\n" + ] + } + ], + "source": [ + "guidance = practitioner_next_steps(res)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.14.4" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/tutorials/README.md b/docs/tutorials/README.md index d36b8ad04..46bbc627c 100644 --- a/docs/tutorials/README.md +++ b/docs/tutorials/README.md @@ -186,6 +186,13 @@ Chang (2020) double/debiased machine learning DiD: cross-fitted ML nuisances on - The repeated-cross-section lane (`panel=False`) with a `SurveyDesign` (PSU-cohesive folds, design-df t inference) - Companion drift-test file (`tests/test_t32_dml_did_drift.py`) +### 33. Bad Controls (`33_bad_controls.ipynb`) +Caetano, Callaway, Payne & Sant'Anna (2026) "bad controls" - covariates that treatment can change - on a staggered version of the paper's DGP 1: +- The naive TWFE regression with the bad control at `t` misses by the full treatment effect on the covariate (the paper's `tau^use` bias) +- Approach 1 (the pre-treatment bad control as a base-period covariate) with and without the confounders `W` +- The `DMLDiD` bad-control lane (`fit(bad_control=, bad_control_covariates=)`), the choice of `W` (Remark 5's lagged outcome), reading `bad_control_summary()` (pre rows pre-test MP-5/MP-8, post rows check that treatment moves the covariate), a ridge refit through the split-half nested stage, and the lane's restrictions +- Companion drift-test file (`tests/test_t33_bad_controls_drift.py`) + ## Running the Notebooks 1. Install diff-diff with dependencies: diff --git a/docs/tutorials/index.rst b/docs/tutorials/index.rst index 63467834c..5e6596ee3 100644 --- a/docs/tutorials/index.rst +++ b/docs/tutorials/index.rst @@ -1,5 +1,5 @@ .. meta:: - :description: Hands-on diff-diff tutorials — 33 Jupyter notebooks covering basic 2x2 DiD, staggered adoption, synthetic DiD, power analysis, and business applications. + :description: Hands-on diff-diff tutorials — 34 Jupyter notebooks covering basic 2x2 DiD, staggered adoption, synthetic DiD, power analysis, and business applications. :keywords: DiD tutorial, difference-in-differences examples, causal inference notebooks Tutorials @@ -252,6 +252,14 @@ Modern estimators for designs the basic toolkit cannot handle. Cross-fitted ML nuisances for nonlinear confounding - watch a misspecified linear model fail where the sieve recovers the truth. + .. grid-item-card:: Bad Controls (Caetano et al.) + :link: 33_bad_controls + :link-type: doc + + Covariates that treatment can change: the include-at-t bias, + Approach 1 via base-period covariates, the DMLDiD bad-control + lane, and reading the ATT_X pre-test. + .. toctree:: :maxdepth: 1 @@ -270,6 +278,7 @@ Modern estimators for designs the basic toolkit cannot handle. Regression Discontinuity (RDD) <28_rdd_scholarship_illusion> LWDiD Rolling Transformations <31_lwdid> Double ML DiD (Chang 2020) <32_dml_did> + Bad Controls (Caetano et al.) <33_bad_controls> Study Design ------------ diff --git a/tests/test_dml_did.py b/tests/test_dml_did.py index e10bdf9af..9fa0814f5 100644 --- a/tests/test_dml_did.py +++ b/tests/test_dml_did.py @@ -2813,6 +2813,11 @@ def test_reporting_surfaces(self, bc_fitted, fitted): ) out = practitioner_next_steps(bc_fitted, verbose=False) + assert out["estimator"] == "DMLDiD (CCPS 2026 bad-control score)" + assert ( + practitioner_next_steps(fitted, verbose=False)["estimator"] + == "DMLDiD (Chang 2020 double/debiased ML)" + ) step = next(s for s in out["next_steps"] if "alt = DMLDiD(" in (s.get("code") or "")) code = step["code"] fit_part = code.split(".fit(", 1)[1] diff --git a/tests/test_t33_bad_controls_drift.py b/tests/test_t33_bad_controls_drift.py new file mode 100644 index 000000000..a117a7f15 --- /dev/null +++ b/tests/test_t33_bad_controls_drift.py @@ -0,0 +1,434 @@ +"""Drift detection for Tutorial 33 (``docs/tutorials/33_bad_controls.ipynb``). + +The tutorial narrative quotes locked, seed-specific numbers (the naive TWFE +bias, the Approach-1 fits with and without ``W``, the bad-control lane and its +``W`` variants, the six ``ATT_X(g, t)`` rows and the event study). ``pytest +--nbmake`` only checks that cells *execute*; it does not check the prose or +the committed outputs (``nbsphinx_execute = "never"`` renders the committed +outputs verbatim). Three layers here: + +1. ``assert_quotes_in_rendered`` pins the load-bearing quoted values and the + pre/post ``ATT_X`` reading against the committed rendered surface. +2. Full re-derivation: the DGP is rebuilt from the locked seed and every + quoted estimate re-checked at ``atol=5e-4``; the pre-period placebo rows of + the correctly specified fits are guarded at ``|t| < 2`` (the narrative + rests on them being unremarkable). +3. ``ALL_CODE_CELL_HASHES`` pins every code cell's normalized source, and + source-fragment pins keep the duplicated DGP below in sync with the + notebook cell it mirrors. +""" + +from __future__ import annotations + +import hashlib +import json +import warnings +from pathlib import Path + +import numpy as np +import pandas as pd +import pytest + +from diff_diff import CallawaySantAnna, DMLDiD, LinearRegression +from diff_diff.utils import within_transform + +from ._tutorial_drift import assert_quotes_in_rendered, notebook_markdown + +NB = "docs/tutorials/33_bad_controls.ipynb" + +# sha256[:16] of EVERY code cell's normalized source, in notebook order - +# the complete stale-output contract (see test_all_code_cells_hash_pinned) +ALL_CODE_CELL_HASHES = [ + "9e484fe6603ca81c", + "65244e784d2b85f8", + "b44219f7b25a0eae", + "dae3beed78d843ae", + "9b2c8d153cfa8488", + "a6d67e8cfd91946a", + "0c8cf0d2120a3941", + "2c6701475db185fe", + "9ee221cd97948c35", +] + +FIT_KW = dict(outcome="y", unit="unit", time="time", first_treat="first_treat") +CG = dict(control_group="not_yet_treated") +ATOL = 5e-4 + +# Locked numbers (seed 5). Every value is quoted in the notebook prose or +# rendered in its outputs. +TWFE_INCLUDE = (0.4886, 0.0142) +CS_A1 = (1.0540, 0.0235) +CS_A1_W = (0.9754, 0.0274) +LANE_W = (0.9766, 0.0260) +LANE_Y = (1.0169, 0.0259) +LANE_NOW = (1.0591, 0.0235) +LANE_RIDGE = (0.9770, 0.0261) +ATT_X_ROWS = { # (g, t): (att_x, se_x) + (3, 2): (0.0019, 0.0166), + (3, 3): (0.5137, 0.0156), + (3, 4): (0.5055, 0.0299), + (4, 2): (0.0120, 0.0169), + (4, 3): (0.0061, 0.0190), + (4, 4): (0.4764, 0.0263), +} +EVENT_STUDY = { + -2: (0.0128, 0.0281), + -1: (-0.0044, 0.0208), + 0: (0.9837, 0.0226), + 1: (0.9629, 0.0412), +} +COHORTS = {0: 974, 3: 531, 4: 495} + +# Fragments of the notebook's DGP cell that the mirror below reproduces. +DGP_SOURCE_FRAGMENTS = [ + "rng = np.random.default_rng(5)", + "W = 0.8 * eta + 0.3 * Z + 0.2 * rng.standard_normal(n)", + "D = (0.2 * Z + 0.4 * W + 0.3 * eta + rng.standard_normal(n)) > 0", + "g = np.where(D, rng.choice([3, 4], size=n), 0)", + "X0[:, 1] = 0.5 * eta + 0.4 * Z + 0.3 * rng.standard_normal(n)", + "X0[:, t] = 0.7 * X0[:, t - 1] + 0.3 * Z + 0.2 * W + 0.15 + 0.3 * rng.standard_normal(n)", + "x_t = X0[:, t] + 0.5 * post", + "y_t = 0.3 * t + 0.5 * eta + 0.3 * Z + X0[:, t] + 0.3 * rng.standard_normal(n) + post * (0.5 + 0.5)", +] + + +def _load_nb(): + nb_path = Path(__file__).resolve().parents[1] / NB + if not nb_path.exists(): + pytest.skip("notebook not available in this CI environment") + return json.loads(nb_path.read_text()) + + +def _norm(src: str) -> str: + return "\n".join(ln.rstrip() for ln in src.strip().splitlines()) + + +def _code_cell_hashes(): + hashes = [] + for c in _load_nb()["cells"]: + if c["cell_type"] != "code": + continue + hashes.append(hashlib.sha256(_norm("".join(c["source"])).encode()).hexdigest()[:16]) + return hashes + + +def make_panel() -> pd.DataFrame: + """Mirror of the notebook's DGP cell (staggered variant of the paper's DGP 1).""" + n, T = 2000, 4 + rng = np.random.default_rng(5) + Z = rng.standard_normal(n) + eta = rng.standard_normal(n) + W = 0.8 * eta + 0.3 * Z + 0.2 * rng.standard_normal(n) + D = (0.2 * Z + 0.4 * W + 0.3 * eta + rng.standard_normal(n)) > 0 + g = np.where(D, rng.choice([3, 4], size=n), 0) + X0 = np.empty((n, T + 1)) + X0[:, 1] = 0.5 * eta + 0.4 * Z + 0.3 * rng.standard_normal(n) + for t in range(2, T + 1): + X0[:, t] = 0.7 * X0[:, t - 1] + 0.3 * Z + 0.2 * W + 0.15 + 0.3 * rng.standard_normal(n) + frames = [] + for t in range(1, T + 1): + post = (g > 0) & (t >= g) + x_t = X0[:, t] + 0.5 * post + y_t = ( + 0.3 * t + + 0.5 * eta + + 0.3 * Z + + X0[:, t] + + 0.3 * rng.standard_normal(n) + + post * (0.5 + 0.5) + ) + frames.append( + pd.DataFrame( + { + "unit": np.arange(n), + "time": t, + "first_treat": g, + "y": y_t, + "x": x_t, + "z": Z, + "w": W, + } + ) + ) + df = pd.concat(frames, ignore_index=True) + df["post"] = ((df["first_treat"] > 0) & (df["time"] >= df["first_treat"])).astype(float) + return df + + +def _twfe(df, cols): + d = within_transform(df, variables=["y", "post", *cols], unit="unit", time="time") + X = d[[f"{c}_demeaned" for c in ["post", *cols]]].to_numpy() + y = d["y_demeaned"].to_numpy() + lr = LinearRegression(include_intercept=False, cluster_ids=df["unit"].to_numpy()).fit( + X, y, cluster_k_adjustment=4 + ) + return float(lr.coefficients_[0]), float(np.sqrt(lr.vcov_[0, 0])) + + +def _lane(df, **kw): + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + return DMLDiD(seed=0, **CG, **kw.pop("ctor", {})).fit( + df, **FIT_KW, covariates=["z"], bad_control="x", **kw + ) + + +@pytest.fixture(scope="module") +def panel_df(): + return make_panel() + + +@pytest.fixture(scope="module") +def lane_fit(panel_df): + return _lane(panel_df, bad_control_covariates=["w"]) + + +class TestRenderedSurface: + def test_comparison_table_quotes(self): + assert_quotes_in_rendered( + NB, + ["0.4886", "1.0540", "0.9754", "0.9766", "1.0169", "1.0591", "0.9770"], + surface="output", + ) + md = notebook_markdown(NB) + for quoted in ( + "**0.4886 ± 0.0142**", + "**1.0540 ± 0.0235**", + "**0.9754 ± 0.0274**", + "**0.9766 ± 0.0260**", + "**1.0169 ± 0.0259**", + "**1.0591 ± 0.0235**", + "**0.9770 ± 0.0261**", + "**974**", + "**531**", + "**495**", + ): + assert quoted in md, f"prose quote missing: {quoted}" + + def test_att_x_and_event_study_quotes(self): + assert_quotes_in_rendered( + NB, + [ + "0.0019", + "0.0120", + "0.0061", + "0.5137", + "0.5055", + "0.4764", + "0.9837", + "0.9629", + "-0.0044", + "0.0128", + ], + surface="output", + ) + md = notebook_markdown(NB) + for quoted in ( + "**0.0019**", + "**0.0120**", + "**0.0061**", + "**0.5137**", + "**0.5055**", + "**0.4764**", + ): + assert quoted in md, f"prose quote missing: {quoted}" + + def test_pre_post_att_x_reading(self): + """The Remark 6 reading (PR-B CI-review correction): pre rows pre-test + the identifying assumptions and should be zero; post rows check that + treatment moves the covariate; a zero does not prove the converse.""" + md = " ".join(notebook_markdown(NB).split()) # whitespace-normalized prose + assert "They should be zero" in md + assert "possible violation" in md + assert "does not establish the converse" in md + assert "*not* evidence that the covariate is a bad control" in md + + def test_restrictions_and_warnings_wording(self): + md = notebook_markdown(NB) + assert "must not appear in `covariates`" in md + assert_quotes_in_rendered( + NB, + [ + "were clipped", + "will be trimmed", + "nested_stage: {'split_half'}", + "warnings raised by the fit", + ], + surface="output", + ) + assert "no exclude arm" in md or 'does not run a "drop X" arm' in md + assert "seed" in md.lower() + # The overlap caveat (MP-6 has no uniform lower bound under Gaussian + # tails) must stay next to the "hold by construction" claim. + flat = " ".join(md.split()) + assert "does *not* hold uniformly" in flat + # MP-6 is the never-treated side (propensity bounded away from ONE), and + # trimming is regularization, not identification (round-6 review). + assert "bounded away from zero" in flat and "bounded away from one" in flat + assert "not a repair of MP-6" in flat + assert "no fixed positive lower bound" not in flat + # Approach 1 is not immune to future-informed columns. + assert "impossible by construction" not in flat + assert "user's responsibility" in flat + # learner-sensitivity claim scoped to this refit and draw + assert "insensitive to this linear-to-ridge refit on this draw" in flat + assert "does not depend on the outcome learner" not in flat + assert "every identifying assumption holds" not in flat.lower() + + def test_identification_caveats_present(self): + """The two identification caveats the CI review asked for: MP-7 is named + for both staggered routes (endpoint-only history), and the W=[y] fit is + labelled misspecified with its closeness to the truth disclaimed.""" + flat = " ".join(notebook_markdown(NB).split()) + # Approach 1's assumption map (Theorem 1 / Proposition 1 / Proposition 3): + # the maintained conditional parallel trends is Assumption 2 / MP-4; + # Assumptions 4 / 5 (MP-8 / MP-9) are unconfoundedness / redundancy. + assert "Assumption 2 / MP-4" in flat + assert "Assumption 4 / MP-8" in flat and "Assumption 5 / MP-9" in flat + assert "MP-1 to MP-4, MP-6, MP-7 and MP-8 or MP-9" in flat + assert "Assumptions 1-3" in flat + assert "expanded Approach-1 covariate set" in flat + # the phrase survives only inside the disclaimer that it is NOT a label + assert flat.count("parallel trends given $X_{g-1}$") == 1 + assert 'is a "parallel trends given $X_{g-1}$" assumption' in flat + assert flat.count("MP-7") >= 3 + assert "only through" in flat and "two endpoints" in flat + assert 'MP-5 does **not** hold with `W=["y"]` here' in flat + assert "not an identification diagnostic" in flat + assert "deliberately misspecified sensitivity fit" in flat + assert "practical default" not in flat + # Both approaches permit treatment to affect X; Approach 2 is distinguished + # by W-conditional path unconfoundedness + the nested score (round-5 review). + assert "Both approaches allow treatment to" in flat + assert "omitted-confounder demonstration" not in flat + assert "out of scope here, not harmless" in flat + + def test_source_cells_match_rederived_dgp(self): + nb = _load_nb() + dgp_cells = [ + c + for c in nb["cells"] + if c["cell_type"] == "code" and "rng = np.random.default_rng(5)" in "".join(c["source"]) + ] + assert len(dgp_cells) == 1 + dgp_src = _norm("".join(dgp_cells[0]["source"])) + for frag in DGP_SOURCE_FRAGMENTS: + assert frag in dgp_src, f"DGP cell drifted from the test mirror: {frag}" + + def test_all_code_cells_hash_pinned(self): + cells = _code_cell_hashes() + assert cells == ALL_CODE_CELL_HASHES, ( + "notebook code cells changed - re-execute the notebook and re-lock " + f"ALL_CODE_CELL_HASHES plus any affected rederivation constants. Got: {cells}" + ) + + def test_hash_guard_detects_mutation(self): + nb = _load_nb() + src = next( + "".join(c["source"]) + for c in nb["cells"] + if c["cell_type"] == "code" and 'bad_control_covariates=["w"]' in "".join(c["source"]) + ) + mutated = src.replace('bad_control_covariates=["w"]', 'bad_control_covariates=["y"]', 1) + h = lambda x: hashlib.sha256(_norm(x).encode()).hexdigest()[:16] # noqa: E731 + assert h(src) != h(mutated) + assert h(src) in ALL_CODE_CELL_HASHES + + def test_practitioner_banner_names_the_lane(self): + assert_quotes_in_rendered( + NB, ["Practitioner Guidance — DMLDiD (CCPS 2026 bad-control score)"], surface="output" + ) + from ._tutorial_drift import notebook_output_text + + assert "DMLDiD (Chang 2020 double/debiased ML)" not in notebook_output_text(NB) + + def test_paper_reference_present(self): + md = notebook_markdown(NB) + assert "arXiv:2608.03881" in md + assert "Caetano" in md and "Sant'Anna" in md + + def test_notebook_hygiene(self): + nb = _load_nb() + assert nb["metadata"]["kernelspec"]["name"] == "python3" + errors = [ + out + for cell in nb["cells"] + if cell["cell_type"] == "code" + for out in cell.get("outputs", []) + if out.get("output_type") == "error" + ] + assert not errors + + +class TestRederivation: + def test_cohort_split(self, panel_df): + counts = panel_df.groupby("unit")["first_treat"].first().value_counts().to_dict() + assert {int(k): int(v) for k, v in counts.items()} == COHORTS + + def test_twfe_include_bias(self, panel_df): + att, se = _twfe(panel_df, ["x"]) + np.testing.assert_allclose([att, se], TWFE_INCLUDE, atol=ATOL) + assert att < 0.6 # the paper's -0.5 bias + + def test_approach_one(self, panel_df): + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + a1 = CallawaySantAnna(**CG).fit(panel_df, **FIT_KW, covariates=["x", "z"]) + a1w = CallawaySantAnna(**CG).fit(panel_df, **FIT_KW, covariates=["x", "z", "w"]) + np.testing.assert_allclose([a1.att, a1.se], CS_A1, atol=ATOL) + np.testing.assert_allclose([a1w.att, a1w.se], CS_A1_W, atol=ATOL) + assert abs(a1.att - 1.0) > 2 * a1.se # biased without W + assert abs(a1w.att - 1.0) < 2 * a1w.se + # narrative guard: the correctly specified lane's pre rows are unremarkable + for (g, t), cell in a1w.group_time_effects.items(): + if t < g and cell.get("skip_reason") is None and np.isfinite(cell["se"]): + assert abs(cell["effect"] / cell["se"]) < 2, (g, t, cell) + + def test_bad_control_lane(self, panel_df, lane_fit): + res = lane_fit + np.testing.assert_allclose([res.att, res.se], LANE_W, atol=ATOL) + assert res.bad_control == "x" and res.bad_control_covariates == ("w",) + for (g, t), d in res.cross_fit_diagnostics.items(): + assert d.get("skip_reason") is None + assert d["n_clipped_omega"] > 0 + assert d["nested_stage"] == "in_sample" + for (g, t), cell in res.group_time_effects.items(): + if t < g: + assert abs(cell["effect"] / cell["se"]) < 2, (g, t, cell) + + def test_w_variants(self, panel_df): + res_y = _lane(panel_df, bad_control_covariates=["y"]) + res_now = _lane(panel_df) + np.testing.assert_allclose([res_y.att, res_y.se], LANE_Y, atol=ATOL) + np.testing.assert_allclose([res_now.att, res_now.se], LANE_NOW, atol=ATOL) + assert abs(res_now.att - 1.0) > 2 * res_now.se # no-W mirrors Approach 1 without W + + def test_ridge_refit_split_half(self, panel_df): + res_ridge = _lane( + panel_df, bad_control_covariates=["w"], ctor=dict(outcome_learner="ridge") + ) + np.testing.assert_allclose([res_ridge.att, res_ridge.se], LANE_RIDGE, atol=ATOL) + assert {d["nested_stage"] for d in res_ridge.cross_fit_diagnostics.values()} == { + "split_half" + } + + def test_att_x_rows(self, lane_fit): + tab = lane_fit.bad_control_summary() + assert len(tab) == 6 + assert {(int(r.group), int(r.time)) for r in tab.itertuples()} == set(ATT_X_ROWS) + for _, row in tab.iterrows(): + key = (int(row["group"]), int(row["time"])) + np.testing.assert_allclose([row["att_x"], row["se_x"]], ATT_X_ROWS[key], atol=ATOL) + if row["post"]: + assert abs(row["att_x"] - 0.5) < 3 * row["se_x"] + else: + assert abs(row["att_x"] / row["se_x"]) < 2 # narrative guard + + def test_event_study_rows(self, lane_fit): + es = lane_fit.aggregate("event_study").to_dataframe() + assert set(es["event_time"].astype(int)) == set(EVENT_STUDY) + for _, row in es.iterrows(): + e = int(row["event_time"]) + np.testing.assert_allclose([row["att"], row["se"]], EVENT_STUDY[e], atol=ATOL) + if e < 0: + assert abs(row["att"] / row["se"]) < 2 # narrative guard