docs(methodology): Caetano, Callaway, Payne & Sant'Anna (2026) bad-controls paper review (PR-A)

[igerber]

Summary

• Add docs/methodology/papers/caetano-2026-review.md: paper-sourced fidelity review of Caetano, Callaway, Payne & Sant'Anna (2026), "Difference-in-differences with 'bad controls'" (arXiv:2608.03881v2, 2 Sep 2026) together with its Supplementary Appendix (CCPS_2026_SA_v1). Both PDFs are SHA-256 pinned in the header; all equation / assumption / theorem / table numbering follows arXiv v2 and SA v1. This is PR-A of the 2-PR methodology protocol for a prospective bad-controls extension of the DiD-with-covariates family; no code changes.
• Deliberate exception to the published-source rule. This is an unpublished preprint; the exception is made for the authors' standing (Callaway, Caetano). Note that "Sant'Anna" here is Hugo Sant'Anna (UAB), not Pedro H. C. Sant'Anna; the review header records this.
• The review transcribes: the formal definition of a bad control (Conditions 1-2, Definition 1) and the identification failure (Eqs. 2-3); the bias of the include / discard conventions (Section 3, τ^use / τ^discard); Approach 1 (pre-treatment conditioning; Theorem 1, Proposition 1, Assumptions 4-5) and Approach 2 (covariate unconfoundedness; Theorem 2 / Eq. 4, Assumptions 6-7, Remarks 2-3); the staggered results (Assumptions MP-1 to MP-9, Theorem 3, Propositions 2-3, Remarks 4-6 incl. the ATT_X(g,t) pre-test); the imputation estimator (Assumption 8, Eqs. 5-7, Proposition 4, influence function S8); the Neyman-orthogonal DR / DML estimator (Eqs. 8-11, Propositions 5-7, Algorithm 1 verbatim, Assumptions 9 / S1 / S2, the SB.1-SB.3 proof structure incl. the staggered substitution rule and aggregation IF); the SC linearity alternative (Assumptions S3-S5, Proposition S1, Corollary S1); all five Monte Carlo DGPs with Tables S1-S6 in full; and the NLSY job-displacement application (Table 1, the nine estimators, Figures 6-8, footnotes 9-10).
• Records the GPL-3 status of the authors' badcontrols R package as a one-line Note: (black-box numerical oracle only; no source port - the earlier direct-port PR feat: port twfeweights/ptetools/badcontrols R packages (R-compat) #753 was declined on this basis) and a clearly labelled "Relation to Existing diff-diff Estimators" section mapping Approach 1 onto CallawaySantAnna's base-period covariates (already computable) and the DR estimator onto the DMLDiD cross-fitting stack (Equation 10 reduces to the Sant'Anna-Zhao / Chang panel score when no bad control and no W are present).
• 18 numbered items in "Gaps and Uncertainties" with page references (no explicit plug-in variance for the imputation estimator; no bootstrap, clustering, or trimming rule; ATT_X(g,t) stated identified but no estimand written; the SA Figure S1 "four main DGPs" vs five panels; unstated application control group / anticipation / SE method; binary bad control undiscussed; as-printed typos), recorded rather than silently resolved.
• Adds the changelog.d/20260905-bad-controls-paper-review.md fragment (Documentation).

Methodology references (required if estimator / math changes)

• Method name(s): prospective bad-controls extension (paper review only; working name BadControlsDiD in the review, no estimator implemented in this PR)
• Paper / source link(s): Caetano, C., Callaway, B., Payne, S., & Sant'Anna, H. (2026). Difference-in-differences with "bad controls". arXiv:2608.03881v2, https://arxiv.org/abs/2608.03881; Supplementary Appendix v1, https://bcallaway11.github.io/files/badcontrols/CCPS_2026_SA_v1.pdf
• Any intentional deviations from the source (and why): None implemented (docs-only). Conditions the paper leaves open but an implementation needs (a trimming / overlap rule, a fold count, a non-cross-fitted parametric-DR variance, an ATT_X(g,t) estimator) are labelled as implementation-required / deviation-to-be-documented inside the review, not resolved.

Validation

• Tests added/updated: No test changes (documentation-only).
• Backtest / simulation / notebook evidence (if applicable): N/A. Spot-checked against the PDF text: Equation 10, Algorithm 1, Table S2, Table 1, footnotes 9-10 match; the changelog fragment passes changelog_compile.py check; premerge_scan.py reports 0 findings.

Security / privacy

• Confirm no secrets/PII in this PR: Yes

Changelog

• changelog.d/ fragment added (or N/A - no user-visible change): Yes

[github-actions]

Overall assessment: ⚠️ Needs changes

Executive summary

• Docs-only PR; no estimator or runtime behavior changes.
• Core Eq. 4, Eq. 10, S8, and existing-estimator mappings are internally consistent.
• Two methodological claims overstate what the formal conditions establish.
• The changelog fragment passes validation; no security concerns.
• One implementation ambiguity remains around regression intercepts.

Methodology

P1 — Zero bias is not equivalent to failure of Conditions 1 or 2

Location: docs/methodology/papers/caetano-2026-review.md:L44-L48, L252-L266

Impact: The displayed bias integrals may equal zero through moment cancellation even when the relevant conditional distributions differ and outcome relevance holds. Thus Figures 2–3 give sufficient benchmark cases for zero bias, not necessary conditions. Saying estimators are biased “whenever” a bad control exists and unbiased “only if” a condition fails misstates the identification argument.

Concrete fix: Replace the categorical wording with “generally biased” and state that zero bias occurs when the corresponding displayed integral is zero. Describe failures of Conditions 1 or 2 as sufficient cases. If mirroring the paper’s prose verbatim, label it as such and note the cancellation qualification.

P1 — ATT_X is not a complete test for a bad control

Location: docs/methodology/papers/caetano-2026-review.md:L92-L102, L353-L369

Impact: Condition 2 concerns equality of entire conditional distributions, while ATT_X(g,t) tests only a conditional-average effect. A zero ATT_X can hide distributional or heterogeneous effects, and a nonzero value addresses Condition 2 only—not Condition 1’s outcome relevance. Calling it a test of whether X_t “actually” is a bad control can produce false reassurance.

Concrete fix: Call ATT_X a mean-effect diagnostic, explicitly state its one-sided evidentiary scope, and require a separate outcome-relevance assessment before describing X_t as a bad control.

No existing estimator entry in REGISTRY.md is changed, so documented-deviation mitigation does not apply.

Code Quality

No findings; no code changed.

Performance

P3 — OLS complexity omits the solve term

Location: docs/methodology/papers/caetano-2026-review.md:L1208-L1211

Impact: O(nk²) omits the O(k³) matrix factorization/inversion cost.

Concrete fix: State O(nk² + k³) per fit.

Maintainability

No findings. The review is version- and checksum-pinned.

Tech Debt

No untracked correctness debt introduced. Prospective implementation gaps are explicitly enumerated in the review.

Security

No findings. No secrets or sensitive data are introduced; the GPL compatibility warning is appropriate.

Documentation/Tests

P2 — Intercept convention is ambiguous

Location: docs/methodology/papers/caetano-2026-review.md:L429-L454, L826-L860

Impact: The stated linear models and R_i/S_i omit an intercept, while the DGPs contain constants (θ₂−θ₁=0.3 and 0.15) and classify the imputation estimator as consistent. A literal no-intercept implementation would contradict those validation targets.

Concrete fix: Clarify whether a constant is implicit in Z or added by every regression. Update R_i, S_i, the IF notation, and implementation checklist consistently; if the paper is silent, record this in “Gaps and Uncertainties.”

The required changelog.d/ fragment is present and passes changelog_compile.py check.

Path to Approval

1. Qualify the Section 3 bias claims so Conditions 1/2 failures are sufficient—not necessary—zero-bias cases.
2. Document that ATT_X is a mean-effect diagnostic and cannot by itself establish either full Condition 2 or bad-control status.

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: 46b59db1bc68356a3f83c4cd05339eb1d46494a5

---

Overall assessment: ✅ Looks good

Executive summary

• Both prior P1 findings are resolved with explicit methodological qualifications.
• Intercept and OLS-complexity concerns are also addressed.
• No estimator or runtime behavior changes.
• One minor score-family wording issue remains.
• Changelog validation passes; no security concerns.

Methodology

P2 — Chang and Sant’Anna–Zhao scores are conflated

Location: docs/methodology/papers/caetano-2026-review.md:L1297-L1304

Impact: Equation 10 reduces specifically to Chang’s unconditional-π score. The Registry documents Sant’Anna–Zhao’s self-normalized score as a distinct finite-sample estimator with different influence-function machinery. The displayed formula and DMLDiD recommendation are correct, but the combined “Sant’Anna-Zhao / Chang” label could misguide future implementation.

Concrete fix: Replace the label with “Chang panel DR score,” or clarify that Sant’Anna–Zhao is population-level equivalent while its finite-sample normalization and influence function differ.

The previous bias-cancellation and ATT_X findings are resolved at L44-L50, L92-L108, and L382-L391.

Code Quality

No findings; no code changed.

Performance

No findings. Complexity now correctly includes O(nk² + k³).

Maintainability

No findings. Sources are version- and checksum-pinned, and unresolved choices are clearly separated from paper claims.

Tech Debt

No untracked implementation debt is introduced; this PR is a prospective paper review only.

Security

No findings.

Documentation/Tests

No findings. The intercept convention is now explicit, and the required changelog fragment passes changelog_compile.py check. No tests are required for this docs-only change.

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: 750a0701573fd595c68a6349bf563d21cf185284

---

Overall assessment: ✅ Looks good

Executive summary

• The prior P2 is resolved: Chang and Sant’Anna–Zhao score families are now clearly distinguished.
• Prior bias-cancellation and ATT_X qualifications remain addressed.
• No estimator, inference, or runtime behavior changed.
• The changelog fragment follows project conventions and validates successfully.

Methodology

P3 — Prior score-family wording resolved

• Location: docs/methodology/papers/caetano-2026-review.md:L1292-L1305
• Impact: The text correctly identifies Equation 10’s reduction as Chang’s unconditional-(\pi) panel score and distinguishes Sant’Anna–Zhao’s self-normalized finite-sample score and influence function, consistent with REGISTRY.md.
• Concrete fix: None required.

Code Quality

No findings; this is documentation-only.

Performance

No findings; no runtime paths changed.

Maintainability

No findings. Paper claims, implementation requirements, and unresolved choices are clearly separated.

Tech Debt

No findings. Prospective implementation gaps are explicitly documented without introducing code debt.

Security

No findings; no secrets or executable changes were introduced.

Documentation/Tests

No findings. The required documentation changelog fragment is present and changelog_compile.py check passes. Tests are not required for this docs-only change.