Skip to content

docs(methodology): Deaner & Ku (2026) causal duration DiD paper review (PR-A) - #825

Closed
igerber wants to merge 2 commits into
mainfrom
codex/eval-paper-be416efe
Closed

docs(methodology): Deaner & Ku (2026) causal duration DiD paper review (PR-A)#825
igerber wants to merge 2 commits into
mainfrom
codex/eval-paper-be416efe

Conversation

@igerber

@igerber igerber commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Summary

Adds the methodology review for Deaner & Ku, Causal Duration Analysis with Diff-in-Diff
(arXiv 2405.05220v2), as the Step-1 artifact for a prospective DurationDiD estimator for
binary absorbing-state outcomes (exit from unemployment, passing an exam, divorce, parole
revocation), where parallel trends in mean outcomes generally fails and the paper instead
imposes common-dynamics or proportional-hazards restrictions on counterfactual hazard rates.

Docs-only. No estimator, tests, dependencies, tooling pins or public API change in this PR.
The branch is built on its pinned base; pyproject.toml and .github/workflows/lint.yml
are byte-identical to that base, so a comparison against a later main may show unrelated
tooling differences (a Ruff point-release bump) that this PR does not introduce.

Files:

  • docs/methodology/papers/deaner-ku-2026-review.md (new, ~1,300 lines): source-pinned
    review with a prospective registry entry, implementation notes, a tuning-parameter table,
    a paper-versus-reference-code fidelity list, the approved first-estimator scope, and a
    numbered gaps list.
  • changelog.d/20260905-duration-did-paper-review.md (new): Documentation fragment.
  • ROADMAP.md: the stale "Causal Duration Analysis with DiD" entry (which cited a 2025
    conference paper and overstated the method as free of hazard assumptions) is removed from
    Long-term Research Directions; the estimator is added under Shipping Next with the
    approved first scope.

What the review contains

  • Setting, notation, target parameter, Assumptions 1-2, and the paper's own account of why
    mean parallel trends generally fails for absorbing outcomes (with its stated exception).
  • Identification: Eqs. 1.1-2.30, Theorems 1-3 and A.1, Remarks 1-4 (including the mapping to
    Wooldridge 2023's exponential model), the general linear-restriction framework nesting
    triple-difference and synthetic-control analogues.
  • Estimation: Eqs. 3.1-3.8 as printed, with as-printed anomalies flagged rather than
    corrected (notably the proportional-hazards constant in (3.5), for which four candidate
    estimators are enumerated and their population limits derived).
  • Inference: the individual-level block bootstrap, pointwise and uniform bands, and the
    pre-treatment specification test, with Algorithms 1 and 2 transcribed verbatim, both
    printed forms of the test statistic recorded, and a fully written-out p-value proposal
    labelled as a library extension (the paper defines none).
  • Deferred extensions and relevant proof details: covariate
    balancing (Section 2.2, Theorem 3), staggered adoption (Appendix A.3), the semiparametric
    covariate approach (Appendix A.4, Theorem 4), and the proofs (Appendix E).
  • The Appendix C simulation design with Table 1 and all fifty Table 2 cells as a future test
    oracle (a common-dynamics design; a proportional-hazards fixture is required separately),
    and the Austrian unemployment-insurance application numbers as replication targets.
  • A tuning-parameter table and edge-case handling proposals: exact-zero survival by role,
    the proportional-hazards parameter domain, zero-denominator and zero-bootstrap-SD
    policies, invalid-draw handling per bootstrap family, absorbed-time conversion contract,
    time-grid and dtype contracts, and minimum data requirements stated as counts over the
    observed grid.
  • A source-reproducibility procedure: the review header pins the arXiv v2 PDF, the six
    reference-code files at a fixed commit and the three dated web snapshots by URL and
    SHA-256, and states the fetch-and-hash commands to rebuild the local cache (which is a
    git-excluded working directory, not part of the repository).
  • Registry readiness: every library policy or extension that goes beyond the paper (the
    p-value, exact-zero-survival, zero-SD, invalid-draw and zero-denominator policies, the
    diagnostic range, the proportional-hazards domain check) carries the registry-recognised
    - **Note:** label, and the copy instruction says how to finalise the wording when the
    estimator ships.
  • Reference-code notes on the authors' Stata command and MATLAB function: duplicated file
    copies (the MATLAB file does not parse as cached), a code-only one-sided p-value, a
    Stata-versus-MATLAB discrepancy at the default burn-in, sampling-weight handling that
    differs by language, and the Monte Carlo script's conventions (per-arm sample size,
    discretised treatment period, band-based rejection column, two result workbooks).

Approved first-estimator scope (recorded in the review)

Core two-group DurationDiD with common treatment timing, both the common-dynamics and
proportional-hazards specifications, block-bootstrap inference with pointwise and uniform
bands, and pre-treatment diagnostics. Covariate adjustment, K-group linear restrictions,
Kaplan-Meier censoring input, staggered adoption and repeated-cross-section inference are
deferred to follow-on PRs. Two decisions are marked blocking for the estimator PR's plan:
the proportional-hazards constant formula and the inference construction against the
library's joint-NaN contract; the review records a recommendation and alternatives for each.

Methodology references (required if estimator / math changes)

  • Method name(s): Duration difference-in-differences (common dynamics / proportional
    hazards on counterfactual hazard rates); prospective class DurationDiD.
  • Paper / source link(s): Deaner, B., & Ku, H. (2026). Causal Duration Analysis with
    Diff-in-Diff. arXiv:2405.05220v2, https://arxiv.org/abs/2405.05220v2 (unpublished preprint;
    revise-and-resubmit at Quantitative Economics per the authors' pages). Reference code:
    https://github.com/ben-deaner-teaching/Duration-DiD at commit 202e92ef (no LICENSE
    file; read as a black-box reference only, nothing ported). The PDF, each code file and the
    status pages are SHA-256 pinned in the review header.
  • Any intentional deviations from the source (and why): none implemented (docs only). The
    review is a deliberate exception to the published-source rule, made because no shipped
    estimator handles absorbing-state outcomes; every page, equation and algorithm reference
    is pinned to arXiv v2 and the header instructs re-checking against the published version.
    Source typos and internal inconsistencies are transcribed as printed and catalogued in the
    gaps list, never silently corrected. Library-side proposals that go beyond the paper
    (the p-value, zero-survival and zero-SD policies, the scaled proportional-hazards slope
    as recommended default) are labelled as extensions or decisions, not as paper statements.

Validation

  • Tests added/updated: none (docs-only).
  • Tests actually run (from the checkout's .venv):
    • pytest tests/test_naming_guard.py tests/test_changelog_fragments.py tests/test_tracking_files.py tests/test_docs_ia.py -q - 203 passed on the final bytes.
    • An earlier full docs-lane run including tests/test_doc_deps_integrity.py and
      tests/test_v4_matrix.py - 678 passed, 23 skipped.
    • python3 .claude/scripts/changelog_compile.py check - OK.
    • python3 .claude/scripts/premerge_scan.py - no methodology-pattern findings (no Python
      changed).
    • Scans of the review for absolute paths and for naming-guard token forms - clean.
    • Cached-source integrity: find .workflow/sources -type f -exec shasum -a 256 {} +
      matches the ten inventory digests.
    • The document records source coverage, source-code line references and remaining
      transcription/figure uncertainties alongside the pinned source inventory.
  • Backtest / simulation / notebook evidence: not applicable. The authors' Monte Carlo and
    application scripts were read, not executed; statements about which script objects feed
    Table 2 are structural readings and are marked as such in the review.
  • Not run: ruff, black, mypy - no Python source changed in this PR.

Material limitations

  • The source is an unpublished preprint; numbering may shift in a published version.
  • The reference implementation is unlicensed and ships as two concatenated copies per file
    (the MATLAB file is parse-invalid as cached), so it can serve only as an equation-level
    reference; any future parity fixture needs a documented, non-distributed extraction.
  • The paper gives no analytic variance, no p-value, no K-group or staggered version of the
    specification test, and only a sketch of censoring handling. The review distinguishes
    proposed library conventions from extensions deferred to later work.
  • Publication status and the v1 date rest on the authors' web pages and the arXiv listing
    as cached on 2026-09-05.

Security / privacy

  • Confirm no secrets/PII in this PR: confirmed. Documentation only; no data, credentials or
    personal information; the unlicensed reference code is cited by commit and line, not
    copied.

Changelog

  • changelog.d/ fragment added (or N/A - no user-visible change):
    changelog.d/20260905-duration-did-paper-review.md (Documentation).

@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown

Overall assessment: ✅ Looks good

Executive summary

  • No estimator, statistical output, public API, or default behavior changes.
  • The prospective DurationDiD review clearly separates source formulas, source anomalies, and proposed library extensions.
  • P2: the claimed local source inventory is absent, limiting reproducibility.
  • P2: an unrelated Ruff downgrade contradicts the “docs-only” characterization.

Methodology

  • Severity: P2 — Missing source inventory
    Location: docs/methodology/papers/deaner-ku-2026-review.md:L28-L43
    Impact: The document claims .workflow/sources/ and .workflow/sources.json are available, but neither exists in HEAD. Exact transcriptions, hashes, and reference-code line claims cannot be reproduced from this checkout.
    Concrete fix: Add the manifest/source snapshot to the review environment, or remove the local-cache claim and provide a reproducible fetch-and-hash procedure using the pinned URLs and commit.

No implemented methodology, weighting, identification, or variance mismatch was found. The affected methods are prospective common-dynamics and proportional-hazards DurationDiD; the unresolved PH estimator and inference choices are appropriately identified as blocking future decisions.

Code Quality

  • Severity: P2 — Unrelated lint-tool downgrade
    Location: .github/workflows/lint.yml:L41-L42, pyproject.toml:L63-L68, changelog.d/20260905-duration-did-paper-review.md:L20
    Impact: Ruff changes from 0.16.5 to 0.16.4 without explanation in a PR described as docs-only. Point releases can alter lint results and CI behavior.
    Concrete fix: Revert both pin changes here, or move them to a focused tooling change documenting why 0.16.4 is required.

Performance

No findings; runtime code is unchanged.

Maintainability

  • Severity: P3 — Prospective Registry text is not fully copy-ready
    Location: docs/methodology/papers/deaner-ku-2026-review.md:L54-L58, L695-L712, L749-L760
    Impact: Several deliberate library policies/extensions lack the Registry-recognized **Note:** or **Deviation from R:** labels despite the instruction to copy this section when the estimator ships.
    Concrete fix: Apply recognized labels now or amend the copy instruction to require relabeling during the estimator PR.

Tech Debt

No current implementation debt is introduced. Deferred estimator capabilities are explicitly scoped as future work.

Security

No secrets, credentials, PII, or executable unlicensed source were added.

Documentation/Tests

The required changelog fragment is present and its format passes the changelog compiler. No estimator tests are required for this documentation-only methodology artifact.

@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown

🔁 AI review rerun (requested by @igerber)

Head SHA: 9ac0c62ea0d1646ba5d8aa86b74a05c2e6339c69


Overall assessment: ✅ Looks good

Executive summary

  • No estimator, statistical output, API, or default behavior changes.
  • Prospective common-dynamics and proportional-hazards DurationDiD methodology is clearly separated from library extensions.
  • P2: source-cache reproducibility remains incomplete.
  • P2: the unrelated Ruff downgrade remains.
  • No unmitigated P0/P1 findings.

Methodology

  • Severity: P2 — Source-cache claim remains inaccurate
    Location: docs/methodology/papers/deaner-ku-2026-review.md:L28-L57
    Impact: .workflow/ is absent and not git-ignored (git check-ignore finds no rule). The rebuild instructions also omit the three mutable web snapshots and .workflow/sources.json, so they cannot reconstruct the claimed ten-file dated inventory.
    Concrete fix: Add /.workflow/ to .gitignore and complete the reconstruction procedure for all inventory entries and manifest, or narrow the claims to the reproducible paper and commit-pinned code sources.

  • Severity: P3 — Documented methodology extensions
    Impact: The p-value, PH-domain validation, zero-survival/zero-SD policies, and invalid-draw handling are correctly marked with **Note:**; they are intentional, documented extensions.
    Concrete fix: None required.

Code Quality

  • Severity: P2 — Unrelated Ruff downgrade remains
    Location: .github/workflows/lint.yml:L41-L42, pyproject.toml:L63-L68, changelog.d/20260905-duration-did-paper-review.md:L20-L21
    Impact: Ruff is downgraded from 0.16.5 to 0.16.4 in a fragment described as docs-only, potentially changing CI lint behavior without explanation.
    Concrete fix: Restore 0.16.5, or document and isolate the tooling downgrade.

Performance

No findings; runtime code is unchanged.

Maintainability

  • Severity: P3 — One prospective Registry choice lacks a Note label
    Location: docs/methodology/papers/deaner-ku-2026-review.md:L989-L990, L1228-L1231
    Impact: The checklist chooses the reference-code correction—uniform bands over post-treatment horizons—rather than Algorithm 1’s printed range, but the copy-ready requirement is not **Note:**-labeled.
    Concrete fix: Add a **Note:** identifying this source correction before copying the entry into REGISTRY.md.

Tech Debt

No findings. Deferred estimator capabilities introduce no current implementation debt.

Security

No secrets, PII, or unlicensed source code were added.

Documentation/Tests

The required changelog fragment is present, and the changelog compiler check passes. No estimator tests are required for this documentation-only artifact.

@igerber igerber closed this Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant