From 3c82a57dcd49fafd5209c7cbd9c170136658246c Mon Sep 17 00:00:00 2001 From: igerber Date: Sat, 15 Aug 2026 17:29:45 -0400 Subject: [PATCH 1/2] feat(reports): DiagnosticReport consumes the post-fit aggregate('event_study') surface On a modern plain fit (no fit-time aggregate=), DiagnosticReport now derives the EventStudyResults container internally - once, cached, fail-soft - so the event-study-gated checks run without the deprecated kwarg the 4.0 release removes: parallel_trends / pretrends_power / sensitivity on CallawaySantAnna (the container is M-093-admitted into the consumers with pinned raw-route parity), parallel_trends (pretrends=True fits) and heterogeneity on ImputationDiD / TwoStageDiD, and heterogeneity on ContinuousDiD. Raw-field precedence is absolute (a present event_study_effects - the requested-but-empty {} sentinel included - is never re-derived, preserving fit-time balance_e semantics); the M-093 consumer-admission set, dCDH's placebo branch, and the EventStudyResults input rejection are unchanged. Derivation failures fail closed to explicit per-check skip reasons (never masked by other availability legs), remediation strings are estimator- accurate (Wooldridge's in-place aggregate; the staggered-DDD canonical fit-time route; MPD's omitted-reference case), and derived-route sections carry pre_period_source provenance - attached exactly to the checks whose gate consulted the derived surface, on success, error, and skip paths alike. Warnings emitted by a kit recompute are captured exception-safely and republished on the consuming sections and (deduplicated) the top-level channel. BusinessReport lifts the provenance key through its whitelist. Docs: report API pages migrate to plain-fit examples; the applicable_checks laziness guarantee is amended (one cached derivation may run there); REPORTING.md documents the new data source and schema key; REGISTRY.md gains the M-024-style plain-fit notes for CS/Imputation/TwoStage/Continuous; the M-020/M-021/M-022/M-025 ledger rows gain notes+code_refs; migration-4.0.md notes the checks survive the kwarg migration. Retires the TODO.md row; a pre-existing runner-skip bookkeeping gap is recorded as a new TODO row. --- CHANGELOG.md | 33 ++ TODO.md | 2 +- diff_diff/_reporting_helpers.py | 15 +- diff_diff/business_report.py | 24 +- diff_diff/diagnostic_report.py | 680 +++++++++++++++++++++++++----- diff_diff/guides/llms-full.txt | 14 +- docs/api/business_report.rst | 5 +- docs/api/diagnostic_report.rst | 11 +- docs/methodology/REGISTRY.md | 7 +- docs/methodology/REPORTING.md | 72 +++- docs/migration-4.0.md | 12 + docs/v4-deprecations.yaml | 16 +- tests/test_business_report.py | 54 +++ tests/test_diagnostic_report.py | 718 ++++++++++++++++++++++++++++++++ 14 files changed, 1507 insertions(+), 156 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 19d3e2a13..f5c99a601 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,6 +50,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 identification). ### Changed +- **DiagnosticReport's event-study-gated checks now consume the post-fit + `results.aggregate('event_study')` surface** (the 3.9 M-020 family; + retires the TODO "diagnostic_report ES-gated checks" row): on a modern + plain fit whose raw `event_study_effects` field is absent, the report + derives the `EventStudyResults` container internally — once, cached, + fail-soft — so `parallel_trends`, `pretrends_power`, `sensitivity` + (CallawaySantAnna; the container is M-093-admitted into + `compute_pretrends_power`/`HonestDiD` with pinned raw-route parity) and + `heterogeneity` (ImputationDiD / TwoStageDiD / ContinuousDiD, whose + plain-fit heterogeneity previously skipped) run without the deprecated + fit-time kwarg. Derived-route sections carry the additive + `pre_period_source="aggregate_event_study"` schema key (BusinessReport + lifts it into its `pre_trends` block), and warnings re-emitted by a kit + recompute are captured and re-published on the consuming section. + Derivation failures fail closed to explicit per-check skip reasons — a + bootstrapped fit's `NotImplementedError`, a kit-less unpickle's + `ValueError` — and the remediation strings no longer steer users to the + dying kwarg (Wooldridge points at its own in-place + `results.aggregate(type='event_study')`; the staggered-DDD reason names + the fit-time kwarg as that surface's canonical route per M-140; a + one-pre-period MultiPeriodDiD fit gets an accurate + reference-period message). Deliberately KEPT: raw-field precedence (a + present `event_study_effects` — the requested-but-empty `{}` sentinel + included — is authoritative and never re-derived, preserving fit-time + `balance_e` semantics), dCDH's `placebo_event_study` branch, the + Spillover/Stacked estimator-accurate messages, the `EventStudyResults` + INPUT rejection at DR/BR construction (admission is its own TODO row), + the M-093 consumer-admission set (CS only on the derived route), and no + auto-call of Wooldridge's mutating `aggregate()`. `applicable_checks` + may now trigger the one cached derivation; the docstring/RST laziness + guarantee is amended accordingly. The report API pages + (`docs/api/diagnostic_report.rst`, `business_report.rst`) migrate their + examples off the fit-time kwarg. - **Narrative docs migrated off the deprecated fit-time `aggregate=`** (the 3.9 M-020 family; TODO "fit-time aggregate= teachings" sweep, RST half): `choosing_estimator.rst`, `python_comparison.rst` and `r_comparison.rst` now diff --git a/TODO.md b/TODO.md index 744476378..9630b1594 100644 --- a/TODO.md +++ b/TODO.md @@ -28,6 +28,7 @@ Related tracking surfaces: | `ContinuousDiD.pscore_trim` still validates `0.0 <= x < 0.5`, i.e. it admits `0`, while `TripleDifference` tightened to `0 < x < 0.5` in phase 3(b) (row M-142) on the grounds that `trim=0` disables the `np.clip(pscore, trim, 1-trim)` overlap guard keeping the `1/(1-p)` weights finite. The same argument applies to ContinuousDiD; aligning it was out of scope for a DDD merge and is recorded in the REGISTRY staggered-mode Note rather than left as silent drift. `TripleDifference` additionally gained a TYPE guard in 3(b) (reject bool/non-real-scalar/non-finite BEFORE the range comparison) because a bare `0 < x < 0.5` raises an incidental `TypeError` on `None`/str/complex/list, an ambiguous-truth error on a multi-element array, and silently ACCEPTS a 1-element array as the parameter; `ContinuousDiD`'s `np.isfinite(self.pscore_trim) and ...` has the same hole. Aligning both is one change - promote the guard to a shared `utils.validate_pscore_trim(value, *, allow_zero)` alongside `validate_n_bootstrap` rather than copying it | `diff_diff/continuous_did.py`, `diff_diff/utils.py` | 3(b) | Quick | Low | | Staggered-mode cluster-robust ANALYTICAL SEs: `cluster=` raises in `TripleDifference`'s staggered mode (and is accepted-then-ignored on the deprecated class), so clustered inference there is bootstrap-only. Implementing a clustered analytical path for the GMM-combined influence function would let the raise become a real lane | `diff_diff/_staggered_triple_diff_engine.py` | 3(b) | Heavy | Low | | diagnostic_report admission for `EventStudyResults` surfaces (the TWFE event-study mode + `aggregate('event_study')` containers): DiagnosticReport/BusinessReport now REJECT the surface explicitly (Phase 3(a); previously a silent zero-check report / all-null headline) and practitioner_next_steps serves the generic fall-through - admission needs source-aware routing (the type-name-keyed `_APPLICABILITY`/`_HANDLERS` registries cannot discriminate the unified container's producers) and a scalar-vs-per-period headline design; MPD-native results received {parallel_trends, pretrends_power, sensitivity, bacon, design_effect} | `diff_diff/diagnostic_report.py`, `diff_diff/business_report.py`, `diff_diff/practitioner.py` | 3(a) | Mid | Medium | +| DiagnosticReport public skip bookkeeping omits RUNNER-level skips: `applicable_checks` reflects only gate outcomes, so a check whose runner returns `status="skipped"` (heterogeneity's empty/non-finite branches, `_pt_event_study`'s empty-coefs branch - a pre-existing pattern, now also reachable via a failed post-fit event-study derivation on bootstrapped / kit-less ImputationDiD/TwoStageDiD/ContinuousDiD fits) stays listed as applicable while `skipped_checks`/`schema["skipped"]` omit it, so automation reading the public fields can misclassify the check as completed; reconcile runner-returned skipped sections into the public bookkeeping (all checks, one convention) or resolve those availabilities at the gate | `diff_diff/diagnostic_report.py` | derived-ES review | Mid | Low | | Align the NATIVE pretrends pre_periods= contract with the container routes' fail-closed validation: `_extract_pre_period_params`'s MPD branch silently filters an explicit `pre_periods=` selection (unknown labels, the reference, unusable-inference rows dropped without error, caller order preserved) while both container routes validate every requested label and enforce calendar chronology (the relative route since M-024; the calendar route since 3(a) R8) - the relative route also silently collapses DUPLICATE requested labels (the calendar route rejects them), and both filter on SE only while the calendar route additionally requires a finite EFFECT (3(a) R9); one contract across all three routes, with pinned rejection messages | `diff_diff/pretrends.py` | 3(a) R8 | Quick | Low | | `EventStudyResults` inference-provenance fields: the container records no `vcov_type`/`cluster_name`/`n_clusters`/`df_convention`/Conley metadata, so a serialized surface cannot distinguish unit auto-clustering from explicit clustering, survey, Conley, or the one-way carve-out (3(a) R9 review). Adding them is a cross-producer M-092 schema amendment (six builders, to_dict/summary rendering, surface-suite pins) - follow the pre-cut amendment convention (optional fields appended last, ledger note same-diff) rather than bolting onto one producer | `diff_diff/results_base.py` | 3(a) R9 | Mid | Low | | Opt-in singleton-group pruning for TwoWayFixedEffects (static + event-study mode; reghdfe parity): singleton units/periods are currently RETAINED class-wide - the within-demeaned row is zero so points are unchanged, but N/G/residual-df count it and CR1/finite-sample SEs shift (~0.41019 -> 0.40962 measured; REGISTRY "Deviation from R" Note, R5 review) - reghdfe iteratively drops singletons by default while fixest retains them (diff-diff matches fixest); an opt-in knob needs iterative unit+period pruning with consistent cluster/survey/replicate/Conley array subsetting and a default-flip decision protocol (moves published SEs) | `diff_diff/twfe.py`, `diff_diff/estimators.py`, `diff_diff/utils.py` | 3(a) R5 | Mid | Low | @@ -35,7 +36,6 @@ Related tracking surfaces: | EfficientDiD `aggregate()` recompute levels (event_study/group) on bootstrapped fits fail closed ('simple' relays since the M-027 per-level convergence); wiring `BootstrapReplaySpec` (or retaining the n_bootstrap x n_gt draw matrix materialized at fit) would enable exact post-fit replay of percentile inference | `diff_diff/efficient_did_results.py`, `diff_diff/aggregation.py` | 2(b) PR-3a | Mid | Low | | ImputationDiD/TwoStageDiD `aggregate()` recompute levels on bootstrapped fits fail closed ('simple' relays since the M-027 per-level convergence; M-021/M-022); ImputationDiD's per-target psi machinery makes seeded replay tractable (the panel-backed kit retains everything the psi precompute reads), TwoStageDiD's per-level GMM scores are function-locals and would need retention | `diff_diff/imputation_results.py`, `diff_diff/two_stage_results.py`, `diff_diff/aggregation.py` | 2(b) PR-3b | Mid | Low | | ContinuousDiD `aggregate('event_study')` on bootstrapped fits fails closed (M-025); a seeded post-fit bootstrap-ES replay is tractable - the multiplier draws are seeded (`np.random.default_rng(self.seed)`) - but needs the FULL per-cell `_bootstrap_info` (bread/ee_treated/Psi_eval/dPsi_*/beta_pred) the pruned kit deliberately drops, so shipping it means a kit-payload change with its own memory contract | `diff_diff/continuous_did_aggregation.py`, `diff_diff/continuous_did_results.py` | 2(b) PR-3c | Mid | Low | -| diagnostic_report's ES-gated checks read the raw `event_study_effects` field, which post-fit `results.aggregate()` never populates - their remediation strings steer users to the deprecated fit-time kwarg (qualified "deprecated but functional until 4.0" since 2(b) PR-3b); teach the checks to consume a post-fit container (or recompute via the kit) before 4.0 removes the kwarg. The report API pages deliberately keep their fit-time examples until this lands (post-fit-aggregated results would produce empty ES read-outs) | `diff_diff/diagnostic_report.py`, `docs/api/business_report.rst`, `docs/api/diagnostic_report.rst` | 2(b) PR-3b | Mid | Medium | | EfficientDiD, ImputationDiD, ContinuousDiD and HeterogeneousAdoptionDiD are the outstanding M-092 event-study df-provenance holes: the container's per-row df is all-NaN even on survey fits where a finite `_survey_df` governed the p-values (the container-level scalar `df_survey` IS exposed - the hole is the PER-ROW column only; no event_study_df/df_inference field; pre-existing, NOT a regression of the M-023 PR - today's builder output is identical). The kits now retain the scalar (ImputationDiD's since 2(b) PR-3b, ContinuousDiD's since 2(b) PR-3c - same shape: scalar `df_survey` exposed, per-row column all-NaN, identical to each fit-time surface); threading it into the per-row channel is a contained follow-up | `diff_diff/efficient_did_results.py`, `diff_diff/imputation_results.py`, `diff_diff/continuous_did_results.py`, `diff_diff/results_base.py` | 2(b) PR-3a | Quick | Low | | practitioner `step_name="heterogeneity"` producer-side collisions: three OTHER estimators' advice steps reuse the key with non-heterogeneity labels (`:975` ContinuousDiD dose-response, `:1022` Triple placebo-group, `:1413` LPDiD WAS arrays), so DiagnosticReport's heterogeneity completion silently drops that unrelated advice from `next_steps` via `_filter_steps` - the same latent collision fixed for StackedDiD in M-024 (renamed to `sub_experiment_balance`). Renaming these changes those estimators' report output; audit + rename with per-estimator pins. | `diff_diff/practitioner.py` | 2(b) PR-2 review R9 | Quick | Low | | PreTrendsPower `violation='linear'` on CS `base_period='varying'` input targets the wrong alternative: `δ_pre = M · \|t\|` assumes level coefficients against a common reference, but varying-base pre-treatment effects are consecutive-period comparisons (constant increments under a linear trend). Both CS-sourced routes now WARN (REGISTRY PreTrendsPower Note), and universal-base GAPPED grids fail closed via the `reference_event_times` common-reference guard; what remains is the varying-base resolution - either transforming the violation vector through each coefficient's actual base mapping (needs per-horizon base provenance) or requiring `base_period='universal'` for the linear benchmark - a per-estimator methodology decision with a hand-calculated linear-violation gate | `diff_diff/pretrends.py` | 2(b) PR-1 R5 | Mid | Medium | diff --git a/diff_diff/_reporting_helpers.py b/diff_diff/_reporting_helpers.py index 647c1cf0d..4e8703de5 100644 --- a/diff_diff/_reporting_helpers.py +++ b/diff_diff/_reporting_helpers.py @@ -57,9 +57,10 @@ def describe_target_parameter(results: Any) -> Dict[str, Any]: the horizon / group target. - ``CallawaySantAnna``: ``overall_att`` is cohort-size-weighted across post-treatment ``ATT(g, t)`` cells regardless of the - fit-time ``aggregate`` kwarg. The event-study / group - aggregations live on dedicated fields - (``event_study_effects`` / ``group_effects``). + fit-time ``aggregate`` kwarg. The event-study / group tables are + produced post-fit via ``results.aggregate('event_study'/'group')`` + (the deprecated fit-time ``aggregate=`` kwarg populates the legacy + ``event_study_effects`` / ``group_effects`` fields until 4.0). - ``ContinuousDiD``: the regime (PT vs. SPT) is a user-level assumption, not a library setting. The ``definition`` names both regime readings (``ATT^loc`` under PT, @@ -137,9 +138,11 @@ def describe_target_parameter(results: Any) -> Dict[str, Any]: "A cohort-size-weighted average of group-time ATTs " "``ATT(g, t)`` across post-treatment cells (``t >= g``). " "``overall_att`` is the simple-aggregation headline regardless " - "of the fit-time ``aggregate`` kwarg; event-study and group " - "aggregations populate ``event_study_effects`` / " - "``group_effects`` fields when requested." + "of aggregation choices; event-study and group tables are " + "produced post-fit via " + "``results.aggregate('event_study'/'group')`` (the deprecated " + "fit-time ``aggregate=`` kwarg populates the legacy " + "``event_study_effects`` / ``group_effects`` fields until 4.0)." ), "aggregation": "simple", "headline_attribute": "overall_att", diff --git a/diff_diff/business_report.py b/diff_diff/business_report.py index bb40e6024..012d4a4bd 100644 --- a/diff_diff/business_report.py +++ b/diff_diff/business_report.py @@ -13,9 +13,13 @@ - Plain English, not academic jargon. The library ships this in addition to, not in place of, the estimator's existing ``results.summary()`` academic output. -- No estimator fitting and no variance re-derivation. Every effect, SE, p-value, - CI, and sensitivity bound is either read from ``results`` or produced by an - existing diff-diff utility. The report layer does compose a few cross-period +- No estimator fitting. Every effect, SE, p-value, CI, and sensitivity bound + is either read from ``results``, derived by the auto-constructed + ``DiagnosticReport`` from the result's own post-fit + ``aggregate('event_study')`` surface (a view or retained-kit recompute, + used only when the raw ``event_study_effects`` field is absent; see the + ``DiagnosticReport`` module docstring), or produced by an existing + diff-diff utility. The report layer does compose a few cross-period summaries from per-period inputs already on the result (joint-Wald / Bonferroni pre-trends p-value, MDV-to-ATT ratio, heterogeneity dispersion over post-treatment effects); see ``docs/methodology/REPORTING.md`` for the full @@ -351,8 +355,10 @@ def __init__( "precomputed= contains keys that are not implemented: " f"{sorted(_br_unsupported)}. Supported keys: " f"{sorted(_br_supported_precomputed)}. ``design_effect``, " - "``heterogeneity``, and ``epv`` are read directly from the " - "fitted result and do not accept precomputed overrides." + "``heterogeneity``, and ``epv`` are read from the fitted " + "result (heterogeneity may also derive the post-fit " + "aggregate('event_study') surface) and do not accept " + "precomputed overrides." ) resolved_alpha = alpha if alpha is not None else getattr(results, "alpha", 0.05) @@ -956,6 +962,14 @@ def _lift_pre_trends(dr: Optional[Dict[str, Any]]) -> Dict[str, Any]: # ``verdict == "inconclusive"`` per ``_pt_event_study``'s # inconclusive branch (``diagnostic_report.py:999``). "n_dropped_undefined": pt.get("n_dropped_undefined"), + # Provenance of the pre-period surface: "aggregate_event_study" + # when DR derived it from the post-fit + # ``results.aggregate('event_study')`` container. BR always emits + # the key — ``None`` on raw-field routes (DR itself omits the key + # there; ``dict.get`` maps that to None). Lifted explicitly for + # the same reason as ``n_dropped_undefined`` — this function is a + # field whitelist. + "pre_period_source": pt.get("pre_period_source"), "reason": pt.get("reason"), # Carry the denominator df through when the survey F-reference # branch was used so BR consumers can flag the finite-sample diff --git a/diff_diff/diagnostic_report.py b/diff_diff/diagnostic_report.py index e757b5813..be3ae7674 100644 --- a/diff_diff/diagnostic_report.py +++ b/diff_diff/diagnostic_report.py @@ -10,17 +10,24 @@ - No hard pass/fail gates. Severity is conveyed by natural-language phrasing, not a traffic-light enum. See ``docs/methodology/REPORTING.md``. -- No estimator fitting and no variance re-derivation from raw data. Every - effect, SE, p-value, CI, and sensitivity bound is either read from - ``results`` or produced by an existing diff-diff utility. May call +- No estimator fitting. Every effect, SE, p-value, CI, and sensitivity + bound is either read from ``results``, derived from the result's own + post-fit ``aggregate('event_study')`` surface (a view or retained-kit + recompute — for ImputationDiD a panel-backed recompute, for TwoStageDiD + a fresh Stage-2 OLS + GMM sandwich over the retained frame; used only + when the raw ``event_study_effects`` field is absent, and failing + closed to an explicit skip on bootstrapped / kit-less fits), or + produced by an existing diff-diff utility. May call ``check_parallel_trends`` / ``BaconDecomposition`` / ``EfficientDiD.hausman_pretest`` when the caller supplies the panel + column kwargs. Report-layer cross-period aggregations (joint-Wald / Bonferroni pre-trends p-value, heterogeneity dispersion over post-treatment effects) are enumerated in ``docs/methodology/REPORTING.md``. -- Lazy evaluation. ``DiagnosticReport(results, ...)`` is free; ``run_all()`` - triggers compute and caches. +- Lazy evaluation. ``DiagnosticReport(results, ...)`` is free; accessing + ``applicable_checks`` may derive the post-fit event-study surface once + (a view or kit recompute, cached for the report's lifetime); + ``run_all()`` triggers the full check computation and caches. - Never prove a null. Pre-trends phrasing uses power information from ``compute_pretrends_power`` to distinguish well-powered from underpowered non-violations. @@ -252,6 +259,16 @@ "SyntheticControlResults": "scm_fit", } +# Producers whose DERIVED post-fit ``aggregate('event_study')`` container may +# be handed to the external consumers (``compute_pretrends_power`` / +# ``HonestDiD.sensitivity_analysis``). Ledger row M-093 admits containers +# from CS + Stacked (relative route) + TWFE (calendar route) only; Stacked +# never reaches the derived route (its raw event-study surface is always +# populated since M-024) and TWFE is not a DiagnosticReport input, so CS is +# the whole set. Widening admission is a per-estimator methodology decision +# (M-093), never a report-layer convenience. +_DERIVED_SURFACE_CONSUMER_SOURCES: FrozenSet[str] = frozenset({"CallawaySantAnnaResults"}) + @dataclass(frozen=True) class DiagnosticReportResults(Diagnostic): @@ -368,8 +385,10 @@ class DiagnosticReport: - ``"bacon"`` — a ``BaconDecompositionResults`` object. Other sections (``design_effect``, ``heterogeneity``, ``epv``) are - read directly from the fitted result object and do not currently - accept precomputed values — there is no expensive call to bypass. + read from the fitted result object and do not currently accept + precomputed values (``heterogeneity`` may additionally derive the + post-fit ``aggregate('event_study')`` surface when the raw fields + are absent; ``design_effect`` / ``epv`` are pure reads). ``placebo`` is reserved in the schema but opt-in / deferred in MVP for the generic battery; ``SyntheticControl`` surfaces its in-space placebo under ``estimator_native_diagnostics`` (run ``results.in_space_placebo()``). @@ -487,9 +506,24 @@ def __init__( "precomputed= contains keys that are not implemented: " f"{sorted(_unsupported)}. Supported keys: " f"{sorted(_supported_precomputed)}. ``design_effect``, " - "``heterogeneity``, and ``epv`` are read directly from the " - "fitted result and do not accept precomputed overrides." + "``heterogeneity``, and ``epv`` are read from the fitted " + "result (heterogeneity may also derive the post-fit " + "aggregate('event_study') surface) and do not accept " + "precomputed overrides." ) + # Post-fit event-study derivation cache (see + # _resolve_event_study_surface): a triple + # (surface, surface_dict, why) once resolved, plus the bare + # exception string and any warnings captured during derivation. + self._derived_es_cache: Optional[ + Tuple[Optional[Any], Optional[Dict[Any, Dict[str, float]]], Optional[str]] + ] = None + self._derived_es_failure: Optional[str] = None + self._derived_es_warnings: List[str] = [] + # Which checks' GATES consulted the resolver — a user-opt-out or + # precomputed skip never consults it, and must not carry derived + # provenance/warnings it played no part in. + self._derived_es_consulted: set = set() # Estimator-aware precomputed validation. SDiD / TROP route # robustness to ``estimator_native_diagnostics`` (SDiD: weighted @@ -611,9 +645,13 @@ def to_dataframe(self) -> pd.DataFrame: def applicable_checks(self) -> Tuple[str, ...]: """Names of checks that will run, given estimator + instance + options. - No compute is triggered; this reflects only the applicability matrix - filtered by instance state (survey_metadata, epv_diagnostics, vcov) - and the user's ``run_*`` flags. + Reflects the applicability matrix filtered by instance state + (survey_metadata, epv_diagnostics, vcov) and the user's ``run_*`` + flags. When the fit's raw ``event_study_effects`` field is absent, + accessing this property may derive the post-fit + ``aggregate('event_study')`` surface once (a view or kit + recompute, cached for the report's lifetime); ``run_all()`` + triggers the full check computation. """ return tuple(sorted(self._compute_applicable_checks()[0])) @@ -701,6 +739,128 @@ def _compute_applicable_checks(self) -> Tuple[set, Dict[str, str]]: return applicable, skipped + def _resolve_event_study_surface( + self, + ) -> Tuple[Optional[Any], Optional[Dict[Any, Dict[str, float]]], Optional[str]]: + """Resolve the post-fit event-study surface for this fit, once. + + Returns the cached triple ``(surface, surface_dict, why)``: + + - ``surface`` — the ``EventStudyResults`` container from + ``results.aggregate('event_study')``, or ``None``; + - ``surface_dict`` — the container adapted to the legacy + ``event_study_effects`` dict shape (``_surface_to_event_study_dict``), + converted eagerly here so adapter errors degrade to a skip reason + instead of escaping through the unguarded gate call sites; + - ``why`` — a plain-English reason no surface is available, or + ``None`` (both ``None`` means the raw field is present and + authoritative). + + The raw ``event_study_effects`` field always takes precedence when it + is not ``None`` — including the requested-but-empty ``{}`` sentinel, + which encodes fit-time configuration (e.g. an EfficientDiD + ``balance_e=`` that emptied the window) that a re-derivation without + those settings would silently overturn. Derivation failures + (bootstrapped fits, missing kits) are cached as skip reasons, never + raised; warnings emitted by a kit recompute are captured into + ``self._derived_es_warnings`` and re-published by the consuming + runners (record-and-republish, mirroring the sensitivity helper). + """ + if self._derived_es_cache is not None: + return self._derived_es_cache + import warnings as _warnings + + r = self._results + name = type(r).__name__ + prefix = "No pre-period event-study coefficients are exposed on this fit" + surface: Optional[Any] = None + surface_dict: Optional[Dict[Any, Dict[str, float]]] = None + why: Optional[str] = None + if getattr(r, "event_study_effects", None) is not None: + # Raw precedence, ``is not None`` deliberately: ``{}`` is the + # REQUESTED-but-no-estimable-horizons sentinel and must never be + # silently re-derived (the fit-time balance_e/aggregate settings + # behind it are not retained for replay). + pass + elif name == "WooldridgeDiDResults": + # Its bespoke aggregate() MUTATES the results object in place; + # the report must never auto-call it on the user's object. + why = ( + f"{prefix}. Wooldridge event-study output is produced by " + "results.aggregate(type='event_study'), which computes and " + "stores the per-event-time surface on the results object in " + "place; call it on the results object, then re-run the " + "report." + ) + elif name == "StaggeredTripleDiffResults": + # Fit-time aggregate= is the CANONICAL route here (ledger row + # M-140, not deprecated); the container gains post-fit + # aggregate() with the M-014 unification at 4.0. + why = ( + f"{prefix}. Re-fit with fit(..., aggregate='event_study') on " + "TripleDifference's staggered mode to populate the " + "per-event-time output - the fit-time kwarg is the canonical " + "event-study route for staggered triple-difference (its " + "results container gains post-fit aggregate() at 4.0)." + ) + elif "event_study" not in getattr(type(r), "_AGGREGATE_SUPPORTED", ()): + why = ( + f"{prefix}, and {name} does not support post-fit " + "results.aggregate('event_study'), so the report cannot " + "derive the surface." + ) + else: + from diff_diff.results_base import EventStudyResults as _ESR + + caught: List[Any] = [] + try: + with _warnings.catch_warnings(record=True) as caught: + # record=True alone installs no filter: without the + # explicit simplefilter an ambient warnings-as-errors + # filter would convert the kits' fit-time re-emissions + # into spurious derivation failures. + _warnings.simplefilter("always") + candidate = r.aggregate("event_study") + if not isinstance(candidate, _ESR): + # Recorded as a derivation failure so downstream + # consumers (heterogeneity context, warning + # republication on gate-skips) treat this branch + # exactly like a raised exception. + self._derived_es_failure = ( + f"TypeError: results.aggregate('event_study') " + f"returned {type(candidate).__name__}" + ) + why = ( + f"{prefix}, and results.aggregate('event_study') " + f"returned an unexpected {type(candidate).__name__} " + "instead of an EventStudyResults container, so the " + "report cannot derive the surface." + ) + else: + surface = candidate + surface_dict = _surface_to_event_study_dict(candidate) + except Exception as exc: # noqa: BLE001 — fail-soft by design: + # expected failures are NotImplementedError (bootstrap + # gates, pretrends+replicate) and ValueError (missing kit), + # but the surface builder can raise bare TypeError and this + # resolver runs on the applicable_checks path with no outer + # guard; an escaped exception would hard-fail the report. + self._derived_es_failure = f"{type(exc).__name__}: {exc}" + surface = None + surface_dict = None + why = ( + f"{prefix}, and the event-study surface could not be " + "derived via results.aggregate('event_study'): " + f"{self._derived_es_failure}" + ) + # Persist captured warnings on BOTH the success and the failure + # path (a warning followed by an exception must not be + # discarded); the schema assembly republishes them even when + # every derived-route consumer errors or skips. + self._derived_es_warnings = [str(w.message) for w in caught] + self._derived_es_cache = (surface, surface_dict, why) + return self._derived_es_cache + def _instance_skip_reason(self, check: str) -> Optional[str]: """Return a plain-English reason this check cannot run on this instance, or None.""" r = self._results @@ -807,13 +967,42 @@ def _instance_skip_reason(self, check: str) -> Optional[str]: "pre-periods. Re-fit with kappa_pre >= 2 so " "pre-treatment event-study coefficients exist." ) + # ``pre_period_effects`` producers (MultiPeriodDiD): a + # valid one-pre-period fit has an EMPTY dict here because + # its sole pre-period is the omitted reference — that is + # not a missing post-fit aggregation route, so it must + # not fall through to the resolver's taxonomy. + if getattr(r, "pre_period_effects", None) is not None: + return ( + "No estimated pre-period coefficients exist on " + "this fit: every pre-treatment period is the " + "omitted reference. Re-fit with more " + "pre-treatment periods to enable pre-trend " + "checks." + ) + self._derived_es_consulted.add("parallel_trends") + surface, surface_dict, why = self._resolve_event_study_surface() + if surface is not None: + pre_coefs, n_dropped_undefined = _collect_pre_period_coefs( + r, surface_dict=surface_dict + ) + if pre_coefs or n_dropped_undefined: + return None # runner consumes the derived surface + return ( + "No estimated pre-treatment horizons exist on the " + "event-study surface derived via " + "results.aggregate('event_study'): the event " + "window has no pre-periods. Widen the pre-event " + "window at fit time to enable pre-trend checks." + ) + if why is not None: + return why + # Raw field present (incl. the requested-but-empty ``{}`` + # sentinel) but no estimated pre-treatment horizons. return ( - "No pre-period event-study coefficients are exposed on " - "this fit. For staggered estimators, re-fit with " - "aggregate='event_study' to populate event-study " - "output (deprecated but functional until 4.0 - these " - "checks read the raw event_study_effects field, which " - "post-fit results.aggregate() does not populate)." + "No pre-period event-study coefficients are " + "available: the fit's event-study surface has no " + "estimated pre-treatment horizons." ) # vcov is optional for the Bonferroni fallback. if method == "hausman": @@ -867,17 +1056,33 @@ def _instance_skip_reason(self, check: str) -> Optional[str]: has_vcov = getattr(r, "vcov", None) is not None has_event_vcov = getattr(r, "event_study_vcov", None) is not None has_event_es = getattr(r, "event_study_effects", None) is not None - if not (has_vcov or has_event_vcov or has_event_es): + self._derived_es_consulted.add("pretrends_power") + surface, surface_dict, why = self._resolve_event_study_surface() + if surface is not None and surface.source not in _DERIVED_SURFACE_CONSUMER_SOURCES: + # Fail closed on a non-admitted producer (M-093). Unreachable + # under the current applicability set (MPD/CS/SA); locals + # only — never mutate the cache. + surface = None + surface_dict = None + if not (has_vcov or has_event_vcov or has_event_es or surface is not None): return ( - "Pre-trends power needs either results.vcov or " - "event_study_effects (from aggregate='event_study' on " - "staggered estimators - deprecated but functional until " - "4.0; these checks read the raw field, which post-fit " - "results.aggregate() does not populate); neither " - "available." - ) - pre_coefs, _ = _collect_pre_period_coefs(r) + "Pre-trends power needs a pre-period event-study " + "surface; this fit exposes neither results.vcov, " + "event_study_vcov, nor event_study_effects, and one " + "could not be derived. " + (why or "") + ).rstrip() + pre_coefs, _ = _collect_pre_period_coefs(r, surface_dict=surface_dict) if len(pre_coefs) < 2: + # A recorded derivation failure is the real cause when the + # raw field is absent — another availability leg (vcov) + # must not mask the actionable remediation with a generic + # coefficient-count reason. + if ( + getattr(r, "event_study_effects", None) is None + and self._derived_es_failure is not None + and why is not None + ): + return why return "Pre-trends power needs >= 2 pre-treatment periods." return None if check == "sensitivity": @@ -942,13 +1147,28 @@ def _instance_skip_reason(self, check: str) -> Optional[str]: has_vcov = getattr(r, "vcov", None) is not None has_event_vcov = getattr(r, "event_study_vcov", None) is not None has_event_es = getattr(r, "event_study_effects", None) is not None - if not (has_vcov or has_event_vcov or has_event_es): + self._derived_es_consulted.add("sensitivity") + surface, surface_dict, why = self._resolve_event_study_surface() + if surface is not None and surface.source not in _DERIVED_SURFACE_CONSUMER_SOURCES: + # Fail closed on a non-admitted producer (M-093); locals only. + surface = None + surface_dict = None + if not (has_vcov or has_event_vcov or has_event_es or surface is not None): return ( - "HonestDiD needs either results.vcov, event_study_vcov, " - "or event_study_effects; none available." - ) - pre_coefs, _ = _collect_pre_period_coefs(r) + "HonestDiD needs a pre-period event-study surface " + "(results.vcov, event_study_vcov, or " + "event_study_effects); none is available and one could " + "not be derived. " + (why or "") + ).rstrip() + pre_coefs, _ = _collect_pre_period_coefs(r, surface_dict=surface_dict) if len(pre_coefs) < 1: + # Same masking guard as the pretrends_power gate above. + if ( + getattr(r, "event_study_effects", None) is None + and self._derived_es_failure is not None + and why is not None + ): + return why return "HonestDiD requires at least one pre-period coefficient." return None if check == "bacon": @@ -1040,6 +1260,17 @@ def _execute(self) -> DiagnosticReportResults: """Run the diagnostic battery and assemble the schema.""" applicable, skipped = self._compute_applicable_checks() + # Checks whose gate consults the derived event-study surface: a + # gate-skipped section still carries the derived-route provenance + # (a SUCCESSFUL derivation that turned out empty/too small must + # stay schema-distinguishable from a raw route) and any captured + # derivation warnings — the record-and-republish promise in + # REPORTING.md covers the consuming section, not just the + # top-level channel. A never-resolved cache (or a raw-route fit, + # where the resolver parks at rung 1) attaches nothing. + _es_gated = {"parallel_trends", "pretrends_power", "sensitivity", "heterogeneity"} + _derived_surface = self._derived_es_cache[0] if self._derived_es_cache is not None else None + # Initialize all schema sections to either "ran"/"skipped"/"not_applicable". sections: Dict[str, Dict[str, Any]] = {} for check in _CHECK_NAMES: @@ -1047,6 +1278,16 @@ def _execute(self) -> DiagnosticReportResults: sections[check] = {"status": "not_run", "reason": "pending implementation"} elif check in skipped: sections[check] = {"status": "skipped", "reason": skipped[check]} + # Only checks whose GATE actually consulted the resolver + # carry the derived provenance/warnings — a user-opt-out or + # precomputed skip played no part in the derivation. + if check in _es_gated and check in self._derived_es_consulted: + if _derived_surface is not None: + sections[check]["pre_period_source"] = "aggregate_event_study" + if ( + _derived_surface is not None or self._derived_es_failure is not None + ) and self._derived_es_warnings: + sections[check]["warnings"] = list(self._derived_es_warnings) else: sections[check] = { "status": "not_applicable", @@ -1157,7 +1398,22 @@ def _execute(self) -> DiagnosticReportResults: for msg in section_warnings: if msg is None: continue + # A derivation warning is copied onto EVERY consuming + # section; the top-level channel keeps one copy (the + # first consuming section's prefix), not one per + # section. Section-local copies are untouched. + if msg in self._derived_es_warnings and any( + w.endswith(f": {msg}") for w in top_warnings + ): + continue top_warnings.append(f"{check}: {msg}") + # Warnings captured while deriving the post-fit event-study surface + # are re-published even when every derived-route consumer errored or + # skipped (record-and-republish; a section-level copy above may + # already carry them — do not duplicate). + for msg in self._derived_es_warnings: + if not any(msg in w for w in top_warnings): + top_warnings.append(f"derived event-study surface: {msg}") # Some sections (e.g., sensitivity skipped for varying-base CS) # also surface methodology-critical context via ``reason`` even # though ``status != "error"``. We do not duplicate those here @@ -1366,7 +1622,26 @@ def _pt_event_study(self) -> Dict[str, Any]: ImputationDiD style, dict of dicts with ``effect``/``se``/``p_value`` keys). """ r = self._results - pre_coefs, n_dropped_undefined = _collect_pre_period_coefs(r) + surface, surface_dict, _why = self._resolve_event_study_surface() + # ``derived`` is True only when the raw field is absent and the + # post-fit surface substituted for it (rung 1 of the resolver + # guarantees ``surface is None`` whenever the raw field is present). + derived = surface is not None + + def _mark_derived(section: Dict[str, Any]) -> Dict[str, Any]: + # Derived-route provenance: every return of this runner carries + # the source key when the surface was derived, so an + # inconclusive/skip on the derived route stays + # schema-distinguishable from a raw-route one; captured + # derivation warnings are re-published (record-and-republish). + if derived: + section["pre_period_source"] = "aggregate_event_study" + if self._derived_es_warnings: + existing = list(section.get("warnings") or []) + section["warnings"] = existing + list(self._derived_es_warnings) + return section + + pre_coefs, n_dropped_undefined = _collect_pre_period_coefs(r, surface_dict=surface_dict) # Round-33 P0 / Round-42 P1 CI review on PR #318: undefined- # inference rows must drive an explicit ``inconclusive`` PT # result rather than either (a) silently shrinking the @@ -1381,32 +1656,36 @@ def _pt_event_study(self) -> Dict[str, Any]: # quote it and stakeholders see an explicit "PT could not be # assessed" warning rather than a silent PT-absent narrative. if n_dropped_undefined > 0: - return { - "status": "ran", - "method": "inconclusive", - "joint_p_value": None, - "test_statistic": None, - "df": len(pre_coefs), - "n_pre_periods": len(pre_coefs), - "n_dropped_undefined": n_dropped_undefined, - "verdict": "inconclusive", - "reason": ( - f"{n_dropped_undefined} pre-period coefficient(s) " - "have undefined inference (non-finite effect / SE or " - "SE <= 0). Per the safe-inference contract " - "(``utils.py`` line 175, REGISTRY.md line 197), this " - "yields NaN downstream; the joint PT test is " - "inconclusive on this fit. Re-fit with a different " - "variance method (bootstrap / cluster) if the " - "affected rows are a small number of cohorts, or " - "investigate why the per-period SE collapsed." - ), - } + return _mark_derived( + { + "status": "ran", + "method": "inconclusive", + "joint_p_value": None, + "test_statistic": None, + "df": len(pre_coefs), + "n_pre_periods": len(pre_coefs), + "n_dropped_undefined": n_dropped_undefined, + "verdict": "inconclusive", + "reason": ( + f"{n_dropped_undefined} pre-period coefficient(s) " + "have undefined inference (non-finite effect / SE or " + "SE <= 0). Per the safe-inference contract " + "(``utils.py`` line 175, REGISTRY.md line 197), this " + "yields NaN downstream; the joint PT test is " + "inconclusive on this fit. Re-fit with a different " + "variance method (bootstrap / cluster) if the " + "affected rows are a small number of cohorts, or " + "investigate why the per-period SE collapsed." + ), + } + ) if not pre_coefs: - return { - "status": "skipped", - "reason": "No pre-period event-study coefficients available.", - } + return _mark_derived( + { + "status": "skipped", + "reason": "No pre-period event-study coefficients available.", + } + ) interaction_indices = getattr(r, "interaction_indices", None) vcov = getattr(r, "vcov", None) @@ -1457,27 +1736,29 @@ def _pt_event_study(self) -> Dict[str, Any]: 1 for (_, _, _, p) in pre_coefs if not (isinstance(p, (int, float)) and np.isfinite(p)) ) if _n_nonfinite_p > 0: - return { - "status": "ran", - "method": "inconclusive", - "joint_p_value": None, - "test_statistic": None, - "df": len(pre_coefs), - "n_pre_periods": len(pre_coefs), - "n_dropped_undefined": _n_nonfinite_p, - "per_period": per_period, - "verdict": "inconclusive", - "reason": ( - f"{_n_nonfinite_p} retained pre-period coefficient(s) " - "have non-finite per-period p-value: the source " - "estimator's inference failed closed (e.g. hc2_bm " - "Bell-McCaffrey DOF unavailable, or collapsed " - "replicate-survey df). A joint Wald over the persisted " - "covariance would silently convert that undefined " - "inference into a finite verdict; the PT test is " - "inconclusive on this fit." - ), - } + return _mark_derived( + { + "status": "ran", + "method": "inconclusive", + "joint_p_value": None, + "test_statistic": None, + "df": len(pre_coefs), + "n_pre_periods": len(pre_coefs), + "n_dropped_undefined": _n_nonfinite_p, + "per_period": per_period, + "verdict": "inconclusive", + "reason": ( + f"{_n_nonfinite_p} retained pre-period coefficient(s) " + "have non-finite per-period p-value: the source " + "estimator's inference failed closed (e.g. hc2_bm " + "Bell-McCaffrey DOF unavailable, or collapsed " + "replicate-survey df). A joint Wald over the persisted " + "covariance would silently convert that undefined " + "inference into a finite verdict; the PT test is " + "inconclusive on this fit." + ), + } + ) vcov_for_wald: Optional[Any] = None idx_map_for_wald: Optional[Any] = None vcov_method_tag = "joint_wald" @@ -1496,6 +1777,20 @@ def _pt_event_study(self) -> Dict[str, Any]: else: es_vcov = getattr(r, "event_study_vcov", None) es_vcov_index = getattr(r, "event_study_vcov_index", None) + if es_vcov is None or es_vcov_index is None: + # Derived-route covariance: the post-fit container + # carries the same event-study vcov + ordered index + # (np.int64 labels hash-equal to the adapter's int + # keys; extra vcov rows are tolerated by the sub-block + # indexing below). + if ( + derived + and surface is not None + and getattr(surface, "vcov", None) is not None + and getattr(surface, "vcov_index", None) is not None + ): + es_vcov = surface.vcov + es_vcov_index = list(surface.vcov_index) if es_vcov is not None and es_vcov_index is not None: vcov_for_wald = es_vcov # ``event_study_vcov_index`` is an ordered list of @@ -1607,27 +1902,29 @@ def _pt_event_study(self) -> Dict[str, Any]: if not (isinstance(p["p_value"], (int, float)) and np.isfinite(p["p_value"])) ) if nan_p_count > 0: - return { - "status": "ran", - "method": "inconclusive", - "joint_p_value": None, - "test_statistic": None, - "df": len(pre_coefs), - "n_pre_periods": len(pre_coefs), - "n_dropped_undefined": nan_p_count, - "per_period": per_period, - "verdict": "inconclusive", - "reason": ( - f"{nan_p_count} retained pre-period coefficient(s) " - "have non-finite per-period p-value (undefined " - "inference per the ``safe_inference`` contract — " - "e.g., replicate-weight survey fits where effective " - "df collapsed). Bonferroni on the remaining subset " - "would silently shrink the test family; the joint " - "PT test is inconclusive on this fit. Inspect the " - "per_period block for the undefined rows." - ), - } + return _mark_derived( + { + "status": "ran", + "method": "inconclusive", + "joint_p_value": None, + "test_statistic": None, + "df": len(pre_coefs), + "n_pre_periods": len(pre_coefs), + "n_dropped_undefined": nan_p_count, + "per_period": per_period, + "verdict": "inconclusive", + "reason": ( + f"{nan_p_count} retained pre-period coefficient(s) " + "have non-finite per-period p-value (undefined " + "inference per the ``safe_inference`` contract — " + "e.g., replicate-weight survey fits where effective " + "df collapsed). Bonferroni on the remaining subset " + "would silently shrink the test family; the joint " + "PT test is inconclusive on this fit. Inspect the " + "per_period block for the undefined rows." + ), + } + ) ps = [p["p_value"] for p in per_period] if ps: joint_p = min(1.0, min(ps) * len(ps)) @@ -1647,7 +1944,7 @@ def _pt_event_study(self) -> Dict[str, Any]: # silently presenting a chi-square-style result. if df_denom is not None: out["df_denom"] = df_denom - return out + return _mark_derived(out) def _check_pretrends_power(self) -> Dict[str, Any]: """Compute pre-trends power (MDV) via ``compute_pretrends_power``. @@ -1660,18 +1957,37 @@ def _check_pretrends_power(self) -> Dict[str, Any]: from diff_diff.pretrends import compute_pretrends_power + # Derived-route substitution: when the raw event-study surface is + # absent and the resolver produced a container from an M-093-admitted + # producer (CS only), hand the container to the consumer — it accepts + # EventStudyResults directly, with pinned parity to the raw route. + surface, _surface_dict, _why = self._resolve_event_study_surface() + target: Any = self._results + pt_derived = False + if surface is not None and surface.source in _DERIVED_SURFACE_CONSUMER_SOURCES: + target = surface + pt_derived = True + try: pp = compute_pretrends_power( - self._results, + target, alpha=self._alpha, target_power=0.80, violation_type="linear", ) except Exception as exc: # noqa: BLE001 - return { + err: Dict[str, Any] = { "status": "error", "reason": f"compute_pretrends_power raised " f"{type(exc).__name__}: {exc}", } + if pt_derived: + # A derived-route failure stays schema-distinguishable from + # a raw-route one (the additive-key contract in + # REPORTING.md), and captured derivation warnings ride out. + err["pre_period_source"] = "aggregate_event_study" + if self._derived_es_warnings: + err["warnings"] = list(self._derived_es_warnings) + return err # Build the schema section and compute the level-scale max-pre- # violation / |ATT| ratio for BR tier classification. Post-PR-B @@ -1705,7 +2021,7 @@ def _check_pretrends_power(self) -> Dict[str, Any]: if cov_source == "unknown": cov_source = self._infer_cov_source(self._results) tier = _apply_diag_fallback_downgrade(_power_tier(ratio), cov_source) - return { + section: Dict[str, Any] = { "status": "ran", "method": "compute_pretrends_power", "violation_type": getattr(pp, "violation_type", "linear"), @@ -1724,6 +2040,11 @@ def _check_pretrends_power(self) -> Dict[str, Any]: "tier": tier, "covariance_source": cov_source, } + if pt_derived: + section["pre_period_source"] = "aggregate_event_study" + if self._derived_es_warnings: + section["warnings"] = list(self._derived_es_warnings) + return section def _format_precomputed_pretrends_power(self, obj: Any) -> Dict[str, Any]: """Adapt a pre-computed ``PreTrendsPowerResults`` to the schema shape. @@ -1906,6 +2227,19 @@ def _check_sensitivity(self) -> Dict[str, Any]: import warnings as _warnings + # Derived-route substitution: hand HonestDiD the post-fit + # container when the raw event-study surface is absent and the + # producer is M-093-admitted (CS only); HonestDiD accepts + # EventStudyResults directly with pinned raw-route parity. + # Resolved OUTSIDE the try so ``sens_derived`` is always bound in + # the except handler (the resolver itself never raises). + surface, _surface_dict, _why = self._resolve_event_study_surface() + sens_target: Any = self._results + sens_derived = False + if surface is not None and surface.source in _DERIVED_SURFACE_CONSUMER_SOURCES: + sens_target = surface + sens_derived = True + try: from typing import cast @@ -1926,18 +2260,28 @@ def _check_sensitivity(self) -> Dict[str, Any]: alpha=self._alpha, ) sens = honest.sensitivity_analysis( - self._results, + sens_target, M_grid=list(self._sensitivity_M_grid), ) except Exception as exc: # noqa: BLE001 - return { + err: Dict[str, Any] = { "status": "error", "method": self._sensitivity_method, "reason": f"HonestDiD.sensitivity_analysis raised " f"{type(exc).__name__}: {exc}", } + if sens_derived: + # Same additive-key contract as the pretrends_power error + # path: derived-route failures carry their provenance. + err["pre_period_source"] = "aggregate_event_study" + if self._derived_es_warnings: + err["warnings"] = list(self._derived_es_warnings) + return err captured = [str(w.message) for w in caught if issubclass(w.category, Warning)] formatted = self._format_sensitivity_results(sens) + if sens_derived: + formatted["pre_period_source"] = "aggregate_event_study" + captured = list(self._derived_es_warnings) + captured if captured: formatted["warnings"] = captured return formatted @@ -2244,14 +2588,58 @@ def _check_design_effect(self) -> Dict[str, Any]: "band_label": band_label, } + def _surface_post_effect_scalars(self, surface: Any) -> List[float]: + """Post-treatment effect scalars from a derived event-study surface. + + Mirrors ``_collect_effect_scalars``'s event-study branch: rows at or + after the anticipation-aware boundary, reference rows excluded, + finite effects only. + """ + boundary = _pre_post_boundary(self._results) + vals: List[float] = [] + for k, att, is_ref in zip(surface.event_time, surface.att, surface.is_reference): + if bool(is_ref): + continue + try: + rel = int(k) + except (TypeError, ValueError): + continue + if rel < boundary: + continue + att_f = float(att) + if np.isfinite(att_f): + vals.append(att_f) + return vals + def _check_heterogeneity(self) -> Dict[str, Any]: """Compute effect-stability metrics (CV, range, sign consistency).""" effects = self._collect_effect_scalars() + het_derived = False if not effects: - return { - "status": "skipped", - "reason": "No group / event-study / period effects available.", - } + # Derived-surface fallback: the raw effect fields are empty + # (e.g. a plain ImputationDiD/TwoStageDiD/ContinuousDiD fit) — + # consume the post-fit aggregate('event_study') surface when + # one resolves. This is a DR-internal computation over the + # coefficient list; no consumer admission is involved. + surface, _surface_dict, _why = self._resolve_event_study_surface() + if surface is not None: + effects = self._surface_post_effect_scalars(surface) + het_derived = bool(effects) + if not effects: + reason = "No group / event-study / period effects available." + if self._derived_es_failure is not None: + # Post-period phrasing: never inherit the pre-period- + # shaped resolver reason here; append only the bare + # derivation context. + reason += ( + " The event-study surface could not be derived via " + "results.aggregate('event_study'): " + f"{self._derived_es_failure}" + ) + skip: Dict[str, Any] = {"status": "skipped", "reason": reason} + if self._derived_es_warnings: + skip["warnings"] = list(self._derived_es_warnings) + return skip vals = np.array(effects, dtype=float) finite = vals[np.isfinite(vals)] if finite.size == 0: @@ -2265,9 +2653,11 @@ def _check_heterogeneity(self) -> Dict[str, Any]: mx = float(np.max(finite)) cv = sd / abs(mean) if abs(mean) > 0.1 * sd and abs(mean) > 0 else None sign_consistent = bool(np.all(finite >= 0) or np.all(finite <= 0)) - return { + section: Dict[str, Any] = { "status": "ran", - "source": self._heterogeneity_source(), + "source": ( + "aggregate_event_study_post" if het_derived else self._heterogeneity_source() + ), "n_effects": int(finite.size), "min": mn, "max": mx, @@ -2277,6 +2667,9 @@ def _check_heterogeneity(self) -> Dict[str, Any]: "cv": cv, "sign_consistent": sign_consistent, } + if het_derived and self._derived_es_warnings: + section["warnings"] = list(self._derived_es_warnings) + return section def _check_epv(self) -> Dict[str, Any]: """Read EPV diagnostics from ``results.epv_diagnostics``. @@ -3587,8 +3980,62 @@ def _pre_post_boundary(results: Any) -> int: return -k +def _surface_to_event_study_dict(surface: Any) -> Dict[Any, Dict[str, float]]: + """Adapt an ``EventStudyResults`` container to the legacy + ``event_study_effects`` dict shape consumed by this module's readers. + + Only relative-scale surfaces are adaptable — calendar-scale + ``event_time`` legally carries str/datetime labels that ``int()`` would + mangle. Unreachable via ``_resolve_event_study_surface`` today (no + calendar producer declares ``_AGGREGATE_SUPPORTED``), but the helper + must not depend on caller ordering. + + Rows are SKIPPED when ``is_reference`` is set OR the row's count is + exactly zero (reference rows carry NaN ``n``, so the conditions are + disjoint). The zero-count skip mirrors the raw route's + ``n_groups == 0`` / ``n_obs == 0`` exclusion in + ``_collect_pre_period_coefs`` — the container deliberately preserves + NaN-effect zero-count horizons as NON-reference rows + (``results_base._from_relative_dict``), and without this skip such a + row would inflate ``n_dropped_undefined`` and flip a valid joint-Wald + PT to "inconclusive". NaN se/p on surviving rows pass through so the + undefined-inference guards behave identically to the raw route. + + Keys are Python ``int`` (matching the ImputationDiD/TwoStageDiD raw + key dtype exactly; the CS derived route orders numerically where its + raw ``np.int64`` keys str-sort — presentational only, the two routes + never coexist). + """ + if getattr(surface, "time_scale", "relative") != "relative": + raise ValueError( + "Only relative-scale EventStudyResults surfaces can be adapted " + f"to the event_study_effects dict shape; got time_scale=" + f"{surface.time_scale!r}." + ) + out: Dict[Any, Dict[str, float]] = {} + for k, att, se, p, is_ref, n in zip( + surface.event_time, + surface.att, + surface.se, + surface.p_value, + surface.is_reference, + surface.n, + ): + if bool(is_ref): + continue + if np.isfinite(n) and float(n) == 0.0: + continue + out[int(k)] = { + "effect": float(att), + "se": float(se), + "p_value": float(p), + } + return out + + def _collect_pre_period_coefs( results: Any, + surface_dict: Optional[Dict[Any, Dict[str, float]]] = None, ) -> Tuple[List[Tuple[Any, float, float, Optional[float]]], int]: """Return ``(sorted list of (key, effect, se, p_value), n_dropped_undefined)`` for pre-period coefficients. @@ -3598,6 +4045,10 @@ def _collect_pre_period_coefs( * ``event_study_effects``: dict-of-dict (with ``effect`` / ``se`` / ``p_value`` keys) on the staggered estimators (CS / SA / ImputationDiD / Stacked / EDiD / etc.). Pre-period entries are those with negative relative-time keys. + When the raw field is ``None`` and the caller supplies + ``surface_dict`` (the resolver's pre-converted post-fit + ``aggregate('event_study')`` surface), that dict substitutes for + the raw field; a raw ``{}`` is authoritative and never overridden. * ``placebo_event_study``: dict-of-dict on ``ChaisemartinDHaultfoeuilleResults`` — dCDH's dynamic placebos ``DID^{pl}_l`` are the estimator's pre-period analogue. @@ -3676,7 +4127,10 @@ def _collect_pre_period_coefs( # anticipation window (not true pre-periods) and only use # ``e < -k`` for PT tests. pre_cutoff = _pre_post_boundary(results) - es = getattr(results, "event_study_effects", None) or {} + es = getattr(results, "event_study_effects", None) + if es is None and surface_dict is not None: + es = surface_dict + es = es or {} for k, entry in es.items(): # Pre-period relative-time keys are negative (convention: e=-1, -2, ...). try: diff --git a/diff_diff/guides/llms-full.txt b/diff_diff/guides/llms-full.txt index f19cc7f74..f25300cd4 100644 --- a/diff_diff/guides/llms-full.txt +++ b/diff_diff/guides/llms-full.txt @@ -2821,11 +2821,14 @@ dr = DiagnosticReport( precomputed={"sensitivity": my_honest_did_results}, ) -dr.run_all() # triggers compute, caches +dr.run_all() # triggers the full check computation, caches print(dr.summary()) # overall-interpretation paragraph dr.to_dict() # AI-legible schema dr.to_dataframe() # one row per check dr.applicable_checks # tuple of checks that will run for this estimator + # (may derive the fit's post-fit + # aggregate('event_study') surface once, cached, + # when the raw event_study_effects field is absent) dr.skipped_checks # dict of {check: plain-English reason} ``` @@ -2856,9 +2859,12 @@ Power tier (drives BR phrasing for the `no_detected_violation` verdict): ### Methodology notes -BR and DR do no estimator fitting and do not re-derive variance from -raw data — every effect, SE, p-value, CI, and sensitivity bound is -read from the fitted result or produced by an existing diff-diff +BR and DR do no estimator fitting — every effect, SE, p-value, CI, and +sensitivity bound is read from the fitted result, derived from the +result's own post-fit `aggregate('event_study')` surface (a view or +retained-kit recompute, used only when the raw `event_study_effects` +field is absent; bootstrapped / kit-less fits fail closed to an +explicit skip), or produced by an existing diff-diff utility (may call `check_parallel_trends`, `BaconDecomposition.fit`, or `EfficientDiD.hausman_pretest` when the panel + column kwargs are supplied). The `design_effect` section is read-only: it echoes diff --git a/docs/api/business_report.rst b/docs/api/business_report.rst index ae17bffac..4ffb382e2 100644 --- a/docs/api/business_report.rst +++ b/docs/api/business_report.rst @@ -74,8 +74,11 @@ Example cs = CallawaySantAnna(base_period="universal").fit( df, outcome="revenue", unit="store", time="period", - first_treat="first_treat", aggregate="event_study", + first_treat="first_treat", ) + # The auto-constructed DiagnosticReport derives the event-study + # surface internally via post-fit aggregate('event_study') when the + # pre-trends checks need it. report = BusinessReport( cs, outcome_label="Revenue per store", diff --git a/docs/api/diagnostic_report.rst b/docs/api/diagnostic_report.rst index fb691dc98..f7cf8976d 100644 --- a/docs/api/diagnostic_report.rst +++ b/docs/api/diagnostic_report.rst @@ -7,7 +7,11 @@ Goodman-Bacon, design-effect, EPV, heterogeneity, and estimator-native checks for SyntheticDiD and TROP) into a single report with a stable AI-legible schema. -Construction is free; ``run_all()`` triggers the compute and caches. +Construction is free; accessing ``applicable_checks`` may derive the +fit's post-fit event-study surface once (a view or kit recompute via +``results.aggregate('event_study')``, cached for the report's +lifetime, and only when the raw ``event_study_effects`` field is +absent); ``run_all()`` triggers the full check computation and caches. A second call to ``to_dict()`` or ``summary()`` reuses the cached result. @@ -56,8 +60,11 @@ Example cs = CallawaySantAnna(base_period="universal").fit( df, outcome="outcome", unit="unit", time="period", - first_treat="first_treat", aggregate="event_study", + first_treat="first_treat", ) + # The event-study-gated checks (parallel trends, pre-trends power, + # sensitivity) derive the surface internally via the result's + # post-fit aggregate('event_study') when needed. dr = DiagnosticReport( cs, data=df, diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index 1379f5c7b..5c729db74 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -1051,6 +1051,7 @@ The multiplier bootstrap uses random weights w_i with E[w]=0 and Var(w)=1: - **Note:** Repeated cross-sections (`panel=False`, Phase 7b): supports surveys like BRFSS, ACS annual, and CPS monthly where units are not followed over time. Uses cross-sectional DRDID (Sant'Anna & Zhao 2020, Section 4): `reg` matches `DRDID::reg_did_rc` (Eq 2.2), `dr` matches `DRDID::drdid_rc` (locally efficient, Eq 3.3+3.4 with 4 OLS fits), `ipw` matches `DRDID::std_ipw_did_rc`. Per-observation influence functions instead of per-unit. All three estimation methods support covariates and survey weights. - **Note:** Panel and RCS influence functions use the library-wide `phi_i = psi_i / n` convention (SE = `sqrt(sum(phi^2))`, algebraically equivalent to R's `sd(psi)*sqrt(n-1)/n`). Leading IF terms are computed on psi scale and divided by n; PS nuisance corrections are computed on psi scale (`score @ solve(Hessian)`) with a single `/n` conversion to phi. - **Note:** Non-survey DR path also includes nuisance IF corrections (PS + OR), matching the survey path structure (Phase 7a). Previously used plug-in IF only. As of v3.7 the non-survey reg and ipw paths carry their corrections too (OR estimation-effect / PS score), so the nuisance-IF treatment is method-uniform. +- **Note (post-fit aggregate() - rows M-020/M-117):** `fit(aggregate=)` is deprecated (3.9; removed 4.0) in favor of post-fit `results.aggregate(type, balance_e=)` on the fit-retained aggregation kit ('simple' relays the stored overall inference; 'event_study'/'group' recompute from the kit and fail closed on bootstrapped fits per the M-027 per-level rule). `DiagnosticReport` now derives the event-study container internally on plain fits (raw `event_study_effects` absent), so its `parallel_trends`, `pretrends_power`, and `sensitivity` checks run without the deprecated fit-time kwarg — CS containers are M-093-admitted into `compute_pretrends_power` / `HonestDiD.sensitivity_analysis` with pinned raw-route parity, and the raw field, when present (the requested-but-empty `{}` included), always takes precedence. `heterogeneity` is unaffected (it reads `group_time_effects` on plain fits). Bootstrapped fits surface the fail-closed `NotImplementedError` as an explicit per-check skip reason. **Reference implementation(s):** - R: `did::att_gt()` (Callaway & Sant'Anna's official package) @@ -1422,7 +1423,7 @@ labels.* 6. **Note (discrete-treatment saturated regression — library extension beyond `contdid` v0.1.0):** `treatment_type="discrete"` estimates the dose-response by a **saturated regression** (CGBS 2024 Eq. 4.1) — one indicator per distinct dose level, so `beta_j = mean_{D=d_j}(ΔY − control) = ATT(d_j)` (a per-level 2×2 DiD) — instead of the B-spline sieve. `ACRT(d_j)` is the paper's **backward difference** on the grid `{d_0 = 0, d_1, …, d_J}` (Eq. 4.1 makes `d_0 = 0` the omitted category with `ATT(0) = 0`): `ACRT(d_j) = [ATT(d_j) − ATT(d_{j-1})]/(d_j − d_{j-1})` for `j ≥ 2`, and at the lowest positive level it references the zero-dose baseline, `ACRT(d_1) = [ATT(d_1) − 0]/(d_1 − 0) = ATT(d_1)/d_1`. So a single positive dose (`J = 1`, e.g. binary `D ∈ {0,1}`) yields `ACRT(d_1) = ATT(d_1)/d_1`, and for `d_1 = 1` the documented binary identity `ACRT = ATT` holds exactly. This is a **library extension**: `contdid` v0.1.0 accepts `treatment_type` in its signature but **does not implement the discrete path** (documented "Discrete treatment not yet implemented"), so there is **no external R anchor**. It is instead an *exact* basis swap of the B-spline design/evaluation/derivative trio for an indicator/identity/finite-difference trio; every downstream quantity is linear in `beta`, so the analytical-SE / multiplier-bootstrap / covariate (reg,dr) / survey machinery is reused unchanged and reduces *analytically* to the per-level 2×2 DiD (`bread @ psi_bar = ones(J)`; the common control mean cancels in the `j ≥ 2` adjacent differences whose `L`-rows sum to 0). **reg vs dr:** the constant DR augmentation `η̄_cont` cancels in the `j ≥ 2` differences, so `ACRT(d_j)` point AND SE are identical for `reg`/`dr` there; but `ACRT(d_1) = ATT(d_1)/d_1` references the fixed baseline `ATT(0) = 0` (not shifted by `η̄_cont`), so `reg` and `dr` genuinely **differ at `ACRT(d_1)` by `η̄_cont/d_1`** (and correspondingly in `ACRT^glob` via the `d_1` mass) — the dr influence function carries the augmentation variance at `d_1` (validated: analytical `ACRT(d_1)` SE matches the multiplier bootstrap). Validation (R-free, in CI): exact hand-calc of `ATT(d_j)`/`ACRT`/`overall_att` and the analytical SE against a direct per-level 2×2 reconstruction (`~1e-12`/`~1e-10`), DGP recovery, and MC coverage for analytical + bootstrap (`tests/test_methodology_continuous_did.py::TestDiscreteSaturated`, `tests/test_continuous_did.py::TestDiscreteSaturatedAPI`). **Fail-closed policies (no-silent-failures):** (i) multi-cohort fits with **heterogeneous dose support** across cohorts raise `NotImplementedError` — an absent global level yields a dropped zero column (`att_d[level]=0`) that the plain-sum dose aggregation would bias toward zero (support-aware aggregation is deferred; single-cohort, 2-period, and shared-support multi-cohort are supported); (ii) a requested `dvals` value that is not an observed dose level raises `ValueError` (a saturated model cannot be evaluated off-support); (iii) an over-parameterized fit (`< 2` treated units per level, or `J > n_treated/2`) warns (degenerate per-level SE); (iv) with `survey_design=`, any dose level with **zero effective treated mass in a `(g,t)` cell** raises `ValueError` — a per-cell check (not just the global positive-weight check), so a level that survey/subpopulation weights zero out for one cohort while another cohort keeps it cannot silently drop to a zero-coefficient saturated column. Cross-references `docs/methodology/continuous-did.md` § 5.1. 7. **Note (lowest-dose-as-control, Remark 3.1 — library extension beyond `contdid` v0.1.0):** `control_group="lowest_dose"` implements CGBS 2024 Remark 3.1 for settings with no untreated group (`P(D=0) = 0`): the lowest-dose group `d_L` becomes the comparison and the estimand is `ATT(d) − ATT(d_L)` (SPT), with `ATT(d_L) = 0` the omitted reference. Mechanically it is a **control-group swap** — the D=0 control pool is replaced by the `d_L` group; the entire linear influence-function / bootstrap / event-study / survey machinery is control-group-generic and reused unchanged (`ee_control` already carries the reference-group variance, so **no new SE plumbing**). On the discrete saturated basis the backward-difference operator's reference shifts from `0` to `d_L` (`ACRT(d_1) = ATT(d_1)/(d_1 − d_L)`); on the continuous B-spline path the reference shifts only `μ_0` (the level), leaving `ACRT = spline'` unchanged. `contdid` v0.1.0 does **not** implement Remark 3.1, so there is **no external R anchor**; validation (R-free, in CI): an **exact `d_L → 0` equivalence** anchor (relabelling a `never_treated` panel's D=0 group as a tiny common dose `d_L = ε` reproduces the `never_treated` ATT and SE exactly, for any ε), a discrete hand-calc of `ATT(d)−ATT(d_L)`/`ACRT`/`overall_att`/`overall_acrt` and the per-level 2×2 SE (`~1e-10`), continuous mass-point DGP recovery, analytical-vs-bootstrap SE agreement, a pre-period placebo, and MC coverage (`tests/test_methodology_continuous_did.py::TestLowestDose`, `tests/test_continuous_did.py::TestLowestDoseAPI`). The continuous path requires a genuine **mass point** at the minimum dose (`>= 2` units at `d_L`, i.e. `P(D=d_L) > 0`) — the Remark 3.1 identification condition; a singleton minimum fails closed. **Fail-closed policies (no-silent-failures):** (i) never-treated units present with `lowest_dose` → `ValueError` (they would be silently dropped); (ii) singleton `d_L` (no mass point) → `ValueError`; (iii) no treated dose above `d_L` → `ValueError`; (iv) user `dvals ≤ d_L` → `ValueError` (`d_L` is the omitted reference); (v) survey/subpopulation weighting that leaves the `d_L` group with `< 2` positive-weight units → `ValueError` (a single positive-weight reference unit gives `ee_control = 0`, i.e. zero control-side variance — the effective-`>= 2` analogue of the raw mass-point guard, applied after weighting); (vi) a boundary gap `d_1 − d_L` that is a tiny fraction of the dose range warns (huge boundary ACRT/SE). **Deferred (fail-closed `NotImplementedError` + TODO):** multi-cohort `lowest_dose` (needs a within-cohort reference + support-aware cross-cohort aggregation) and `covariates=` × `lowest_dose` (conditional-PT-relative-to-`d_L` estimand). Cross-references `docs/methodology/continuous-did.md` § 5.6. -8. **Note (post-fit aggregate() - rows M-025/M-122):** `fit(aggregate=)` is deprecated in 3.9 (removed in 4.0; the no-underscore `"eventstudy"` spelling dies with it) in favor of post-fit `results.aggregate(type)` on the unified vocabulary + `'dose'` as this estimator's documented extra level; the PRE-EXISTING fit-time value validation is retained (unknown strings still raise `ValueError` after the deprecation warning - unlike the EfficientDiD/Imputation shims, which never validated). (a) **MIXED view/recompute architecture** (unique among the aggregate-postfit adopters): the dose-response curves and the overall binarized ATT (ATT^{loc} under PT; equals ATT^{glob} under SPT) plus ACRT^{glob} are ALWAYS computed by `fit()` (`aggregate="dose"` was a fit-time no-op), so `aggregate('simple')` (2 rows, targets att/acrt - the dual-estimand case the `AggregationResult.target` column exists for; `n` = the DISJOINT treated+control unit total, `n_kind='units'`) and `aggregate('dose')` (2N target-discriminated rows; labels = the dose grid twice; no count/mass per row) are pure VIEWS over stored public fields, PERMITTED on bootstrap fits (the library-wide per-level relay rule, since M-027 converged CS/EDiD/Imputation/TwoStage onto it) - they relay stored inference verbatim, including the FINITE `safe_inference` t-stat fit stores beside the percentile p/CI on bootstrapped overall rows and the `DoseResponseCurve.to_dataframe`-exact NaN-t derivation on dose rows; only the df column is uniformly NaN under bootstrap. (b) **`aggregate('event_study')` recomputes** the binarized event study from a pruned per-cell IF payload retained on the fit-built kit: per-(g,t) treated/control positional indices, `delta_y_treated`, `ee_control`, masses and the covariate-path `if_att_glob` (O(n_treated+n_control) per cell), unit-level arrays, and - on survey fits - the PANEL-LEVEL `ResolvedSurveyDesign` ref (the recompute performs the unit collapse itself, keeping the moved body verbatim; on replicate designs the (n_obs x R) replicate matrix rides along - the documented memory cost). The K-dimensional spline machinery (bread, `ee_treated`, `Psi_eval`, `dPsi_*`) is NOT retained; no panel data columns and no raw unit identifiers are retained. Replicate-weight designs ARE supported post-fit (IF-based `compute_replicate_if_variance` - no refit replay). Bootstrap fits carry a SCALARS-ONLY kit and the event-study route fails closed (`NotImplementedError`; the deprecated fit-time `aggregate='eventstudy'` computes the bootstrap surface until 4.0, or re-fit with `n_bootstrap=0`) - a seeded post-fit bootstrap replay is the TODO.md row. (c) **Fit-faithful quirk:** when no post-treatment (g,t) cells exist, event-study rows keep NaN inference on BOTH routes (the fit-time surface never fills them). (d) **df provenance:** the stored `dose_response_att.df_survey` channel (the value every fit-time `safe_inference` received) drives the views' df column (finite-and-positive else NaN - the replicate-undefined 0 sentinel reports NaN in the column but feeds the t/p derivation raw); the post-fit event-study container exposes the scalar `df_survey` channel only (all-NaN per-row df - the M-092 completion hole, tracked in TODO.md). (e) **Rendering:** this is the FIRST heterogeneous-`target` `AggregationResult`; `summary()`/`to_dataframe()` gained the target column / first-appearance target-block ordering amendment (normative rule in `docs/v4-design.md` section 6; uniform-target producers byte-stable). (f) **Consumer admission:** `compute_honest_did`/`compute_pretrends_power` reject ContinuousDiD containers BY DESIGN - no joint event-study covariance exists (per-bin IF SEs only) and the binarized bins carry no reference-period normalization at all (see M-093). Warning stacklevels in the moved bodies remain tuned for the fit-time frame depth, so post-fit-route warnings attribute to a library frame (the shipped EfficientDiD convention). +8. **Note (post-fit aggregate() - rows M-025/M-122):** `fit(aggregate=)` is deprecated in 3.9 (removed in 4.0; the no-underscore `"eventstudy"` spelling dies with it) in favor of post-fit `results.aggregate(type)` on the unified vocabulary + `'dose'` as this estimator's documented extra level; the PRE-EXISTING fit-time value validation is retained (unknown strings still raise `ValueError` after the deprecation warning - unlike the EfficientDiD/Imputation shims, which never validated). (a) **MIXED view/recompute architecture** (unique among the aggregate-postfit adopters): the dose-response curves and the overall binarized ATT (ATT^{loc} under PT; equals ATT^{glob} under SPT) plus ACRT^{glob} are ALWAYS computed by `fit()` (`aggregate="dose"` was a fit-time no-op), so `aggregate('simple')` (2 rows, targets att/acrt - the dual-estimand case the `AggregationResult.target` column exists for; `n` = the DISJOINT treated+control unit total, `n_kind='units'`) and `aggregate('dose')` (2N target-discriminated rows; labels = the dose grid twice; no count/mass per row) are pure VIEWS over stored public fields, PERMITTED on bootstrap fits (the library-wide per-level relay rule, since M-027 converged CS/EDiD/Imputation/TwoStage onto it) - they relay stored inference verbatim, including the FINITE `safe_inference` t-stat fit stores beside the percentile p/CI on bootstrapped overall rows and the `DoseResponseCurve.to_dataframe`-exact NaN-t derivation on dose rows; only the df column is uniformly NaN under bootstrap. (b) **`aggregate('event_study')` recomputes** the binarized event study from a pruned per-cell IF payload retained on the fit-built kit: per-(g,t) treated/control positional indices, `delta_y_treated`, `ee_control`, masses and the covariate-path `if_att_glob` (O(n_treated+n_control) per cell), unit-level arrays, and - on survey fits - the PANEL-LEVEL `ResolvedSurveyDesign` ref (the recompute performs the unit collapse itself, keeping the moved body verbatim; on replicate designs the (n_obs x R) replicate matrix rides along - the documented memory cost). The K-dimensional spline machinery (bread, `ee_treated`, `Psi_eval`, `dPsi_*`) is NOT retained; no panel data columns and no raw unit identifiers are retained. Replicate-weight designs ARE supported post-fit (IF-based `compute_replicate_if_variance` - no refit replay). Bootstrap fits carry a SCALARS-ONLY kit and the event-study route fails closed (`NotImplementedError`; the deprecated fit-time `aggregate='eventstudy'` computes the bootstrap surface until 4.0, or re-fit with `n_bootstrap=0`) - a seeded post-fit bootstrap replay is the TODO.md row. (c) **Fit-faithful quirk:** when no post-treatment (g,t) cells exist, event-study rows keep NaN inference on BOTH routes (the fit-time surface never fills them). (d) **df provenance:** the stored `dose_response_att.df_survey` channel (the value every fit-time `safe_inference` received) drives the views' df column (finite-and-positive else NaN - the replicate-undefined 0 sentinel reports NaN in the column but feeds the t/p derivation raw); the post-fit event-study container exposes the scalar `df_survey` channel only (all-NaN per-row df - the M-092 completion hole, tracked in TODO.md). (e) **Rendering:** this is the FIRST heterogeneous-`target` `AggregationResult`; `summary()`/`to_dataframe()` gained the target column / first-appearance target-block ordering amendment (normative rule in `docs/v4-design.md` section 6; uniform-target producers byte-stable). (f) **Consumer admission:** `compute_honest_did`/`compute_pretrends_power` reject ContinuousDiD containers BY DESIGN - no joint event-study covariance exists (per-bin IF SEs only) and the binarized bins carry no reference-period normalization at all (see M-093). Warning stacklevels in the moved bodies remain tuned for the fit-time frame depth, so post-fit-route warnings attribute to a library frame (the shipped EfficientDiD convention). (g) **Report-layer consumption:** `DiagnosticReport` now derives this container internally on plain fits (raw `event_study_effects` absent) for its `heterogeneity` check — the only ES-gated check in ContinuousDiD's applicability; captured recompute warnings are re-published on the section and derivation failures (bootstrap fits) surface as explicit skip context. ### Implementation Checklist @@ -1976,7 +1977,7 @@ where `W_it(h) = 1[K_it = h]` are lead indicators, estimated on `Omega_0` only. - [x] Supports balanced and unbalanced panels (iterative Gauss-Seidel demeaning for exact FE) - [x] Event study and group aggregation -- **Note (post-fit aggregate() - rows M-021/M-118):** `fit(aggregate=, balance_e=)` is deprecated (3.9; removed 4.0; joint FutureWarning, warn-and-still-work; the `imputation_did` wrapper forwards the shared sentinel so plain wrapper calls never fire the aggregate warning; since 3.9 the wrapper itself warns per M-070) in favor of post-fit `ImputationDiDResults.aggregate(type, balance_e=)` - a PANEL-BACKED lazy recompute kit (not an EIF-payload kit: ES/group aggregation is a target-specific Theorem-3 recompute - each `balance_e` re-masks which treated observations enter every horizon and re-solves the untreated projection - so no compact influence payload can replace the frame). (a) RETAINED BUFFERS (memory contract): the kit's bookkeeping holds REFERENCES to the SAME per-fit objects `_fit_data` already retains for `pretrend_test()` - the working panel copy (all user columns plus `_tau_hat`/`_rel_time`/`_never_treated`), the Omega masks, `unit_fe`/`time_fe`/`grand_mean`/`delta_hat`/`kept_cov_mask`, the resolved survey design, and `survey_weights` - ZERO marginal memory, and pickles are unchanged via memoization (`_estimator_ref` already ships these objects); plus value SNAPSHOTS for isolation (a `treatment_groups` copy, config scalars, a `dataclasses.replace` copy of `survey_metadata`, `overall_att`, `n_treated_obs`) and TWO df-provenance scalars (`survey_df_seed`, what the analytical aggregators received; `survey_df_final`, what the stored overall inference received). Each `aggregate()` call runs on a fresh throwaway host with a call-local projection cache (the fit-local factorizations are unpicklable and never retained). (b) `balance_e` uses the BALANCED-WINDOW rule: a cohort is retained iff its observed relative-time set - checked against the FULL panel via `_build_cohort_rel_times()` - covers the contiguous window `[-balance_e, max_h]`; the SAME rule TwoStageDiD uses, divergent from CS/EfficientDiD's anchor-horizon rule. A window no cohort satisfies warns and yields the reference-marker-only dict (a legal near-empty container). (c) BOOTSTRAP fits: 'simple' RELAYS the stored overall quintet verbatim (finite safe_inference t included) with a NaN df column, while the RECOMPUTE levels fail closed (the per-target psi machinery makes exact replay tractable - a TODO row); the prior uniform fail-closed rule was superseded 2026-08-05 with the M-027 per-level convergence. (d) CONTAINER ADMISSION into `compute_honest_did`/`compute_pretrends_power` is REJECTED BY DESIGN (both terminal TypeErrors state it): the surface carries no joint event-study covariance - per-horizon conservative SEs only (container `vcov=None`; the scalar `df_survey` channel is its only df provenance, the per-row hole being the tracked M-092-completion TODO row). (e) RELAY CONVENTIONS: 'simple' relays the stored overall quintet bit-exact with `n = n_treated_obs`, `n_kind="obs"` (the treated/control UNIT sets overlap - a treated unit with pre-periods counts in both - so the CS/EDiD disjoint-units convention cannot apply; |Omega_1| is the population the ATT averages over, of which only finite-tau-hat observations enter the average - `n` reports the raw count, so on partially unidentified fits `n` exceeds the averaged support) and `df = survey_df_final`; 'group' rows carry per-row `df_used` captured at each row's `safe_inference` (the replicate override rewrites it, the bootstrap override clears it, the all-NaN cohort branch writes no key - consumers read via `.get`); 'event_study' rides the shared `_from_relative_dict` builder via a carrier whose metadata is a copy-on-use of the KIT's fit-final metadata copy. REPLICATE-WEIGHT fits replay the extracted `_replicate_override_aggregates` with a LEVEL-MATCHED stack: `compute_replicate_refit_variance` validates replicates JOINTLY (all-finite rows), so `aggregate(L)` reproduces `fit(aggregate=L)` exactly, a `fit(aggregate='all')` surface is NOT the equivalence target when a replicate NaNs on exactly one family's targets, and - the documented migration delta - moving a replicate fit from `fit(aggregate=)` to plain fit + post-fit `aggregate()` can change the public OVERALL row's se/CI/df on such degenerate designs (each surface self-consistent; pinned in the contract tests). `pretrends=True` + replicate: post-fit `aggregate('event_study')` raises the same NotImplementedError the fit-time gate raises (per-replicate lead refits unimplemented); 'group'/'simple' still work. Recompute re-emits the fit-time warnings (LSMR, Prop-5, empty-window) with fit-tuned stacklevels - post-fit attribution lands on a library frame, an accepted verbatim-move trade-off. +- **Note (post-fit aggregate() - rows M-021/M-118):** `fit(aggregate=, balance_e=)` is deprecated (3.9; removed 4.0; joint FutureWarning, warn-and-still-work; the `imputation_did` wrapper forwards the shared sentinel so plain wrapper calls never fire the aggregate warning; since 3.9 the wrapper itself warns per M-070) in favor of post-fit `ImputationDiDResults.aggregate(type, balance_e=)` - a PANEL-BACKED lazy recompute kit (not an EIF-payload kit: ES/group aggregation is a target-specific Theorem-3 recompute - each `balance_e` re-masks which treated observations enter every horizon and re-solves the untreated projection - so no compact influence payload can replace the frame). (a) RETAINED BUFFERS (memory contract): the kit's bookkeeping holds REFERENCES to the SAME per-fit objects `_fit_data` already retains for `pretrend_test()` - the working panel copy (all user columns plus `_tau_hat`/`_rel_time`/`_never_treated`), the Omega masks, `unit_fe`/`time_fe`/`grand_mean`/`delta_hat`/`kept_cov_mask`, the resolved survey design, and `survey_weights` - ZERO marginal memory, and pickles are unchanged via memoization (`_estimator_ref` already ships these objects); plus value SNAPSHOTS for isolation (a `treatment_groups` copy, config scalars, a `dataclasses.replace` copy of `survey_metadata`, `overall_att`, `n_treated_obs`) and TWO df-provenance scalars (`survey_df_seed`, what the analytical aggregators received; `survey_df_final`, what the stored overall inference received). Each `aggregate()` call runs on a fresh throwaway host with a call-local projection cache (the fit-local factorizations are unpicklable and never retained). (b) `balance_e` uses the BALANCED-WINDOW rule: a cohort is retained iff its observed relative-time set - checked against the FULL panel via `_build_cohort_rel_times()` - covers the contiguous window `[-balance_e, max_h]`; the SAME rule TwoStageDiD uses, divergent from CS/EfficientDiD's anchor-horizon rule. A window no cohort satisfies warns and yields the reference-marker-only dict (a legal near-empty container). (c) BOOTSTRAP fits: 'simple' RELAYS the stored overall quintet verbatim (finite safe_inference t included) with a NaN df column, while the RECOMPUTE levels fail closed (the per-target psi machinery makes exact replay tractable - a TODO row); the prior uniform fail-closed rule was superseded 2026-08-05 with the M-027 per-level convergence. (d) CONTAINER ADMISSION into `compute_honest_did`/`compute_pretrends_power` is REJECTED BY DESIGN (both terminal TypeErrors state it): the surface carries no joint event-study covariance - per-horizon conservative SEs only (container `vcov=None`; the scalar `df_survey` channel is its only df provenance, the per-row hole being the tracked M-092-completion TODO row). (e) RELAY CONVENTIONS: 'simple' relays the stored overall quintet bit-exact with `n = n_treated_obs`, `n_kind="obs"` (the treated/control UNIT sets overlap - a treated unit with pre-periods counts in both - so the CS/EDiD disjoint-units convention cannot apply; |Omega_1| is the population the ATT averages over, of which only finite-tau-hat observations enter the average - `n` reports the raw count, so on partially unidentified fits `n` exceeds the averaged support) and `df = survey_df_final`; 'group' rows carry per-row `df_used` captured at each row's `safe_inference` (the replicate override rewrites it, the bootstrap override clears it, the all-NaN cohort branch writes no key - consumers read via `.get`); 'event_study' rides the shared `_from_relative_dict` builder via a carrier whose metadata is a copy-on-use of the KIT's fit-final metadata copy. REPLICATE-WEIGHT fits replay the extracted `_replicate_override_aggregates` with a LEVEL-MATCHED stack: `compute_replicate_refit_variance` validates replicates JOINTLY (all-finite rows), so `aggregate(L)` reproduces `fit(aggregate=L)` exactly, a `fit(aggregate='all')` surface is NOT the equivalence target when a replicate NaNs on exactly one family's targets, and - the documented migration delta - moving a replicate fit from `fit(aggregate=)` to plain fit + post-fit `aggregate()` can change the public OVERALL row's se/CI/df on such degenerate designs (each surface self-consistent; pinned in the contract tests). `pretrends=True` + replicate: post-fit `aggregate('event_study')` raises the same NotImplementedError the fit-time gate raises (per-replicate lead refits unimplemented); 'group'/'simple' still work. Recompute re-emits the fit-time warnings (LSMR, Prop-5, empty-window) with fit-tuned stacklevels - post-fit attribution lands on a library frame, an accepted verbatim-move trade-off. `DiagnosticReport` now derives this container internally on plain fits (raw `event_study_effects` absent), so its `parallel_trends` (`pretrends=True` fits) and `heterogeneity` checks run without the deprecated fit-time kwarg; the re-emitted recompute warnings are captured and re-published on the consuming report section (record-and-republish), and derivation failures (bootstrap, missing kit, the replicate gate above) surface as explicit per-check skip reasons. --- @@ -2065,7 +2066,7 @@ Our implementation uses multiplier bootstrap on the GMM influence function: clus - [x] Multiplier bootstrap on GMM influence function - [x] Event study and overall ATT aggregation -- **Note (post-fit aggregate() - rows M-022/M-119):** `fit(aggregate=, balance_e=)` is deprecated (3.9; removed 4.0; joint FutureWarning, warn-and-still-work; the `two_stage_did` wrapper forwards the shared sentinel so plain wrapper calls never fire the aggregate warning; since 3.9 the wrapper itself warns per M-071) in favor of post-fit `TwoStageDiDResults.aggregate(type, balance_e=)` - a PANEL-BACKED lazy recompute kit: each level is a fresh Stage-2 OLS + joint Gardner-GMM sandwich on a level-specific design, so no compact influence payload exists. (a) RETAINED BUFFERS (memory contract - the FIRST panel retention on TwoStageDiD results, a deliberate break from the CS/EDiD identifier-minimization guarantee, with a `store_kit` opt-out tracked in DEFERRED.md): a COLUMN-SUBSET COPY of the working frame - `unit`/`time`/`outcome`/`first_treat` + covariates + the cluster column (deduplicated: `cluster=` may legally name a core column) + `_never_treated`/`_rel_time`/`_y_tilde` - O(n_obs) on every results object and pickle; the Stage-1 FE model (`unit_fe`/`time_fe`/`grand_mean`/`delta_hat`/`kept_cov_mask`), the Omega masks, the full-domain `keep_mask`, the Wave-E.3-GATED `score_pad_mask`/`cluster_ids_full` values fit actually passed (None unless the always-treated pad was active), `survey_weights`, and the resolved survey design - on replicate designs that adds the O(n_obs x R) replicate matrix; plus value snapshots (`treatment_groups` copy, `ref_period`, `overall_att`, `n_treated_obs`, a `dataclasses.replace` copy of `survey_metadata`) and TWO df scalars (`survey_df_stage2`, the recompute seed; `survey_df_final`, what the stored overall inference received). (b) `balance_e` uses the BALANCED-WINDOW rule (`[-balance_e, max_h]` coverage against the full panel - the ImputationDiD rule, divergent from CS/EfficientDiD's anchor-horizon rule); zero qualifying cohorts warns and yields the reference-row-only dict with `vcov=None`. (c) BOOTSTRAP fits: 'simple' RELAYS the stored overall quintet verbatim (finite safe_inference t included) with a NaN df column, while the RECOMPUTE levels fail closed (per-level GMM scores are function-locals; replay is a TODO row) - the prior uniform rule superseded 2026-08-05 with the M-027 per-level convergence; a fit whose bootstrap FAILED (`bootstrap_results=None`, analytical inference retained) aggregates normally. (d) CONTAINER ADMISSION into `compute_honest_did`/`compute_pretrends_power` is DEFERRED, not by-design (both terminal TypeErrors state it): analytical surfaces DO carry the real joint Gardner-GMM covariance (M-092), but the pre-period coefficients are stage-1 residual MEANS - the reference horizon is dropped from the no-intercept Stage-2 design and the zero anchor row is appended mechanically - not contrasts against the advertised reference, while HonestDiD's Delta^RM/Delta^SD arithmetic hard-codes the `delta_0 = 0` normalization into its boundary/bridge constraints; admission awaits a normalization derivation (either re-estimating Stage 2 with the reference horizon in the design or deriving the residual-to-reference mapping) - the DEFERRED.md paper-gated row. (e) RELAY CONVENTIONS: 'simple' relays the stored overall quintet bit-exact with `n = n_treated_obs`, `n_kind="obs"` (overlapping unit sets - the StackedDiD carve-out class; the pre-filter |Omega_1| count, while the ATT's Stage-2 support excludes rows whose `y_tilde` is non-finite - on such degenerate fits `n` exceeds the averaged support) and `df = survey_df_final` (on replicate fits that value came from the `[overall]`-only joint stack - snapshotted, never re-derived); 'group' relays a SCALAR df broadcast (deliberate divergence from ImputationDiD's per-row `df_used`: `_stage2_group` passes one immutable `survey_df` to every row's `safe_inference`, so the scalar is provenance-exact by construction and the moved method stays verbatim); 'event_study' reproduces the M-092 container contract exactly - analytical fits thread the recomputed joint vcov + `vcov_index` + the finite-and->0 df scalar through the carrier, replicate fits thread `vcov=None`/`index=None` with the REPLAYED level-matched df, and the carrier's metadata is a copy-on-use of the KIT's fit-final metadata copy. REPLICATE-WEIGHT fits replay the extracted `_replay_replicate_inference` with a LEVEL-MATCHED stack (the ImputationDiD semantics: `aggregate(L)` reproduces `fit(aggregate=L)`; `fit(aggregate='all')` is not the equivalence target on degenerate designs; the OVERALL-row migration delta on such designs is documented and pinned). Recompute re-emits fit-time warnings with fit-tuned stacklevels - an accepted verbatim-move trade-off. +- **Note (post-fit aggregate() - rows M-022/M-119):** `fit(aggregate=, balance_e=)` is deprecated (3.9; removed 4.0; joint FutureWarning, warn-and-still-work; the `two_stage_did` wrapper forwards the shared sentinel so plain wrapper calls never fire the aggregate warning; since 3.9 the wrapper itself warns per M-071) in favor of post-fit `TwoStageDiDResults.aggregate(type, balance_e=)` - a PANEL-BACKED lazy recompute kit: each level is a fresh Stage-2 OLS + joint Gardner-GMM sandwich on a level-specific design, so no compact influence payload exists. (a) RETAINED BUFFERS (memory contract - the FIRST panel retention on TwoStageDiD results, a deliberate break from the CS/EDiD identifier-minimization guarantee, with a `store_kit` opt-out tracked in DEFERRED.md): a COLUMN-SUBSET COPY of the working frame - `unit`/`time`/`outcome`/`first_treat` + covariates + the cluster column (deduplicated: `cluster=` may legally name a core column) + `_never_treated`/`_rel_time`/`_y_tilde` - O(n_obs) on every results object and pickle; the Stage-1 FE model (`unit_fe`/`time_fe`/`grand_mean`/`delta_hat`/`kept_cov_mask`), the Omega masks, the full-domain `keep_mask`, the Wave-E.3-GATED `score_pad_mask`/`cluster_ids_full` values fit actually passed (None unless the always-treated pad was active), `survey_weights`, and the resolved survey design - on replicate designs that adds the O(n_obs x R) replicate matrix; plus value snapshots (`treatment_groups` copy, `ref_period`, `overall_att`, `n_treated_obs`, a `dataclasses.replace` copy of `survey_metadata`) and TWO df scalars (`survey_df_stage2`, the recompute seed; `survey_df_final`, what the stored overall inference received). (b) `balance_e` uses the BALANCED-WINDOW rule (`[-balance_e, max_h]` coverage against the full panel - the ImputationDiD rule, divergent from CS/EfficientDiD's anchor-horizon rule); zero qualifying cohorts warns and yields the reference-row-only dict with `vcov=None`. (c) BOOTSTRAP fits: 'simple' RELAYS the stored overall quintet verbatim (finite safe_inference t included) with a NaN df column, while the RECOMPUTE levels fail closed (per-level GMM scores are function-locals; replay is a TODO row) - the prior uniform rule superseded 2026-08-05 with the M-027 per-level convergence; a fit whose bootstrap FAILED (`bootstrap_results=None`, analytical inference retained) aggregates normally. (d) CONTAINER ADMISSION into `compute_honest_did`/`compute_pretrends_power` is DEFERRED, not by-design (both terminal TypeErrors state it): analytical surfaces DO carry the real joint Gardner-GMM covariance (M-092), but the pre-period coefficients are stage-1 residual MEANS - the reference horizon is dropped from the no-intercept Stage-2 design and the zero anchor row is appended mechanically - not contrasts against the advertised reference, while HonestDiD's Delta^RM/Delta^SD arithmetic hard-codes the `delta_0 = 0` normalization into its boundary/bridge constraints; admission awaits a normalization derivation (either re-estimating Stage 2 with the reference horizon in the design or deriving the residual-to-reference mapping) - the DEFERRED.md paper-gated row. (e) RELAY CONVENTIONS: 'simple' relays the stored overall quintet bit-exact with `n = n_treated_obs`, `n_kind="obs"` (overlapping unit sets - the StackedDiD carve-out class; the pre-filter |Omega_1| count, while the ATT's Stage-2 support excludes rows whose `y_tilde` is non-finite - on such degenerate fits `n` exceeds the averaged support) and `df = survey_df_final` (on replicate fits that value came from the `[overall]`-only joint stack - snapshotted, never re-derived); 'group' relays a SCALAR df broadcast (deliberate divergence from ImputationDiD's per-row `df_used`: `_stage2_group` passes one immutable `survey_df` to every row's `safe_inference`, so the scalar is provenance-exact by construction and the moved method stays verbatim); 'event_study' reproduces the M-092 container contract exactly - analytical fits thread the recomputed joint vcov + `vcov_index` + the finite-and->0 df scalar through the carrier, replicate fits thread `vcov=None`/`index=None` with the REPLAYED level-matched df, and the carrier's metadata is a copy-on-use of the KIT's fit-final metadata copy. REPLICATE-WEIGHT fits replay the extracted `_replay_replicate_inference` with a LEVEL-MATCHED stack (the ImputationDiD semantics: `aggregate(L)` reproduces `fit(aggregate=L)`; `fit(aggregate='all')` is not the equivalence target on degenerate designs; the OVERALL-row migration delta on such designs is documented and pinned). Recompute re-emits fit-time warnings with fit-tuned stacklevels - an accepted verbatim-move trade-off. `DiagnosticReport` now derives this container internally on plain fits (raw `event_study_effects` absent), so its `parallel_trends` (`pretrends=True` fits; the recomputed joint vcov drives the joint-Wald path) and `heterogeneity` checks run without the deprecated fit-time kwarg; re-emitted recompute warnings are captured and re-published on the consuming report section, and derivation failures surface as explicit per-check skip reasons. --- diff --git a/docs/methodology/REPORTING.md b/docs/methodology/REPORTING.md index 8e5795b9c..2c7abf2f1 100644 --- a/docs/methodology/REPORTING.md +++ b/docs/methodology/REPORTING.md @@ -16,11 +16,35 @@ here rather than duplicating content. Both modules dispatch by `type(results).__name__` lookup to avoid circular imports across the 16 result classes. They do no estimator -fitting and do not re-derive any variance from raw data; every effect, -SE, p-value, CI, and sensitivity bound is either read from the fitted -result or produced by an existing diff-diff utility -(`compute_honest_did`, `HonestDiD.sensitivity`, `BaconDecomposition`, -`check_parallel_trends`, `compute_pretrends_power`). When the caller +fitting; every effect, SE, p-value, CI, and sensitivity bound is +either read from the fitted result, derived from the result's own +post-fit `aggregate('event_study')` surface, or produced by an +existing diff-diff utility (`compute_honest_did`, +`HonestDiD.sensitivity`, `BaconDecomposition`, +`check_parallel_trends`, `compute_pretrends_power`). The post-fit +derivation is the one report-layer data source beyond direct reads: +when a fit's raw `event_study_effects` field is absent (the modern, +no-fit-time-`aggregate=` path), `DiagnosticReport` resolves +`results.aggregate('event_study')` once (cached) and consumes the +returned `EventStudyResults` container, where applicable per +estimator: CallawaySantAnna for parallel_trends / pretrends_power / +sensitivity, ImputationDiD and TwoStageDiD for parallel_trends +(`pretrends=True` fits) and heterogeneity, ContinuousDiD for +heterogeneity. StackedDiD, SunAbraham, and dCDH never reach the +derived route (raw surface always populated, or the +`placebo_event_study` branch). For CS the derivation is an +influence-kit recompute; ImputationDiD's +kit is a panel-backed Theorem-3 recompute from the retained working +panel, and TwoStageDiD's runs a fresh Stage-2 OLS + GMM sandwich over +a retained frame copy (ledger rows M-021/M-022) — so those two +producers DO recompute variance from their retained kits, exactly as +their own post-fit `aggregate()` does. Bootstrapped and kit-less fits +fail closed: the derivation exception is caught and surfaced as an +explicit per-check skip reason, never substituted with analytical +numbers. The raw field, when present — including the +requested-but-empty `{}` sentinel, which encodes fit-time +configuration such as a `balance_e=` that emptied the window — always +takes precedence and is never re-derived. When the caller passes the raw panel + column kwargs, `DiagnosticReport` may call those utilities on the supplied data (2x2 PT via `check_parallel_trends`, Goodman-Bacon decomposition via @@ -186,7 +210,10 @@ notably `CallawaySantAnna`, `ImputationDiD`, `TwoStageDiD`, and (or `att` / `avg_att`) scalar is ALWAYS the simple weighted aggregation; post-fit `results.aggregate()` (the successor to the deprecated fit-time `aggregate` kwarg, rows M-020..M-027) returns -the horizon / group tables without changing the headline scalar. Disambiguating those tables in prose is +the horizon / group tables without changing the headline scalar — +and `DiagnosticReport` now consumes those post-fit tables +internally for its event-study-gated checks when the raw fields are +absent. Disambiguating those tables in prose is tracked under BR/DR gap #9 (per-cohort narrative rendering). `ContinuousDiDResults` emits a single `"dose_overall"` tag with a @@ -217,8 +244,18 @@ a library setting. event-study or staggered result objects. `check_parallel_trends` in `diff_diff/utils.py` assumes a single binary treatment with universal pre-periods; for staggered and event-study designs, DR reads the - pre-period event-study coefficients directly and constructs a joint - Wald statistic (or Bonferroni fallback when `vcov` is missing). This + pre-period event-study coefficients directly off the fitted result — + or, when the raw `event_study_effects` field is absent, off the + internally derived post-fit `aggregate('event_study')` container, + which carries the same coefficients — and constructs a joint + Wald statistic (or Bonferroni fallback when `vcov` is missing). On + the derived route the PT / pre-trends-power / sensitivity sections + carry the additive schema key + `pre_period_source = "aggregate_event_study"` (raw-field fits omit + the key, mirroring the `df_denom` additive-key convention), and any + warnings the kit recompute re-emits are captured and re-published on + the consuming section's `warnings` list (record-and-republish; never + swallowed). This mirrors the guidance in `practitioner._parallel_trends_step(staggered=True)`. - **Note:** Survey-design threading for fit-faithful Bacon replay. @@ -240,9 +277,10 @@ a library setting. design even when it is available. Users must pass `precomputed={'parallel_trends': ...}` with a survey-aware pretest result to opt in. Event-study PT on staggered estimators is - unaffected — it reads the weighted pre-period coefficients directly - off the fitted result and uses the finite-df reference described - below, so no second replay is needed. + unaffected — it reads the weighted pre-period coefficients off the + fitted result (or off its internally derived post-fit container, + which carries the same weighted coefficients) and uses the finite-df + reference described below, so no second replay is needed. - **Note:** Survey finite-df PT policy. When the fitted result carries a finite `survey_metadata.df_survey`, `_pt_event_study` computes @@ -392,10 +430,18 @@ a library setting. diagonal), TwoStageDiD on the analytical paths only (bootstrap and replicate-weight modes clear it). Where the covariance is present, the PT check takes the joint-Wald path (subject to the hc2_bm - policy and rank guard below). Pretrends POWER: natively + policy and rank guard below); on the derived route the same + covariance arrives via the container's `vcov` / `vcov_index` + fields. Pretrends POWER: natively `compute_pretrends_power()` supports MPD / CS / SunAbraham fits; since row M-024 a Stacked `results.aggregate('event_study')` - CONTAINER also admits (kappa_pre >= 2). Stacked/TwoStage NATIVE + CONTAINER also admits (kappa_pre >= 2). On plain (no fit-time + `aggregate=`) CS fits, `DiagnosticReport` internally derives the CS + container and feeds it to `compute_pretrends_power` / + `HonestDiD.sensitivity_analysis` — CS is M-093-admitted with pinned + raw-route parity; the admission set itself is unchanged (widening + is a per-estimator methodology decision, ledger row M-093). + Stacked/TwoStage NATIVE results remain outside DR's power applicability - within DiagnosticReport their covariance is consumed by the PT check and by the PRECOMPUTED-power provenance classifier only (a stored power diff --git a/docs/migration-4.0.md b/docs/migration-4.0.md index d855bdd7d..bd0094a1e 100644 --- a/docs/migration-4.0.md +++ b/docs/migration-4.0.md @@ -142,6 +142,18 @@ results = CallawaySantAnna().fit(data, ...) event_study = results.aggregate("event_study") ``` +`DiagnosticReport` / `BusinessReport` need no migration step of their own: +where applicable, their event-study-gated checks derive the surface +internally via the result's post-fit `aggregate('event_study')` when the +raw `event_study_effects` field is absent, so moving a fit off the +fit-time keyword no longer silently disables those checks. The routing is +estimator-specific: CallawaySantAnna derives for parallel trends, +pre-trends power, and sensitivity; ImputationDiD and TwoStageDiD for +parallel trends (`pretrends=True` fits) and heterogeneity; ContinuousDiD +for heterogeneity. StackedDiD and SunAbraham always populate the raw +surface (nothing to derive), and ChaisemartinDHaultfoeuille's pre-period +checks read `placebo_event_study` directly. + ```{warning} **Bootstrapped fits have no route yet.** On `CallawaySantAnna`, `ImputationDiD`, `TwoStageDiD`, `EfficientDiD` and `ContinuousDiD`, the post-fit recompute levels raise `NotImplementedError` diff --git a/docs/v4-deprecations.yaml b/docs/v4-deprecations.yaml index 172cee1d6..0a8bbea0c 100644 --- a/docs/v4-deprecations.yaml +++ b/docs/v4-deprecations.yaml @@ -228,8 +228,8 @@ rows: phase: 5 warning: FutureWarning test_ref: tests/test_aggregate_contract.py - code_refs: [diff_diff/staggered.py, diff_diff/staggered_results.py, diff_diff/aggregation.py, diff_diff/practitioner.py, diff_diff/guides/llms-practitioner.txt] - notes: "Shimmed in 3.9: fit(aggregate=) warns via a sentinel default (so a plain fit() never warns) and still returns the fully populated legacy surface; results.aggregate(type=) is the successor. balance_e moves alongside it as its own row [M-117] - it was previously tracked only as prose here, which nothing asserted. VOCABULARY: the closed set is library-wide (simple|event_study|group|calendar); CallawaySantAnna's SUPPORTED SUBSET is simple|event_study|group - it has no calendar aggregator (the DEFERRED 'Calendar-time aggregation' row), and aggregate('calendar') raises naming what is supported. BOOTSTRAP fits: 'simple' RELAYS the stored overall quintet verbatim (percentile se/p/CI beside the finite safe_inference t) with a NaN df column, while the recompute levels (event_study/group) fail closed pending draw retention (BootstrapReplaySpec is the TODO row) - the per-level policy converged with [M-027] (previously ALL levels failed closed; the relay never publishes an analytical df beside percentile inference)." + code_refs: [diff_diff/staggered.py, diff_diff/staggered_results.py, diff_diff/aggregation.py, diff_diff/practitioner.py, diff_diff/guides/llms-practitioner.txt, diff_diff/diagnostic_report.py] + notes: "Shimmed in 3.9: fit(aggregate=) warns via a sentinel default (so a plain fit() never warns) and still returns the fully populated legacy surface; results.aggregate(type=) is the successor. balance_e moves alongside it as its own row [M-117] - it was previously tracked only as prose here, which nothing asserted. VOCABULARY: the closed set is library-wide (simple|event_study|group|calendar); CallawaySantAnna's SUPPORTED SUBSET is simple|event_study|group - it has no calendar aggregator (the DEFERRED 'Calendar-time aggregation' row), and aggregate('calendar') raises naming what is supported. BOOTSTRAP fits: 'simple' RELAYS the stored overall quintet verbatim (percentile se/p/CI beside the finite safe_inference t) with a NaN df column, while the recompute levels (event_study/group) fail closed pending draw retention (BootstrapReplaySpec is the TODO row) - the per-level policy converged with [M-027] (previously ALL levels failed closed; the relay never publishes an analytical df beside percentile inference). DiagnosticReport now derives the event-study surface via post-fit aggregate('event_study') when the raw field is absent, so its ES-gated checks run on plain fits (derivation failures surface as explicit skip reasons)." - id: M-021 kind: param group: aggregate-postfit @@ -242,8 +242,8 @@ rows: phase: 5 warning: FutureWarning test_ref: tests/test_aggregate_contract.py - code_refs: [diff_diff/imputation.py, diff_diff/imputation_aggregation.py, diff_diff/imputation_results.py, diff_diff/imputation_bootstrap.py, diff_diff/aggregation.py, diff_diff/results_base.py, diff_diff/honest_did.py, diff_diff/pretrends.py, diff_diff/practitioner.py, diff_diff/guides/llms-practitioner.txt] - notes: "Shimmed in 3.9: fit(aggregate=) warns via the shared NOT_SUPPLIED sentinel (plain fit() never warns; supplying ANY value, None included, warns - CS-style joint warning with balance_e [M-118], warn-and-still-work; the imputation_did wrapper forwards the sentinel so a plain wrapper call never fires the aggregate warning; since 3.9 the wrapper itself warns per [M-070]). NO fit-time value validation existed and none is added (unknown strings silently act like None; the post-fit successor fails closed via the mixin vocabulary - a behavior improvement). The successor is a PANEL-BACKED lazy recompute kit: ES/group aggregation is a target-specific Theorem-3 recompute from the working panel + untreated FE model (no compact influence payload can honor a different balance_e), so the kit's bookkeeping holds REFERENCES to the SAME per-fit objects self._fit_data already retains for pretrend_test() - ZERO marginal memory and unchanged pickles via memoization (the _estimator_ref field already ships the panel; enumeration in the REGISTRY ImputationDiD Note). Value snapshots (treatment_groups copy, config scalars, a dataclasses.replace copy of survey_metadata, and the survey_df_seed/survey_df_final df channels) isolate recompute and the ES carrier from public-field mutation; aggregate() reads NOTHING mutable off the results object except the deliberate overall-quintet relay and the bootstrap_results gate. SUPPORTED SUBSET simple|event_study|group; calendar/'all' fail closed via the mixin; weights= rejected. Bootstrap fits: 'simple' RELAYS the stored percentile quintet verbatim (finite safe_inference t included) with a NaN df column, while the recompute levels fail closed (the per-target psi machinery makes replay tractable - a TODO row). The prior fail-closed-for-ALL-levels uniform-parity decision was superseded 2026-08-05 with the [M-027] per-level convergence; its rationale - never publish analytical provenance beside percentile inference - is honored by the NaN df column. Replicate-weight fits REPLAY the extracted _replicate_override_aggregates with a LEVEL-MATCHED stack ([overall, ES] or [overall, groups]): compute_replicate_refit_variance validates replicates jointly, so aggregate(L) reproduces fit(aggregate=L) exactly and a fit(aggregate='all') surface is NOT the equivalence target when a replicate NaNs on one family's targets; the same joint-stack coupling means migrating a replicate fit from fit(aggregate=) to plain fit changes the OVERALL row's se/CI/df on degenerate designs (documented migration delta, CHANGELOG + REGISTRY note (e) + a contract-test pin). pretrends=True + replicate: post-fit aggregate('event_study') raises the same NotImplementedError the fit-time gate raises (per-replicate lead refits unimplemented); group/simple still work. Simple relay: n = n_treated_obs with n_kind='obs' (the treated/control unit sets OVERLAP, so the CS/EDiD disjoint-units convention cannot apply - the StackedDiD carve-out class); df = the survey_df_final snapshot (what the stored overall inference received). Group rows record per-row df_used at each safe_inference call (additive row-dict key; the replicate override rewrites it, the bootstrap override clears it, the all-NaN cohort branch writes no key - consumers read via .get). The M-127 df_convention inert-config warning predicate is REVISED to reachability (pretrends AND not-replicate AND (deprecated fit-time ES/all supplied OR n_bootstrap <= 0)) because post-fit aggregate() made the old aggregate-keyed claim false; reachability-BASED, not exact - a fit whose bootstrap later fails (bootstrap_results=None) can still aggregate post-fit, so that corner warns spuriously (recorded on M-127 too). Container admission NOT widened: ImputationDiD is rejected BY DESIGN (no joint ES covariance - per-horizon conservative SEs only; see M-093). balance_e moves as its own row [M-118]." + code_refs: [diff_diff/imputation.py, diff_diff/imputation_aggregation.py, diff_diff/imputation_results.py, diff_diff/imputation_bootstrap.py, diff_diff/aggregation.py, diff_diff/results_base.py, diff_diff/honest_did.py, diff_diff/pretrends.py, diff_diff/practitioner.py, diff_diff/guides/llms-practitioner.txt, diff_diff/diagnostic_report.py] + notes: "Shimmed in 3.9: fit(aggregate=) warns via the shared NOT_SUPPLIED sentinel (plain fit() never warns; supplying ANY value, None included, warns - CS-style joint warning with balance_e [M-118], warn-and-still-work; the imputation_did wrapper forwards the sentinel so a plain wrapper call never fires the aggregate warning; since 3.9 the wrapper itself warns per [M-070]). NO fit-time value validation existed and none is added (unknown strings silently act like None; the post-fit successor fails closed via the mixin vocabulary - a behavior improvement). The successor is a PANEL-BACKED lazy recompute kit: ES/group aggregation is a target-specific Theorem-3 recompute from the working panel + untreated FE model (no compact influence payload can honor a different balance_e), so the kit's bookkeeping holds REFERENCES to the SAME per-fit objects self._fit_data already retains for pretrend_test() - ZERO marginal memory and unchanged pickles via memoization (the _estimator_ref field already ships the panel; enumeration in the REGISTRY ImputationDiD Note). Value snapshots (treatment_groups copy, config scalars, a dataclasses.replace copy of survey_metadata, and the survey_df_seed/survey_df_final df channels) isolate recompute and the ES carrier from public-field mutation; aggregate() reads NOTHING mutable off the results object except the deliberate overall-quintet relay and the bootstrap_results gate. SUPPORTED SUBSET simple|event_study|group; calendar/'all' fail closed via the mixin; weights= rejected. Bootstrap fits: 'simple' RELAYS the stored percentile quintet verbatim (finite safe_inference t included) with a NaN df column, while the recompute levels fail closed (the per-target psi machinery makes replay tractable - a TODO row). The prior fail-closed-for-ALL-levels uniform-parity decision was superseded 2026-08-05 with the [M-027] per-level convergence; its rationale - never publish analytical provenance beside percentile inference - is honored by the NaN df column. Replicate-weight fits REPLAY the extracted _replicate_override_aggregates with a LEVEL-MATCHED stack ([overall, ES] or [overall, groups]): compute_replicate_refit_variance validates replicates jointly, so aggregate(L) reproduces fit(aggregate=L) exactly and a fit(aggregate='all') surface is NOT the equivalence target when a replicate NaNs on one family's targets; the same joint-stack coupling means migrating a replicate fit from fit(aggregate=) to plain fit changes the OVERALL row's se/CI/df on degenerate designs (documented migration delta, CHANGELOG + REGISTRY note (e) + a contract-test pin). pretrends=True + replicate: post-fit aggregate('event_study') raises the same NotImplementedError the fit-time gate raises (per-replicate lead refits unimplemented); group/simple still work. Simple relay: n = n_treated_obs with n_kind='obs' (the treated/control unit sets OVERLAP, so the CS/EDiD disjoint-units convention cannot apply - the StackedDiD carve-out class); df = the survey_df_final snapshot (what the stored overall inference received). Group rows record per-row df_used at each safe_inference call (additive row-dict key; the replicate override rewrites it, the bootstrap override clears it, the all-NaN cohort branch writes no key - consumers read via .get). The M-127 df_convention inert-config warning predicate is REVISED to reachability (pretrends AND not-replicate AND (deprecated fit-time ES/all supplied OR n_bootstrap <= 0)) because post-fit aggregate() made the old aggregate-keyed claim false; reachability-BASED, not exact - a fit whose bootstrap later fails (bootstrap_results=None) can still aggregate post-fit, so that corner warns spuriously (recorded on M-127 too). Container admission NOT widened: ImputationDiD is rejected BY DESIGN (no joint ES covariance - per-horizon conservative SEs only; see M-093). balance_e moves as its own row [M-118]. DiagnosticReport now derives the event-study surface via post-fit aggregate('event_study') when the raw field is absent, so its ES-gated checks run on plain fits (derivation failures surface as explicit skip reasons)." - id: M-022 kind: param group: aggregate-postfit @@ -256,8 +256,8 @@ rows: phase: 5 warning: FutureWarning test_ref: tests/test_aggregate_contract.py - code_refs: [diff_diff/two_stage.py, diff_diff/two_stage_aggregation.py, diff_diff/two_stage_results.py, diff_diff/two_stage_bootstrap.py, diff_diff/aggregation.py, diff_diff/results_base.py, diff_diff/honest_did.py, diff_diff/pretrends.py, diff_diff/practitioner.py, diff_diff/guides/llms-practitioner.txt] - notes: "Shimmed in 3.9: fit(aggregate=) warns via the shared NOT_SUPPLIED sentinel (plain fit() never warns; supplying ANY value, None included, warns - CS-style joint warning with balance_e [M-119], warn-and-still-work; the two_stage_did wrapper forwards the sentinel so a plain wrapper call never fires the aggregate warning; since 3.9 the wrapper itself warns per [M-071]). NO fit-time value validation existed and none is added (unknown strings silently act like None; the post-fit successor fails closed via the mixin vocabulary). The successor is a PANEL-BACKED lazy recompute kit: each level is a fresh Stage-2 OLS + joint Gardner-GMM sandwich, so the kit retains a COLUMN-SUBSET COPY of the working frame (only the columns the moved methods read by name, deduplicated - cluster= may legally name the unit/time/first_treat column) plus the Stage-1 FE model, masks, and survey objects. MEMORY CONTRACT: this is the FIRST panel retention on TwoStageDiD results - O(n_obs) incl. unit/time/cluster identifier columns on every results object and pickle, and replicate designs additionally retain the (n_obs x R) replicate matrix via resolved_survey; the CS/EDiD identifier-minimization guarantee deliberately does NOT hold (a store_kit opt-out is a DEFERRED row). score_pad_mask/cluster_ids_full are stored as the Wave-E.3-GATED values fit actually passed. Value snapshots (treatment_groups copy, overall_att, survey_df_stage2/survey_df_final, a dataclasses.replace copy of survey_metadata) isolate recompute and the ES carrier from public-field mutation. SUPPORTED SUBSET simple|event_study|group; calendar/'all' fail closed; weights= rejected. Bootstrap fits: 'simple' RELAYS the stored percentile quintet verbatim (finite safe_inference t included) with a NaN df column, while the recompute levels fail closed (per-level GMM scores are function-locals; replay is a TODO row) - the prior uniform fail-closed decision superseded 2026-08-05 with the [M-027] per-level convergence; a fit whose bootstrap FAILED (bootstrap_results=None, analytical inference retained) aggregates normally. Replicate-weight fits REPLAY the extracted _replay_replicate_inference with a LEVEL-MATCHED stack - aggregate(L) reproduces fit(aggregate=L) exactly; fit(aggregate='all') is NOT the equivalence target on degenerate designs, and the same joint-stack coupling makes the migration to plain fit change the OVERALL row's se/CI/df there (documented migration delta). Post-fit aggregate('event_study') reproduces the M-092 container contract exactly: analytical fits thread the recomputed joint vcov + vcov_index + the finite-and->0 df scalar through the carrier; replicate fits thread vcov=None/index=None with the REPLAYED level-matched df. Simple relay: n = n_treated_obs with n_kind='obs' (overlapping unit sets - StackedDiD carve-out class); df = the survey_df_final snapshot (on replicate fits that value came from the [overall]-only stack - snapshotted, never re-derived). Group relay df is a SCALAR broadcast (deliberate divergence from ImputationDiD's per-row df_used: _stage2_group passes one immutable survey_df to every row, so the scalar is provenance-exact by construction and the moved method stays verbatim). Container admission NOT widened: DEFERRED pending a normalization derivation - analytical surfaces carry the real joint Gardner-GMM covariance (M-092), but pre-period coefficients are stage-1 residual means, not reference-normalized contrasts, while HonestDiD's Delta arithmetic hard-codes delta_0=0 (see M-093 + the DEFERRED.md paper-gated row). balance_e moves as its own row [M-119]." + code_refs: [diff_diff/two_stage.py, diff_diff/two_stage_aggregation.py, diff_diff/two_stage_results.py, diff_diff/two_stage_bootstrap.py, diff_diff/aggregation.py, diff_diff/results_base.py, diff_diff/honest_did.py, diff_diff/pretrends.py, diff_diff/practitioner.py, diff_diff/guides/llms-practitioner.txt, diff_diff/diagnostic_report.py] + notes: "Shimmed in 3.9: fit(aggregate=) warns via the shared NOT_SUPPLIED sentinel (plain fit() never warns; supplying ANY value, None included, warns - CS-style joint warning with balance_e [M-119], warn-and-still-work; the two_stage_did wrapper forwards the sentinel so a plain wrapper call never fires the aggregate warning; since 3.9 the wrapper itself warns per [M-071]). NO fit-time value validation existed and none is added (unknown strings silently act like None; the post-fit successor fails closed via the mixin vocabulary). The successor is a PANEL-BACKED lazy recompute kit: each level is a fresh Stage-2 OLS + joint Gardner-GMM sandwich, so the kit retains a COLUMN-SUBSET COPY of the working frame (only the columns the moved methods read by name, deduplicated - cluster= may legally name the unit/time/first_treat column) plus the Stage-1 FE model, masks, and survey objects. MEMORY CONTRACT: this is the FIRST panel retention on TwoStageDiD results - O(n_obs) incl. unit/time/cluster identifier columns on every results object and pickle, and replicate designs additionally retain the (n_obs x R) replicate matrix via resolved_survey; the CS/EDiD identifier-minimization guarantee deliberately does NOT hold (a store_kit opt-out is a DEFERRED row). score_pad_mask/cluster_ids_full are stored as the Wave-E.3-GATED values fit actually passed. Value snapshots (treatment_groups copy, overall_att, survey_df_stage2/survey_df_final, a dataclasses.replace copy of survey_metadata) isolate recompute and the ES carrier from public-field mutation. SUPPORTED SUBSET simple|event_study|group; calendar/'all' fail closed; weights= rejected. Bootstrap fits: 'simple' RELAYS the stored percentile quintet verbatim (finite safe_inference t included) with a NaN df column, while the recompute levels fail closed (per-level GMM scores are function-locals; replay is a TODO row) - the prior uniform fail-closed decision superseded 2026-08-05 with the [M-027] per-level convergence; a fit whose bootstrap FAILED (bootstrap_results=None, analytical inference retained) aggregates normally. Replicate-weight fits REPLAY the extracted _replay_replicate_inference with a LEVEL-MATCHED stack - aggregate(L) reproduces fit(aggregate=L) exactly; fit(aggregate='all') is NOT the equivalence target on degenerate designs, and the same joint-stack coupling makes the migration to plain fit change the OVERALL row's se/CI/df there (documented migration delta). Post-fit aggregate('event_study') reproduces the M-092 container contract exactly: analytical fits thread the recomputed joint vcov + vcov_index + the finite-and->0 df scalar through the carrier; replicate fits thread vcov=None/index=None with the REPLAYED level-matched df. Simple relay: n = n_treated_obs with n_kind='obs' (overlapping unit sets - StackedDiD carve-out class); df = the survey_df_final snapshot (on replicate fits that value came from the [overall]-only stack - snapshotted, never re-derived). Group relay df is a SCALAR broadcast (deliberate divergence from ImputationDiD's per-row df_used: _stage2_group passes one immutable survey_df to every row, so the scalar is provenance-exact by construction and the moved method stays verbatim). Container admission NOT widened: DEFERRED pending a normalization derivation - analytical surfaces carry the real joint Gardner-GMM covariance (M-092), but pre-period coefficients are stage-1 residual means, not reference-normalized contrasts, while HonestDiD's Delta arithmetic hard-codes delta_0=0 (see M-093 + the DEFERRED.md paper-gated row). balance_e moves as its own row [M-119]. DiagnosticReport now derives the event-study surface via post-fit aggregate('event_study') when the raw field is absent, so its ES-gated checks run on plain fits (derivation failures surface as explicit skip reasons)." - id: M-023 kind: param group: aggregate-postfit @@ -298,8 +298,8 @@ rows: phase: 5 warning: FutureWarning test_ref: tests/test_aggregate_contract.py - code_refs: [diff_diff/continuous_did.py, diff_diff/continuous_did_aggregation.py, diff_diff/continuous_did_results.py, diff_diff/aggregation.py, diff_diff/results_base.py, diff_diff/honest_did.py, diff_diff/pretrends.py, diff_diff/practitioner.py, diff_diff/guides/llms-practitioner.txt] - notes: "Shimmed in 3.9: fit(aggregate=) warns via the shared NOT_SUPPLIED sentinel (a plain fit() never warns; supplying ANY value incl. None warns once, then the legacy routing runs unchanged). Unlike the EfficientDiD/Imputation shims, fit-time VALUE VALIDATION pre-existed and STAYS: unknown strings still raise ValueError after the warning ((None, 'dose', 'eventstudy') only). The no-underscore 'eventstudy' spelling dies with the param in 4.0; aggregate() accepts only the unified vocabulary + 'dose' as this estimator's documented extra level. MIXED VIEW/RECOMPUTE architecture (unique in the register): 'simple' and 'dose' are pure VIEWS over stored public fields - the dose curves and the overall binarized ATT (ATT^{loc} under PT; equals ATT^{glob} under SPT) plus ACRT^{glob} are ALWAYS computed by fit (aggregate='dose' was a fit-time no-op) - so both levels are PERMITTED on bootstrap fits (the library-wide per-level relay rule, since [M-027] converged CS/EDiD/Imputation/TwoStage onto it), relaying stored inference verbatim: the overall rows carry the FINITE safe_inference t fit stores beside percentile p/CI, the dose rows reproduce DoseResponseCurve.to_dataframe (NaN t under bootstrap), and only the df column is uniformly NaN there. 'event_study' is a PRUNED-IF-PAYLOAD kit recompute (see the AggregationKit docstring variant): per-(g,t) O(n_treated+n_control) IF-ingredient arrays + unit-level arrays + the PANEL-LEVEL resolved survey design (on replicate designs the (n_obs x R) replicate matrix rides along - a unit-level collapse was reviewed and declined for verbatim-move safety); no panel data columns and no raw unit identifiers are retained; bootstrap fits get a SCALARS-ONLY kit and the ES route fails closed (NotImplementedError naming the fit-time route / n_bootstrap=0 refit; ContinuousDiDResults has no bootstrap_results field, so the config gate is the honest one). Replicate-weight designs ARE supported post-fit (IF-based compute_replicate_if_variance - no refit replay, the contrast with M-021/M-022). Fit-faithful quirk: empty-post_gt fits leave ES rows at NaN inference on both routes (has_post_cells flag). CONTAINER SHAPES: simple = 2 rows (targets att/acrt - the dual-estimand case the AggregationResult target column exists for; n = disjoint treated+control units total, n_kind='units'); dose = 2N target-discriminated rows (labels = the dose grid twice; n NaN / n_kind None / weight None - grid evaluation points carry no count or mass); df from the stored dose_response_att.df_survey channel (finite-and->0 else NaN; the raw stored value incl. the replicate 0-sentinel feeds the to_dataframe-exact t/p derivation). Ships the FIRST heterogeneous-target container, with the AggregationResult summary()/to_dataframe rendering amendment (target column + neutral estimate heading when targets mixed; FIRST-APPEARANCE target-block ordering, labels ascending within block under the _sortable guard; uniform-target producers byte-stable) - normative rule in v4-design section 6. Supported subset simple|event_study|dose - group/calendar fail closed via the mixin; balance_e applies to no level (empty _AGGREGATE_BALANCE_E_TYPES); weights= rejected. Admission: honest/pretrends containers rejected BY DESIGN (no joint ES covariance; bins not reference-normalized - see M-093). Bootstrap-ES post-fit replay is the TODO.md row." + code_refs: [diff_diff/continuous_did.py, diff_diff/continuous_did_aggregation.py, diff_diff/continuous_did_results.py, diff_diff/aggregation.py, diff_diff/results_base.py, diff_diff/honest_did.py, diff_diff/pretrends.py, diff_diff/practitioner.py, diff_diff/guides/llms-practitioner.txt, diff_diff/diagnostic_report.py] + notes: "Shimmed in 3.9: fit(aggregate=) warns via the shared NOT_SUPPLIED sentinel (a plain fit() never warns; supplying ANY value incl. None warns once, then the legacy routing runs unchanged). Unlike the EfficientDiD/Imputation shims, fit-time VALUE VALIDATION pre-existed and STAYS: unknown strings still raise ValueError after the warning ((None, 'dose', 'eventstudy') only). The no-underscore 'eventstudy' spelling dies with the param in 4.0; aggregate() accepts only the unified vocabulary + 'dose' as this estimator's documented extra level. MIXED VIEW/RECOMPUTE architecture (unique in the register): 'simple' and 'dose' are pure VIEWS over stored public fields - the dose curves and the overall binarized ATT (ATT^{loc} under PT; equals ATT^{glob} under SPT) plus ACRT^{glob} are ALWAYS computed by fit (aggregate='dose' was a fit-time no-op) - so both levels are PERMITTED on bootstrap fits (the library-wide per-level relay rule, since [M-027] converged CS/EDiD/Imputation/TwoStage onto it), relaying stored inference verbatim: the overall rows carry the FINITE safe_inference t fit stores beside percentile p/CI, the dose rows reproduce DoseResponseCurve.to_dataframe (NaN t under bootstrap), and only the df column is uniformly NaN there. 'event_study' is a PRUNED-IF-PAYLOAD kit recompute (see the AggregationKit docstring variant): per-(g,t) O(n_treated+n_control) IF-ingredient arrays + unit-level arrays + the PANEL-LEVEL resolved survey design (on replicate designs the (n_obs x R) replicate matrix rides along - a unit-level collapse was reviewed and declined for verbatim-move safety); no panel data columns and no raw unit identifiers are retained; bootstrap fits get a SCALARS-ONLY kit and the ES route fails closed (NotImplementedError naming the fit-time route / n_bootstrap=0 refit; ContinuousDiDResults has no bootstrap_results field, so the config gate is the honest one). Replicate-weight designs ARE supported post-fit (IF-based compute_replicate_if_variance - no refit replay, the contrast with M-021/M-022). Fit-faithful quirk: empty-post_gt fits leave ES rows at NaN inference on both routes (has_post_cells flag). CONTAINER SHAPES: simple = 2 rows (targets att/acrt - the dual-estimand case the AggregationResult target column exists for; n = disjoint treated+control units total, n_kind='units'); dose = 2N target-discriminated rows (labels = the dose grid twice; n NaN / n_kind None / weight None - grid evaluation points carry no count or mass); df from the stored dose_response_att.df_survey channel (finite-and->0 else NaN; the raw stored value incl. the replicate 0-sentinel feeds the to_dataframe-exact t/p derivation). Ships the FIRST heterogeneous-target container, with the AggregationResult summary()/to_dataframe rendering amendment (target column + neutral estimate heading when targets mixed; FIRST-APPEARANCE target-block ordering, labels ascending within block under the _sortable guard; uniform-target producers byte-stable) - normative rule in v4-design section 6. Supported subset simple|event_study|dose - group/calendar fail closed via the mixin; balance_e applies to no level (empty _AGGREGATE_BALANCE_E_TYPES); weights= rejected. Admission: honest/pretrends containers rejected BY DESIGN (no joint ES covariance; bins not reference-normalized - see M-093). Bootstrap-ES post-fit replay is the TODO.md row. DiagnosticReport now derives the event-study surface via post-fit aggregate('event_study') when the raw field is absent, so its ES-gated checks run on plain fits (derivation failures surface as explicit skip reasons)." - id: M-026 kind: param group: aggregate-postfit diff --git a/tests/test_business_report.py b/tests/test_business_report.py index 6965ef894..617ea90b6 100644 --- a/tests/test_business_report.py +++ b/tests/test_business_report.py @@ -5026,3 +5026,57 @@ def test_bacon_recommendation_uses_class_names(self): "estimator (CS / SA / BJS / Gardner)" not in source ), "the Bacon recommendation still uses alias shorthand" assert "ImputationDiD, or TwoStageDiD)." in source + + +class TestDerivedEventStudySurfaceLift: + """BR consumes DR's derived post-fit event-study surface transparently.""" + + @pytest.fixture(scope="class") + def cs_plain_fit_br(self): + # Module-scoped fixtures are not shareable across test modules, so + # this small plain fit is duplicated here rather than promoted to + # conftest (dual-review P2). + sdf = generate_staggered_data(n_units=100, n_periods=6, treatment_effect=1.5, seed=7) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + cs = CallawaySantAnna(base_period="universal").fit( + sdf, outcome="outcome", unit="unit", time="period", first_treat="first_treat" + ) + return cs, sdf + + def test_plain_cs_fit_business_report_pt_verdict(self, cs_plain_fit_br): + cs, sdf = cs_plain_fit_br + br = BusinessReport( + cs, + outcome_label="revenue", + treatment_label="the campaign", + data=sdf, + outcome="outcome", + unit="unit", + time="period", + first_treat="first_treat", + ) + d = br.to_dict() + pt = d["pre_trends"] + assert pt["status"] == "computed" + assert pt["verdict"] is not None + # The provenance key crosses the _lift_pre_trends whitelist. + assert pt["pre_period_source"] == "aggregate_event_study" + + def test_raw_route_business_report_pre_period_source_is_none(self): + # Raw-field (fit-time kwarg) route: BR still emits the key, as None. + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + sdf = generate_staggered_data(n_units=100, n_periods=6, treatment_effect=1.5, seed=7) + cs = CallawaySantAnna(base_period="universal").fit( + sdf, + outcome="outcome", + unit="unit", + time="period", + first_treat="first_treat", + aggregate="event_study", + ) + d = BusinessReport(cs, outcome_label="revenue").to_dict() + pt = d["pre_trends"] + assert pt["status"] == "computed" + assert pt["pre_period_source"] is None diff --git a/tests/test_diagnostic_report.py b/tests/test_diagnostic_report.py index 14ee3708b..53be69b68 100644 --- a/tests/test_diagnostic_report.py +++ b/tests/test_diagnostic_report.py @@ -3413,3 +3413,721 @@ def test_bacon_caveat_uses_class_names(self): source = inspect.getsource(dr_mod) assert "(CS / SA / BJS / Gardner)" not in source assert "heterogeneity-robust estimator (CallawaySantAnna, SunAbraham, " in source + + +# --------------------------------------------------------------------------- +# Derived post-fit event-study surface (results.aggregate('event_study')) +# --------------------------------------------------------------------------- + + +@pytest.fixture(scope="module") +def cs_plain_fit(): + """Plain (modern, no fit-time aggregate=) CS fit — the derived-route fixture.""" + sdf = generate_staggered_data(n_units=100, n_periods=6, treatment_effect=1.5, seed=7) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + cs = CallawaySantAnna(base_period="universal").fit( + sdf, outcome="outcome", unit="unit", time="period", first_treat="first_treat" + ) + return cs, sdf + + +@pytest.fixture(scope="module") +def cs_bootstrap_plain_fit(): + # Explicit hard-coded n_bootstrap (report-suite precedent: + # test_business_report.py's SyntheticDiD bootstrap fixture); the only + # assertion on this fixture is the fail-closed skip, so the iteration + # count carries no numeric meaning and is not ci_params-scaled. + sdf = generate_staggered_data(n_units=100, n_periods=6, treatment_effect=1.5, seed=7) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + cs = CallawaySantAnna(base_period="universal", n_bootstrap=50, seed=1).fit( + sdf, outcome="outcome", unit="unit", time="period", first_treat="first_treat" + ) + return cs, sdf + + +@pytest.fixture(scope="module") +def imp_pretrends_fits(): + """(raw-kwarg fit, plain fit) ImputationDiD pair, both pretrends=True. + + ``pretrends=True`` is REQUIRED for estimated pre-period horizons to + exist at all — under the default ``pretrends=False`` the derived + surface carries only the reference lead. + """ + from diff_diff import ImputationDiD + + sdf = generate_staggered_data(n_units=100, n_periods=6, treatment_effect=1.5, seed=7) + kw = dict(outcome="outcome", unit="unit", time="period", first_treat="first_treat") + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + raw = ImputationDiD(pretrends=True).fit(sdf, aggregate="event_study", **kw) + plain = ImputationDiD(pretrends=True).fit(sdf, **kw) + return raw, plain, sdf + + +@pytest.fixture(scope="module") +def ts_pretrends_fits(): + """(raw-kwarg fit, plain fit) TwoStageDiD pair, both pretrends=True.""" + from diff_diff import TwoStageDiD + + sdf = generate_staggered_data(n_units=100, n_periods=6, treatment_effect=1.5, seed=7) + kw = dict(outcome="outcome", unit="unit", time="period", first_treat="first_treat") + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + raw = TwoStageDiD(pretrends=True).fit(sdf, aggregate="event_study", **kw) + plain = TwoStageDiD(pretrends=True).fit(sdf, **kw) + return raw, plain, sdf + + +class TestDerivedEventStudySurface: + """DR obtains the ES surface from post-fit aggregate('event_study').""" + + def test_plain_cs_fit_runs_event_study_gated_checks(self, cs_plain_fit): + cs, sdf = cs_plain_fit + dr = DiagnosticReport( + cs, data=sdf, outcome="outcome", unit="unit", time="period", first_treat="first_treat" + ) + applicable = set(dr.applicable_checks) + assert {"parallel_trends", "pretrends_power", "sensitivity", "heterogeneity"} <= applicable + d = dr.run_all().to_dict() + for check in ("parallel_trends", "pretrends_power", "sensitivity", "heterogeneity"): + assert d[check]["status"] == "ran", (check, d[check]) + + def test_plain_cs_pt_parity_with_kwarg_route(self, cs_plain_fit, cs_fit): + plain, _ = cs_plain_fit + raw, _ = cs_fit + d_plain = DiagnosticReport(plain).run_all().to_dict()["parallel_trends"] + d_raw = DiagnosticReport(raw).run_all().to_dict()["parallel_trends"] + assert d_plain["method"] == "joint_wald_event_study" + assert d_raw["method"] == "joint_wald_event_study" + # Kit recompute is ~1 ULP off fit-time (BLAS reassociation) — NOT + # bit-identity. + np.testing.assert_allclose( + d_plain["joint_p_value"], d_raw["joint_p_value"], rtol=1e-12, atol=1e-12 + ) + np.testing.assert_allclose( + d_plain["test_statistic"], d_raw["test_statistic"], rtol=1e-12, atol=1e-12 + ) + rows_plain = sorted(d_plain["per_period"], key=lambda p: p["period"]) + rows_raw = sorted(d_raw["per_period"], key=lambda p: p["period"]) + assert [p["period"] for p in rows_plain] == [p["period"] for p in rows_raw] + for a, b in zip(rows_plain, rows_raw): + np.testing.assert_allclose(a["coef"], b["coef"], rtol=1e-12, atol=1e-12) + np.testing.assert_allclose(a["se"], b["se"], rtol=1e-12, atol=1e-12) + + def test_plain_cs_pretrends_power_parity(self, cs_plain_fit, cs_fit): + plain, _ = cs_plain_fit + raw, _ = cs_fit + d_plain = DiagnosticReport(plain).run_all().to_dict()["pretrends_power"] + d_raw = DiagnosticReport(raw).run_all().to_dict()["pretrends_power"] + assert d_plain["status"] == d_raw["status"] == "ran" + assert d_plain["covariance_source"] == "full_pre_period_vcov" + assert d_raw["covariance_source"] == "full_pre_period_vcov" + assert d_plain["tier"] == d_raw["tier"] + # SciPy's Genz MVN CDF is internally randomized (~1e-5 jitter + # between identical calls); 1e-3 is the repo's container-parity pin. + np.testing.assert_allclose(d_plain["mdv"], d_raw["mdv"], rtol=1e-3, atol=1e-3) + np.testing.assert_allclose( + d_plain["power_at_violation_magnitude"], + d_raw["power_at_violation_magnitude"], + rtol=1e-3, + atol=1e-3, + ) + + def test_plain_cs_sensitivity_parity(self, cs_plain_fit, cs_fit): + plain, _ = cs_plain_fit + raw, _ = cs_fit + d_plain = DiagnosticReport(plain).run_all().to_dict()["sensitivity"] + d_raw = DiagnosticReport(raw).run_all().to_dict()["sensitivity"] + assert d_plain["status"] == d_raw["status"] == "ran" + # On this fixture breakdown_M is None on BOTH routes — assert the + # shape equality, never a numeric tolerance on it. + assert d_plain["breakdown_M"] is None + assert d_raw["breakdown_M"] is None + assert len(d_plain["grid"]) == len(d_raw["grid"]) + for a, b in zip(d_plain["grid"], d_raw["grid"]): + np.testing.assert_allclose(a["ci_lower"], b["ci_lower"], rtol=1e-12, atol=1e-12) + np.testing.assert_allclose(a["ci_upper"], b["ci_upper"], rtol=1e-12, atol=1e-12) + + def test_derived_route_emits_pre_period_source(self, cs_plain_fit, cs_fit): + plain, _ = cs_plain_fit + raw, _ = cs_fit + d_plain = DiagnosticReport(plain).run_all().to_dict() + d_raw = DiagnosticReport(raw).run_all().to_dict() + assert d_plain["parallel_trends"]["pre_period_source"] == "aggregate_event_study" + assert d_plain["pretrends_power"]["pre_period_source"] == "aggregate_event_study" + assert d_plain["sensitivity"]["pre_period_source"] == "aggregate_event_study" + assert "pre_period_source" not in d_raw["parallel_trends"] + assert "pre_period_source" not in d_raw["pretrends_power"] + assert "pre_period_source" not in d_raw["sensitivity"] + + def test_bootstrap_plain_cs_skips_with_honest_reason(self, cs_bootstrap_plain_fit): + cs, _ = cs_bootstrap_plain_fit + skipped = DiagnosticReport(cs).run_all().skipped_checks + for check in ("parallel_trends", "pretrends_power", "sensitivity"): + assert check in skipped, skipped + assert "aggregate('event_study')" in skipped[check] + assert "bootstrap" in skipped[check] + assert "deprecated" not in skipped[check] + + def test_missing_kit_value_error_leg(self, cs_plain_fit): + import copy + + cs, _ = cs_plain_fit + stripped = copy.copy(cs) + object.__setattr__(stripped, "_aggregation_kit", None) + skipped = DiagnosticReport(stripped).run_all().skipped_checks + # ALL three ES-gated consumers carry the accurate failure reason — + # no availability leg may mask it with a coefficient-count message. + for check in ("parallel_trends", "pretrends_power", "sensitivity"): + assert check in skipped, skipped + assert "ValueError" in skipped[check] + assert "no aggregation kit" in skipped[check] + + def test_imputation_pretrends_replicate_not_implemented_leg(self): + from diff_diff import ImputationDiD, SurveyDesign + + sdf = generate_staggered_data(n_units=100, n_periods=6, treatment_effect=1.5, seed=7).copy() + rng = np.random.default_rng(5) + wmap = {u: rng.uniform(0.5, 2.0) for u in sdf["unit"].unique()} + sdf["w"] = sdf["unit"].map(wmap) + rep_cols = [] + for r in range(8): + col = f"rw{r}" + rep_cols.append(col) + jitter = {u: rng.uniform(0.1, 2.0) for u in sdf["unit"].unique()} + sdf[col] = sdf["unit"].map(jitter) * sdf["w"] + sd = SurveyDesign(weights="w", replicate_weights=rep_cols, replicate_method="JK1") + imp = ImputationDiD(pretrends=True).fit( + sdf, + outcome="outcome", + unit="unit", + time="period", + first_treat="first_treat", + survey_design=sd, + ) + skipped = DiagnosticReport(imp).run_all().skipped_checks + # The non-bootstrap test of the "{TypeName}: {msg}" embedding. + assert "parallel_trends" in skipped + assert "NotImplementedError" in skipped["parallel_trends"] + + def test_wooldridge_reason_points_at_inplace_aggregate(self): + from diff_diff import WooldridgeDiD + + sdf = generate_staggered_data(n_units=100, n_periods=6, treatment_effect=1.5, seed=7) + w = WooldridgeDiD().fit( + sdf, outcome="outcome", unit="unit", time="period", first_treat="first_treat" + ) + skipped = DiagnosticReport(w).run_all().skipped_checks + assert "results.aggregate(type='event_study')" in skipped["parallel_trends"] + assert "deprecated" not in skipped["parallel_trends"] + # DR must never auto-call Wooldridge's MUTATING aggregate. + assert w.event_study_effects is None + + def test_staggered_triple_diff_reason_names_canonical_kwarg(self): + from diff_diff import TripleDifference + + sdf = generate_staggered_data(n_units=100, n_periods=6, treatment_effect=1.5, seed=7) + sdf = sdf.assign(partition=(sdf["unit"] % 2)) + t = TripleDifference().fit( + sdf, + outcome="outcome", + unit="unit", + partition="partition", + time="period", + first_treat="first_treat", + ) + skipped = DiagnosticReport(t).run_all().skipped_checks + assert "aggregate='event_study'" in skipped["parallel_trends"] + assert "canonical" in skipped["parallel_trends"] + assert "deprecated" not in skipped["parallel_trends"] + + def test_plain_imputation_twostage_continuous_heterogeneity( + self, imp_pretrends_fits, ts_pretrends_fits + ): + _, imp_plain, _ = imp_pretrends_fits + _, ts_plain, _ = ts_pretrends_fits + for res in (imp_plain, ts_plain): + d = DiagnosticReport(res).run_all().to_dict()["heterogeneity"] + assert d["status"] == "ran", d + assert d["source"] == "aggregate_event_study_post" + from diff_diff import ContinuousDiD + + rng = np.random.default_rng(3) + n = 200 + cdf = pd.DataFrame({"unit": np.repeat(np.arange(n), 4), "period": np.tile([1, 2, 3, 4], n)}) + cdf["first_treat"] = np.repeat(np.where(np.arange(n) % 2 == 0, 3, 0), 4) + cdf["dose"] = np.repeat(np.where(np.arange(n) % 2 == 0, rng.uniform(0.5, 2.0, n), 0.0), 4) + cdf["outcome"] = rng.normal(0, 1, len(cdf)) + cdf["dose"] * np.maximum( + 0, cdf["period"] - cdf["first_treat"] + 1 + ) * (cdf["first_treat"] > 0) + c = ContinuousDiD().fit( + cdf, + outcome="outcome", + unit="unit", + time="period", + first_treat="first_treat", + dose="dose", + ) + d = DiagnosticReport(c).run_all().to_dict()["heterogeneity"] + assert d["status"] == "ran" + assert d["source"] == "aggregate_event_study_post" + + def test_plain_imputation_twostage_pt_runs(self, imp_pretrends_fits, ts_pretrends_fits): + _, imp_plain, _ = imp_pretrends_fits + _, ts_plain, _ = ts_pretrends_fits + for res in (imp_plain, ts_plain): + d = DiagnosticReport(res).run_all().to_dict()["parallel_trends"] + assert d["status"] == "ran", d + assert d["pre_period_source"] == "aggregate_event_study" + + def test_twostage_without_pretrends_gets_no_pre_horizons_reason(self): + from diff_diff import TwoStageDiD + + sdf = generate_staggered_data(n_units=100, n_periods=6, treatment_effect=1.5, seed=7) + ts = TwoStageDiD().fit( + sdf, outcome="outcome", unit="unit", time="period", first_treat="first_treat" + ) + results = DiagnosticReport(ts).run_all() + skipped = results.skipped_checks + assert "no pre-periods" in skipped["parallel_trends"] + assert "aggregate='event_study'" not in skipped["parallel_trends"] + # A SUCCESSFUL derivation that turned out pre-empty still carries + # the derived-route provenance on the gate-skipped section. + pt = results.to_dict()["parallel_trends"] + assert pt["status"] == "skipped" + assert pt["pre_period_source"] == "aggregate_event_study" + + def test_mpd_one_pre_period_gets_reference_message(self): + rng = np.random.default_rng(5) + rows = [] + for u in range(60): + for t in (1, 2): + rows.append( + { + "unit": u, + "period": t, + "treated": int(u < 30), + "outcome": rng.normal() + (1.0 if (u < 30 and t == 2) else 0.0), + } + ) + mdf = pd.DataFrame(rows) + m = MultiPeriodDiD().fit( + mdf, + outcome="outcome", + treatment="treated", + time="period", + unit="unit", + post_periods=[2], + ) + skipped = DiagnosticReport(m).run_all().skipped_checks + assert "every pre-treatment period is the omitted reference" in skipped["parallel_trends"] + assert "does not support post-fit" not in skipped["parallel_trends"] + + def test_adapter_zero_count_row_excluded(self): + # Mirror of the raw-route pin: an n == 0, NaN-effect NON-reference + # container row is EXCLUDED (never counted in n_dropped_undefined). + from diff_diff.diagnostic_report import ( + _collect_pre_period_coefs, + _surface_to_event_study_dict, + ) + from diff_diff.results_base import EventStudyResults + + surface = EventStudyResults( + event_time=np.array([-3, -2, -1, 0, 1]), + att=np.array([0.1, np.nan, 0.0, 0.5, 0.6]), + se=np.array([0.05, np.nan, np.nan, 0.1, 0.1]), + t_stat=np.array([2.0, np.nan, np.nan, 5.0, 6.0]), + p_value=np.array([0.04, np.nan, np.nan, 0.001, 0.001]), + conf_int_lower=np.array([0.0, np.nan, np.nan, 0.3, 0.4]), + conf_int_upper=np.array([0.2, np.nan, np.nan, 0.7, 0.8]), + is_reference=np.array([False, False, True, False, False]), + n=np.array([40.0, 0.0, np.nan, 40.0, 40.0]), + ) + adapted = _surface_to_event_study_dict(surface) + # Reference row (-1) and zero-count row (-2) both excluded. + assert set(adapted) == {-3, 0, 1} + assert all(isinstance(k, int) for k in adapted) + + class _Shell: + event_study_effects = None + anticipation = 0 + + pre_coefs, n_dropped = _collect_pre_period_coefs(_Shell(), surface_dict=adapted) + assert [k for (k, _, _, _) in pre_coefs] == [-3] + assert n_dropped == 0 + + def test_adapter_rejects_calendar_scale(self): + from diff_diff.diagnostic_report import _surface_to_event_study_dict + from diff_diff.results_base import EventStudyResults + + surface = EventStudyResults( + event_time=np.array(["2001", "2002"], dtype=object), + att=np.array([0.1, 0.2]), + se=np.array([0.05, 0.05]), + t_stat=np.array([2.0, 4.0]), + p_value=np.array([0.04, 0.001]), + conf_int_lower=np.array([0.0, 0.1]), + conf_int_upper=np.array([0.2, 0.3]), + is_reference=np.array([False, False]), + n=np.array([40.0, 40.0]), + time_scale="calendar", + source="TwoWayFixedEffects", + post_periods=["2002"], + ) + with pytest.raises(ValueError, match="relative-scale"): + _surface_to_event_study_dict(surface) + + def test_imputation_twostage_raw_vs_derived_pt_parity( + self, imp_pretrends_fits, ts_pretrends_fits + ): + imp_raw, imp_plain, _ = imp_pretrends_fits + ts_raw, ts_plain, _ = ts_pretrends_fits + # ImputationDiD carries no event-study vcov on either route -> both + # take the Bonferroni path; joint p compared at 1e-12. + d_raw = DiagnosticReport(imp_raw).run_all().to_dict()["parallel_trends"] + d_plain = DiagnosticReport(imp_plain).run_all().to_dict()["parallel_trends"] + assert d_raw["method"] == d_plain["method"] == "bonferroni" + np.testing.assert_allclose( + d_plain["joint_p_value"], d_raw["joint_p_value"], rtol=1e-12, atol=1e-12 + ) + # TwoStageDiD persists the ES vcov on both routes -> joint Wald. + d_raw = DiagnosticReport(ts_raw).run_all().to_dict()["parallel_trends"] + d_plain = DiagnosticReport(ts_plain).run_all().to_dict()["parallel_trends"] + assert d_raw["method"] == d_plain["method"] == "joint_wald_event_study" + np.testing.assert_allclose( + d_plain["test_statistic"], d_raw["test_statistic"], rtol=1e-12, atol=1e-12 + ) + np.testing.assert_allclose( + d_plain["joint_p_value"], d_raw["joint_p_value"], rtol=1e-12, atol=1e-12 + ) + + @staticmethod + def _assert_heterogeneity_parity(raw_fit, plain_fit): + d_raw = DiagnosticReport(raw_fit).run_all().to_dict()["heterogeneity"] + d_plain = DiagnosticReport(plain_fit).run_all().to_dict()["heterogeneity"] + assert d_raw["source"] == "event_study_effects_post" + assert d_plain["source"] == "aggregate_event_study_post" + assert d_plain["n_effects"] == d_raw["n_effects"] + assert d_plain["sign_consistent"] == d_raw["sign_consistent"] + for key in ("min", "max", "mean", "sd", "range"): + np.testing.assert_allclose(d_plain[key], d_raw[key], rtol=1e-12, atol=1e-12) + + def test_imputation_heterogeneity_numeric_parity(self, imp_pretrends_fits): + imp_raw, imp_plain, _ = imp_pretrends_fits + self._assert_heterogeneity_parity(imp_raw, imp_plain) + + def test_twostage_heterogeneity_numeric_parity(self, ts_pretrends_fits): + ts_raw, ts_plain, _ = ts_pretrends_fits + self._assert_heterogeneity_parity(ts_raw, ts_plain) + + def test_continuous_heterogeneity_numeric_parity(self): + from diff_diff import ContinuousDiD + + rng = np.random.default_rng(3) + n = 200 + cdf = pd.DataFrame({"unit": np.repeat(np.arange(n), 4), "period": np.tile([1, 2, 3, 4], n)}) + cdf["first_treat"] = np.repeat(np.where(np.arange(n) % 2 == 0, 3, 0), 4) + cdf["dose"] = np.repeat(np.where(np.arange(n) % 2 == 0, rng.uniform(0.5, 2.0, n), 0.0), 4) + cdf["outcome"] = rng.normal(0, 1, len(cdf)) + cdf["dose"] * np.maximum( + 0, cdf["period"] - cdf["first_treat"] + 1 + ) * (cdf["first_treat"] > 0) + kw = dict( + outcome="outcome", unit="unit", time="period", first_treat="first_treat", dose="dose" + ) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + # ContinuousDiD's fit-time spelling has no underscore. + raw = ContinuousDiD().fit(cdf, aggregate="eventstudy", **kw) + plain = ContinuousDiD().fit(cdf, **kw) + self._assert_heterogeneity_parity(raw, plain) + + def test_pre_period_source_on_inconclusive_derived_return(self, cs_plain_fit): + cs, _ = cs_plain_fit + dr = DiagnosticReport(cs) + surface, surface_dict, _ = dr._resolve_event_study_surface() + assert surface is not None + # Force an undefined-inference pre row through the derived dict so + # the runner takes the dropped-undefined inconclusive branch. + poisoned = dict(surface_dict) + first_pre = min(k for k in poisoned if k < 0) + poisoned[first_pre] = {"effect": np.nan, "se": np.nan, "p_value": np.nan} + dr._derived_es_cache = (surface, poisoned, None) + d = dr.run_all().to_dict()["parallel_trends"] + assert d["method"] == "inconclusive" + assert d["pre_period_source"] == "aggregate_event_study" + + def test_anticipation_boundary_respected_on_derived_route(self): + sdf = generate_staggered_data(n_units=120, n_periods=8, treatment_effect=1.5, seed=7) + cs = CallawaySantAnna(base_period="universal", anticipation=1).fit( + sdf, outcome="outcome", unit="unit", time="period", first_treat="first_treat" + ) + d = DiagnosticReport(cs).run_all().to_dict()["parallel_trends"] + assert d["status"] == "ran" + assert d["pre_period_source"] == "aggregate_event_study" + assert all(p["period"] < -1 for p in d["per_period"]) + + def test_requested_but_empty_raw_field_is_authoritative(self): + # Round-4 dual-review P0 pin: a fit whose fit-time aggregate=/ + # balance_e= produced the REQUESTED-but-empty {} sentinel must + # behave exactly as today — no silent re-derivation (which would + # drop the fit-time balance_e and report effects over cohorts the + # fit deliberately excluded). + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + sdf = generate_staggered_data(n_units=100, n_periods=6, treatment_effect=1.5, seed=7) + e = EfficientDiD().fit( + sdf, + outcome="outcome", + unit="unit", + time="period", + first_treat="first_treat", + aggregate="event_study", + balance_e=4, + ) + assert e.event_study_effects == {} + calls = {"n": 0} + orig = type(e).aggregate + + def counting(self, *args, **kwargs): + calls["n"] += 1 + return orig(self, *args, **kwargs) + + with patch.object(type(e), "aggregate", counting): + d = DiagnosticReport(e).run_all().to_dict() + assert d["heterogeneity"]["status"] == "skipped" + assert calls["n"] == 0 + + def test_derived_route_warnings_republished(self, cs_plain_fit): + cs, _ = cs_plain_fit + orig = type(cs).aggregate + + def warning_aggregate(self, *args, **kwargs): + warnings.warn("synthetic kit-recompute caveat", UserWarning, stacklevel=2) + return orig(self, *args, **kwargs) + + with patch.object(type(cs), "aggregate", warning_aggregate): + d = DiagnosticReport(cs).run_all().to_dict() + pt = d["parallel_trends"] + assert pt["status"] == "ran" + assert any("synthetic kit-recompute caveat" in w for w in pt.get("warnings", [])) + # Record-and-republish reaches the top-level warnings channel too. + assert any("synthetic kit-recompute caveat" in w for w in d["warnings"]) + + def test_no_mutation_after_run_all(self, cs_plain_fit): + cs, _ = cs_plain_fit + DiagnosticReport(cs).run_all() + assert cs.event_study_effects is None + + def test_raw_precedence_never_calls_aggregate(self, cs_fit): + cs, _ = cs_fit + + def boom(self, *args, **kwargs): + raise AssertionError("aggregate() must not be called when the raw field is present") + + with patch.object(type(cs), "aggregate", boom): + d = DiagnosticReport(cs).run_all().to_dict() + assert d["parallel_trends"]["status"] == "ran" + + def test_derivation_is_cached_single_call(self, cs_plain_fit): + cs, _ = cs_plain_fit + calls = {"n": 0} + orig = type(cs).aggregate + + def counting(self, *args, **kwargs): + calls["n"] += 1 + return orig(self, *args, **kwargs) + + with patch.object(type(cs), "aggregate", counting): + dr = DiagnosticReport(cs) + dr.applicable_checks + dr.run_all() + assert calls["n"] == 1 + + def test_survey_backed_raw_vs_derived_pt_parity(self): + # Round-1 local-review P2: the derived-surface covariance composed + # with the report-level finite-df (F(k, df_survey)) inference had + # no direct raw-vs-derived pin. + from diff_diff import SurveyDesign, TwoStageDiD + + rng = np.random.default_rng(9) + sdf = generate_staggered_data(n_units=100, n_periods=6, treatment_effect=1.5, seed=7).copy() + units = sdf["unit"].unique() + sdf["w"] = sdf["unit"].map({u: rng.uniform(0.5, 2.0) for u in units}) + sdf["stratum"] = sdf["unit"].map({u: int(u) % 4 for u in units}) + sdf["psu"] = sdf["unit"] + sd = SurveyDesign(weights="w", strata="stratum", psu="psu") + kw = dict( + outcome="outcome", + unit="unit", + time="period", + first_treat="first_treat", + survey_design=sd, + ) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + cs_raw = CallawaySantAnna(base_period="universal").fit( + sdf, aggregate="event_study", **kw + ) + cs_plain = CallawaySantAnna(base_period="universal").fit(sdf, **kw) + ts_raw = TwoStageDiD(pretrends=True).fit(sdf, aggregate="event_study", **kw) + ts_plain = TwoStageDiD(pretrends=True).fit(sdf, **kw) + for raw, plain in ((cs_raw, cs_plain), (ts_raw, ts_plain)): + d_raw = DiagnosticReport(raw).run_all().to_dict()["parallel_trends"] + d_plain = DiagnosticReport(plain).run_all().to_dict()["parallel_trends"] + assert d_raw["method"] == "joint_wald_event_study_survey" + assert d_plain["method"] == "joint_wald_event_study_survey" + np.testing.assert_allclose( + d_plain["test_statistic"], d_raw["test_statistic"], rtol=1e-12, atol=1e-12 + ) + np.testing.assert_allclose( + d_plain["joint_p_value"], d_raw["joint_p_value"], rtol=1e-12, atol=1e-12 + ) + np.testing.assert_allclose( + d_plain["df_denom"], d_raw["df_denom"], rtol=1e-12, atol=1e-12 + ) + assert d_plain["pre_period_source"] == "aggregate_event_study" + assert "pre_period_source" not in d_raw + + def test_derived_route_error_paths_carry_provenance(self, cs_plain_fit): + # A derived-route consumer failure must stay schema-distinguishable + # from a raw-route one (the additive-key contract). + cs, _ = cs_plain_fit + + def boom(*args, **kwargs): + raise RuntimeError("synthetic consumer failure") + + with patch("diff_diff.pretrends.compute_pretrends_power", boom): + d = DiagnosticReport(cs).run_all().to_dict()["pretrends_power"] + assert d["status"] == "error" + assert d["pre_period_source"] == "aggregate_event_study" + + from diff_diff.honest_did import HonestDiD + + with patch.object(HonestDiD, "sensitivity_analysis", boom): + d = DiagnosticReport(cs).run_all().to_dict()["sensitivity"] + assert d["status"] == "error" + assert d["pre_period_source"] == "aggregate_event_study" + + def test_warning_then_failure_rides_skipped_section_and_top_level(self, cs_plain_fit): + # A warning emitted by aggregate('event_study') immediately before + # the call raises must appear BOTH on the gate-skipped consuming + # section and in the top-level warnings channel. + cs, _ = cs_plain_fit + + def warn_then_raise(self, *args, **kwargs): + warnings.warn("kit caveat before failure", UserWarning, stacklevel=2) + raise NotImplementedError("synthetic derivation failure") + + with patch.object(type(cs), "aggregate", warn_then_raise): + results = DiagnosticReport(cs).run_all() + d = results.to_dict() + pt = d["parallel_trends"] + assert pt["status"] == "skipped" + assert "NotImplementedError" in pt["reason"] + assert any("kit caveat before failure" in w for w in pt.get("warnings", [])) + assert any("kit caveat before failure" in w for w in d["warnings"]) + + def test_successful_empty_derivation_warning_rides_skipped_section(self): + # A successful reference-only derivation that EMITS a warning: + # the warning appears on the gate-skipped consuming section AND at + # the top level, and the section carries the derived provenance. + from diff_diff import TwoStageDiD + + sdf = generate_staggered_data(n_units=100, n_periods=6, treatment_effect=1.5, seed=7) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + ts = TwoStageDiD().fit( + sdf, outcome="outcome", unit="unit", time="period", first_treat="first_treat" + ) + orig = type(ts).aggregate + + def warn_then_succeed(self, *args, **kwargs): + warnings.warn("reference-only surface caveat", UserWarning, stacklevel=2) + return orig(self, *args, **kwargs) + + with patch.object(type(ts), "aggregate", warn_then_succeed): + d = DiagnosticReport(ts).run_all().to_dict() + pt = d["parallel_trends"] + assert pt["status"] == "skipped" + assert pt["pre_period_source"] == "aggregate_event_study" + assert any("reference-only surface caveat" in w for w in pt.get("warnings", [])) + assert any("reference-only surface caveat" in w for w in d["warnings"]) + + def test_wrong_container_type_warning_rides_skipped_section(self, cs_plain_fit): + # Defensive branch: aggregate() warns and then returns a + # non-EventStudyResults object — treated exactly like a raised + # exception (failure recorded, warning republished on the section). + cs, _ = cs_plain_fit + + def warn_then_wrong_type(self, *args, **kwargs): + warnings.warn("wrong-type caveat", UserWarning, stacklevel=2) + return {"not": "a container"} + + with patch.object(type(cs), "aggregate", warn_then_wrong_type): + d = DiagnosticReport(cs).run_all().to_dict() + for check in ("parallel_trends", "pretrends_power", "sensitivity"): + assert d[check]["status"] == "skipped", d[check] + assert "unexpected dict" in d[check]["reason"] + pt = d["parallel_trends"] + assert any("wrong-type caveat" in w for w in pt.get("warnings", [])) + assert any("wrong-type caveat" in w for w in d["warnings"]) + + def test_user_opted_out_check_carries_no_derived_provenance(self, cs_plain_fit): + # A run_*=False opt-out skip never consulted the resolver and must + # stay a plain opt-out skip — no provenance, no derivation warnings + # — while the consuming sections still carry them. + cs, _ = cs_plain_fit + orig = type(cs).aggregate + + def warning_aggregate(self, *args, **kwargs): + warnings.warn("kit caveat", UserWarning, stacklevel=2) + return orig(self, *args, **kwargs) + + with patch.object(type(cs), "aggregate", warning_aggregate): + d = DiagnosticReport(cs, run_sensitivity=False).run_all().to_dict() + sens = d["sensitivity"] + assert sens["status"] == "skipped" + assert "user opted out" in sens["reason"] + assert "pre_period_source" not in sens + assert "warnings" not in sens + pt = d["parallel_trends"] + assert pt["status"] == "ran" + assert any("kit caveat" in w for w in pt.get("warnings", [])) + + def test_derived_warning_single_copy_at_top_level(self, cs_plain_fit): + # One derivation event -> one top-level copy, even when three + # consuming sections each carry the section-local copy. + cs, _ = cs_plain_fit + orig = type(cs).aggregate + + def warning_aggregate(self, *args, **kwargs): + warnings.warn("single-copy caveat", UserWarning, stacklevel=2) + return orig(self, *args, **kwargs) + + with patch.object(type(cs), "aggregate", warning_aggregate): + d = DiagnosticReport(cs).run_all().to_dict() + for check in ("parallel_trends", "pretrends_power", "sensitivity"): + assert any("single-copy caveat" in w for w in d[check].get("warnings", [])) + assert sum("single-copy caveat" in w for w in d["warnings"]) == 1 + + def test_bootstrap_failure_heterogeneity_skips_with_context(self): + # Derived heterogeneity producers on a bootstrapped fit: the runner + # section fails closed with the derivation context and publishes no + # numeric heterogeneity fields. + from diff_diff import ImputationDiD + + sdf = generate_staggered_data(n_units=100, n_periods=6, treatment_effect=1.5, seed=7) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + imp = ImputationDiD(pretrends=True, n_bootstrap=19, seed=1).fit( + sdf, outcome="outcome", unit="unit", time="period", first_treat="first_treat" + ) + d = DiagnosticReport(imp).run_all().to_dict() + het = d["heterogeneity"] + assert het["status"] == "skipped" + assert "aggregate('event_study')" in het["reason"] + assert "NotImplementedError" in het["reason"] + assert "n_effects" not in het + assert "mean" not in het From 624e299b72e6f3f871c2c967508a3dba714ca68f Mon Sep 17 00:00:00 2001 From: igerber Date: Sat, 15 Aug 2026 17:49:22 -0400 Subject: [PATCH 2/2] fix(reports): BusinessReport skip-path lift keeps derived pre_period_source provenance The _lift_pre_trends early return (PT status != 'ran') dropped the pre_period_source key DiagnosticReport attaches to gate-skipped sections whose derivation succeeded but produced no pre-horizons; lift it on the skip path too (None on raw routes, matching the computed path). Pinned with a default-TwoStageDiD BusinessReport test. --- diff_diff/business_report.py | 4 ++++ tests/test_business_report.py | 16 ++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/diff_diff/business_report.py b/diff_diff/business_report.py index 012d4a4bd..e8d9c224b 100644 --- a/diff_diff/business_report.py +++ b/diff_diff/business_report.py @@ -947,6 +947,10 @@ def _lift_pre_trends(dr: Optional[Dict[str, Any]]) -> Dict[str, Any]: return { "status": pt.get("status", "not_run"), "reason": pt.get("reason"), + # DR attaches derived-surface provenance to gate-skipped + # sections too (a successful-but-empty derivation) — the skip + # path must not drop it (None on raw routes, same as below). + "pre_period_source": pt.get("pre_period_source"), } return { "status": "computed", diff --git a/tests/test_business_report.py b/tests/test_business_report.py index 617ea90b6..651dd3fcb 100644 --- a/tests/test_business_report.py +++ b/tests/test_business_report.py @@ -5080,3 +5080,19 @@ def test_raw_route_business_report_pre_period_source_is_none(self): pt = d["pre_trends"] assert pt["status"] == "computed" assert pt["pre_period_source"] is None + + def test_skipped_pt_still_lifts_derived_provenance(self): + # CI review P2: a default TwoStageDiD fit (pretrends=False) derives a + # pre-empty surface, DR gate-skips PT with derived provenance, and BR's + # skip-path lift must carry the key rather than drop it. + from diff_diff import TwoStageDiD + + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + sdf = generate_staggered_data(n_units=100, n_periods=6, treatment_effect=1.5, seed=7) + ts = TwoStageDiD().fit( + sdf, outcome="outcome", unit="unit", time="period", first_treat="first_treat" + ) + pt = BusinessReport(ts, outcome_label="y").to_dict()["pre_trends"] + assert pt["status"] == "skipped" + assert pt["pre_period_source"] == "aggregate_event_study"