diff --git a/CHANGELOG.md b/CHANGELOG.md index 21bd0fa53..7e4938c66 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added +- **MMM exporters: container mode on the 3.9 aggregation surface.** Both + `to_pymc_marketing_lift_test` and `to_meridian_roi_prior` now accept + `aggregation_result=` - the pinned `AggregationResult` returned by post-fit + `results.aggregate('simple')` (one experiment row) or `'group'` (one row per + cohort) - together with the new `scale=`, deriving `effect = att * scale` and + `se = se * scale` per row in `to_dataframe()` order (the order every per-row + sequence kwarg aligns to). `scale` is always required with a container: a + finite positive number (scalar or per-row), or `"auto"`, which reads the + container's per-row `n` and is honored ONLY for ImputationDiD/TwoStageDiD + fits - the audited producers whose `n` is exactly the treated unit-periods + the ATT averages over. `"auto"` is an explicit acknowledgement of three + assumptions the container cannot verify (additive-level outcome, unweighted + fit, fully identified effects - both estimators warn at fit time on the + degenerate identification branch, where `att * n` overcounts; behaviorally + pinned in the tests). Everything else fails closed with the remedy inline: + raw results objects and `EventStudyResults` (the exporters never call + `aggregate()` themselves), `calendar`/`dose`/unknown levels, non-`"att"` + target rows (ContinuousDiD's `acrt`, HAD's WAS, dCDH's estimand relays - + container rejected whole, no silent row filtering), rows without usable + inference, and `scale="auto"` on any other estimator provenance + (CallawaySantAnna/EfficientDiD/StackedDiD hints explain their `n` semantics; + routing keys on the container's `estimator` field, never on `n_kind`, which + serves only as a drift guard). The explicit `delta_y`/`sigma` and + `incremental_outcome`/`incremental_outcome_se` route is unchanged and the two + routes are mutually exclusive. Docs: the REGISTRY MMM section, `api/mmm.rst` + (whose examples are now self-contained and executed by + `tests/test_doc_snippets.py`), and both LLM guides drop the pre-3.9 "does + not introspect result objects / deferred to the post-4.0 aggregate() layer" + framing - that layer shipped in 3.9 and this is its consumer. The + estimator-owned-totals remainder for CS/EfficientDiD/StackedDiD stays + tracked in DEFERRED.md. + ### Changed - **Narrative docs migrated off the deprecated fit-time `aggregate=`** (the 3.9 M-020 family; TODO "fit-time aggregate= teachings" sweep, RST half): diff --git a/DEFERRED.md b/DEFERRED.md index 10f7a2e6b..cc6c07565 100644 --- a/DEFERRED.md +++ b/DEFERRED.md @@ -72,7 +72,7 @@ For survey-specific limitations (`NotImplementedError` paths), see the | Issue | Location | PR | Priority | |-------|----------|----|----------| | EfficientDiD/ImputationDiD/TwoStageDiD kit retention has no memory opt-out: EfficientDiD's kit always retains the per-(g,t) EIF dict (M-023), ImputationDiD's references the `_fit_data` panel objects (M-021, pre-existing retention), and TwoStageDiD's retains a column-subset working-frame copy (M-022, the first new panel retention); a `store_kit=False` constructor knob would restore an opt-out at the cost of post-fit aggregation on that result | `diff_diff/efficient_did.py`, `diff_diff/imputation.py`, `diff_diff/two_stage.py` | 2(b) PR-3a | Low | -| MMM interop: result-derived scaling for the exporters - let a fitted result produce the scoped incremental outcome + SE directly (via the post-4.0 `results.aggregate()` layer, where the estimator owns its aggregation weights/balance/survey masses), so callers need not hand-scale the ATT. v1 is explicit-in by design; this is the seamless follow-up. | `diff_diff/mmm.py`, `docs/v4-design.md` | mmm-interop | Low | +| MMM interop: estimator-owned total incremental outcome for CallawaySantAnna/EfficientDiD/StackedDiD aggregations - their `AggregationResult` containers report units/cells (no treated-obs mass), so container-derived scaling (`aggregation_result=` + `scale="auto"`, shipped for ImputationDiD/TwoStageDiD) cannot cover them and their exporter route requires a caller-supplied numeric `scale=`. Closing the gap means each estimator's own aggregation (which owns its weights/balance/survey masses) computing a total effect + SE. | `diff_diff/mmm.py`, `diff_diff/staggered_aggregation.py` | mmm-interop | Low | | `SyntheticControl` fit-snapshot residency (`_SyntheticControlFitSnapshot`) — **investigated 2026-07-07, parked**: the snapshot ALIASES the fit's own working pivots (zero extra construction cost); the retained residency implements the documented freeze contract (post-fit mutation of estimator inputs must not change `in_space_placebo()` / `leave_one_out()` / conformal output on an already-returned results object, and `__getstate__` already excludes it from pickles). A compact array representation saves only pandas overhead (the float panel dominates); releasing residency needs new API surface (`release`/opt-out flag) or a freeze-contract change. Revisit on user demand for very large donor panels. | `synthetic_control.py`, `synthetic_control_results.py` | follow-up | Low | | Stratified survey-PSU multiplier-weight draw-tiling — **investigated 2026-07-07, parked**: the stratified generator (`generate_survey_multiplier_weights_batch`) consumes ONE sequential rng stream stratum-major (`rng.choice(size=(n_bootstrap, n_h))` per stratum, then lonely-PSU pooling), so draw-chunked assembly CANNOT reproduce the stream bit-identically (contra the old row's parenthetical) — it would need per-stratum generator state skipping (PCG64.advance + per-weight-type variate accounting; fragile) or a stream-layout change (MC-level SE changes → baseline/golden recapture + REGISTRY note). Stratified designs have few PSUs, so the full `(n_bootstrap × n_psu)` matrix rarely matters; unstratified (the large-`n_units` case) is already tiled. Revisit only if a large-PSU stratified design hits memory, as a documented stream change. | `diff_diff/bootstrap_chunking.py::iter_survey_multiplier_weight_blocks` | follow-up | Low | | ChangesInChanges FULL Melly-Santangelo covariate estimator (monotonized integrated-indicator conditional CDFs, treated-post `F_{X|11}` integration, exchangeable bootstrap with variance-weighted KS bands, tail trimming, pre-period specification test). The simplified qte-`xformla` form of the MS pipeline is already supported (`covariates=` on both estimators, parity-tested vs qte 1.3.1); the FULL estimator has no reference implementation (the MS Stata code is the only one; distinct from Kranker's `cic`) and would need simulation-based validation. Reviewed: `docs/methodology/papers/melly-santangelo-2015-review.md`. | `diff_diff/changes_in_changes.py` | #682 | Low | diff --git a/diff_diff/guides/llms-full.txt b/diff_diff/guides/llms-full.txt index 921d8e07f..f19cc7f74 100644 --- a/diff_diff/guides/llms-full.txt +++ b/diff_diff/guides/llms-full.txt @@ -2873,21 +2873,28 @@ stability policy. ## MMM Calibration Export Interop builders converting experiment results into Marketing Mix Model -(MMM) calibration inputs. Pure numpy/pandas - no MMM package is imported, -no result object is introspected (the module is purely additive). Design: -EXPLICIT IN, VALIDATED OUT. Reconciling an experiment estimate to a -calibration input needs the target MMM's row granularity (per-geo vs -national), its time window, and the outcome scale (additive levels vs -log/rate/share) - none of which diff-diff can see - so the CALLER supplies -the already-scoped incremental outcome and its SE (read off summary(), -aggregated to the population/window one MMM row represents). diff-diff -assembles the schema, enforces each consumer's guards, converts to the -lognormal parameterization, pools, and emits snippets. Deriving totals -from a fit is deferred to the post-4.0 results.aggregate() layer, where -the estimator owns its aggregation weights/balance/survey masses. +(MMM) calibration inputs. Pure numpy/pandas - no MMM package is imported. +Design: EXPLICIT NUMBERS IN, OR THE PINNED AGGREGATION CONTRACT IN - +VALIDATED OUT. Reconciling an experiment estimate to a calibration input +needs the target MMM's row granularity (per-geo vs national), its time +window, and the outcome scale (additive levels vs log/rate/share), so the +default route stays explicit: the CALLER supplies the already-scoped +incremental outcome and its SE (read off summary(), aggregated to the +population/window one MMM row represents). Alternatively both exporters +take aggregation_result= (the AggregationResult from post-fit +results.aggregate('simple'|'group')) + scale=, deriving effect = +att * scale and se = se * scale per row in to_dataframe() order; +scale="auto" (per-row n off the container) is honored ONLY for +ImputationDiD/TwoStageDiD fits - it acknowledges an additive-level +outcome, an unweighted fit, and fully identified effects - and every +other estimator, raw results objects, and EventStudyResults fail closed +(the module never calls aggregate() itself). diff-diff assembles the +schema, enforces each consumer's guards, converts to the lognormal +parameterization, pools, and emits snippets. ```python from diff_diff import ( + ImputationDiD, SyntheticDiD, to_pymc_marketing_lift_test, to_meridian_roi_prior, @@ -2936,9 +2943,27 @@ print(prior.to_code( # ready-to-paste PriorDistribution + roi_calibration_period="mask", # AND time scope (mask expr) or # full_model_window=True; sets )) # media_prior_type accordingly. + +# Container route (ImputationDiD/TwoStageDiD): derive the totals from the +# post-fit aggregation instead of hand-scoping. +res = ImputationDiD().fit(data, outcome="outcome", unit="unit", + time="period", first_treat="first_treat") +prior2 = to_meridian_roi_prior( + aggregation_result=res.aggregate("simple"), + scale="auto", # att x n (n = treated unit-periods here) + spend=200_000.0, +) ``` Key points: +- Container mode: aggregation_result= accepts ONLY AggregationResult + ('simple' = one experiment row; 'group' = one per cohort, rows in + to_dataframe() order - per-row scale/spend/x/delta_x align to it). + scale= is always required: numeric, or "auto" for the audited + ImputationDiD/TwoStageDiD producers only (CS/EfficientDiD/StackedDiD + n counts units/cells, not treated unit-periods - pass a numeric scale). + A 'group' container's cohorts come from ONE fit: widen se_widening for + Meridian pooling, and note PyMC scores each row as independent evidence. - Lift frame: one row per experiment, columns [channel, *dims, x, delta_x, delta_y, sigma]; guards on sigma>0, delta_x!=0, x>=0, x+delta_x>=0, finite delta_y; dims must not collide with reserved columns and must diff --git a/diff_diff/guides/llms.txt b/diff_diff/guides/llms.txt index 24339e0d1..75b0d5227 100644 --- a/diff_diff/guides/llms.txt +++ b/diff_diff/guides/llms.txt @@ -90,7 +90,7 @@ The site is organized into 5 sections, each with a landing page: - [Honest DiD](https://diff-diff.readthedocs.io/en/stable/api/honest_did.html): Rambachan & Roth (2023) sensitivity analysis — robust CI under parallel trends violations, breakdown values - [Pre-Trends Power Analysis](https://diff-diff.readthedocs.io/en/stable/api/pretrends.html): Roth (2022) Section II.A-B no-individually-significant (NIS) box-probability pretest power + minimum detectable violation; `pretest_form='nis'` (default) implements the paper's primary form, `pretest_form='wald'` retained as paper-supported alternative (Propositions 1+3+4 all apply); linear-violation MDV in Roth's γ units when relative-time labels are threaded through `fit()`; full Σ_22 routing on non-bootstrap CallawaySantAnna and SunAbraham adapters and on admitted CS-/StackedDiD-sourced `aggregate('event_study')` containers (StackedDiD persists its ES VCV in every inference mode) - [Power Analysis](https://diff-diff.readthedocs.io/en/stable/api/power.html): Analytical and simulation-based power analysis — MDE, sample size, power curves for study design -- [MMM Calibration Export](https://diff-diff.readthedocs.io/en/stable/api/mmm.html): Assemble Marketing Mix Model calibration inputs from experiment results (explicit-in / validated-out - the caller passes the already-scoped incremental outcome + SE, the module does NOT rescale a headline ATT). `to_pymc_marketing_lift_test(channel, x, delta_x, delta_y, sigma, dims=, on_wrong_sign=)` builds the PyMC-Marketing/prophetverse lift-test DataFrame with sign/zero/positivity guards. `to_meridian_roi_prior(incremental_outcome, incremental_outcome_se, spend, parameter="roi_m"|"mroi_m", se_widening=)` builds Google Meridian lognormal ROI priors (spend-weighted pooling, lognormal parity with `lognormal_dist_from_mean_std`, channel- and time-scoped `.to_code()` snippet setting `media_prior_type`). Pure numpy/pandas; imports no MMM package; does not introspect result objects. Deriving totals from a fit is deferred to the post-4.0 `results.aggregate()` layer. +- [MMM Calibration Export](https://diff-diff.readthedocs.io/en/stable/api/mmm.html): Assemble Marketing Mix Model calibration inputs from experiment results. Two input routes: explicit already-scoped numbers, or the pinned `AggregationResult` container from post-fit `results.aggregate('simple'|'group')` via `aggregation_result=` + `scale=` (effect = att x scale per row, in `to_dataframe()` order; `scale="auto"` reads the container's treated-obs count and is honored ONLY for ImputationDiD/TwoStageDiD fits - it acknowledges an additive-level outcome, an unweighted fit, and fully identified effects; other estimators and raw results objects fail closed). `to_pymc_marketing_lift_test(channel, x, delta_x, delta_y=, sigma=, aggregation_result=, scale=, dims=, on_wrong_sign=)` builds the PyMC-Marketing/prophetverse lift-test DataFrame with sign/zero/positivity guards. `to_meridian_roi_prior(incremental_outcome=, incremental_outcome_se=, aggregation_result=, scale=, spend, parameter="roi_m"|"mroi_m", se_widening=)` builds Google Meridian lognormal ROI priors (spend-weighted pooling, lognormal parity with `lognormal_dist_from_mean_std`, channel- and time-scoped `.to_code()` snippet setting `media_prior_type`; widen `se_widening` when pooling a group-level container's same-fit cohorts). Pure numpy/pandas; imports no MMM package; never calls `aggregate()` itself. - Conley spatial HAC SE (`vcov_type="conley"`) on cross-sectional `LinearRegression` / `compute_robust_vcov` PLUS panel `DifferenceInDifferences` / `MultiPeriodDiD` / `TwoWayFixedEffects` (with `conley_lag_cutoff=` for within-unit Bartlett temporal HAC) — Conley (1999) spatial-correlation-aware SEs with haversine/euclidean/callable distance metric and Bartlett/uniform spatial kernel; panel path uses the R `conleyreg`-form block-decomposed sandwich (within-period spatial + within-unit Bartlett serial, same-time excluded); parity vs R `conleyreg` (Düsterhöft 2021) on cross-sectional AND panel `lag_cutoff > 0` fixtures. Combining with explicit `cluster=` applies the combined spatial + cluster product kernel `K_total[i,j] = K_space · 1{c_i = c_j}` (cluster must be constant within each unit across periods on the panel path; validator-enforced). DiD takes `unit=` as a fit-time kwarg when `vcov_type="conley"` (not on `__init__`). Sparse k-d-tree fast path auto-activates for `n > 5_000` with bartlett kernel + haversine/euclidean metric ## Tutorials diff --git a/diff_diff/imputation_results.py b/diff_diff/imputation_results.py index d38b2f584..1743e645a 100644 --- a/diff_diff/imputation_results.py +++ b/diff_diff/imputation_results.py @@ -386,7 +386,10 @@ def _aggregate_simple_result(self, kit: Any) -> AggregationResult: CS/EDiD disjoint-units convention cannot apply (the StackedDiD carve-out class); the treated-observation count is the population the overall ATT averages over and matches every other Imputation - row's n semantics. + row's n semantics. Of that population, only finite-tau-hat + observations enter the average - ``n`` reports the raw count, so + on partially unidentified fits (which warn at fit time) ``n`` + exceeds the averaged support. ``df`` is the kit's ``survey_df_final`` snapshot - the exact value the STORED overall ``safe_inference`` received (on a replicate fit diff --git a/diff_diff/mmm.py b/diff_diff/mmm.py index 588d5c863..7599bd8e1 100644 --- a/diff_diff/mmm.py +++ b/diff_diff/mmm.py @@ -20,16 +20,24 @@ ``mu = ln(m) - ln((s/m)^2 + 1)/2``, ``sigma = sqrt(ln((s/m)^2 + 1))``. See https://developers.google.com/meridian/docs/advanced-modeling/set-custom-priors-past-experiments. -**Design: explicit in, validated out.** These functions do NOT rescale a result's -headline ATT. Reconciling an experiment's estimate to a calibration input requires -context diff-diff cannot see - the target MMM's row granularity (per-geo vs national), -its time window, and the outcome's scale (additive levels vs a log/rate/share) - so -the CALLER supplies the already-scoped incremental outcome and its standard error (the -numbers they read off a fitted result's ``summary()``, aggregated to the population and -window their MMM row represents). diff-diff does only what it can verify: assemble the -exact target schema, enforce the sign/positivity/monotonicity guards each consumer -requires, convert to the lognormal parameterization (with parity to Google's closed -form), pool multiple experiments, and emit ready-to-paste snippets. It is pure +**Design: explicit numbers in, or the pinned aggregation contract in - validated out.** +Reconciling an experiment's estimate to a calibration input requires context diff-diff +cannot always see - the target MMM's row granularity (per-geo vs national), its time +window, and the outcome's scale (additive levels vs a log/rate/share). The default +route therefore stays fully explicit: the CALLER supplies the already-scoped +incremental outcome and its standard error (the numbers they read off a fitted +result's ``summary()``, aggregated to the population and window their MMM row +represents). Alternatively, both exporters accept ``aggregation_result=`` - the pinned +:class:`~diff_diff.aggregation.AggregationResult` container returned by +``results.aggregate('simple')`` / ``results.aggregate('group')`` - together with +``scale=``, and derive ``effect = att * scale`` / ``se_out = se * scale`` per row. +The derivation is fail-closed: raw results objects are rejected (this module never +calls ``aggregate()`` itself), and ``scale="auto"`` (reading the row count off the +container) is honored only for the audited producers whose ``n`` matches the ATT's +averaged support on unweighted, fully identified fits. diff-diff does only what it can verify: +assemble the exact target schema, enforce the sign/positivity/monotonicity guards each +consumer requires, convert to the lognormal parameterization (with parity to Google's +closed form), pool multiple experiments, and emit ready-to-paste snippets. It is pure numpy/pandas and never imports an MMM package. """ @@ -39,16 +47,65 @@ import math import warnings from dataclasses import dataclass, field -from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple, Union +from typing import ( + Any, + Dict, + List, + Literal, + Mapping, + Optional, + Sequence, + Tuple, + Union, +) import numpy as np import pandas as pd +# Runtime import (no cycle: aggregation.py depends only on results_base), so the +# exporters' public annotations resolve under typing.get_type_hints(). +from diff_diff.aggregation import AggregationResult +from diff_diff.results_base import BaseResults + __all__ = ["MeridianROIPrior", "to_meridian_roi_prior", "to_pymc_marketing_lift_test"] _WRONG_SIGN_POLICIES = ("raise", "drop", "keep") _LIFT_TEST_RESERVED = frozenset({"channel", "x", "delta_x", "delta_y", "sigma"}) +# Container-mode routing (audited against each producer's aggregation source): +# scale="auto" reads per-row n off the container ONLY for these producers, whose n +# matches the treated observations (unit-periods) the ATT averages over on +# unweighted, fully identified fits, at both +# supported levels. Routing keys on AggregationResult.estimator - NEVER on n_kind +# alone: CallawaySantAnna repeated-cross-section fits report n_kind="obs" with +# n = treated+control observations, so n_kind is only a drift sanity guard here. +_SCALE_AUTO_ESTIMATORS = frozenset({"ImputationDiD", "TwoStageDiD"}) +_CONTAINER_LEVELS = ("simple", "group") + +# Why scale="auto" is refused, per audited non-allowlisted producer. The generic +# closing prescription in the error is an EXAMPLE for the unweighted additive case, +# never a universal formula. +_SCALE_HINTS = { + "CallawaySantAnna": ( + "CallawaySantAnna's n is not a treated unit-period count: the 'simple' " + "container reports treated+control units (treated+control OBSERVATIONS on " + "repeated-cross-section fits, where n_kind='obs' still does not mean " + "treated unit-periods), and the 'group' container reports contributing " + "(g, t) cells (n_kind='cells')." + ), + "EfficientDiD": ( + "EfficientDiD's n counts disjoint treated+control units ('simple') or " + "contributing (g, t) cells ('group'), not treated unit-periods." + ), + "StackedDiD": ( + "StackedDiD's n is a deduplicated distinct-treated-unit count - units, not " + "treated unit-periods, so no per-observation mass exists - and under " + "weighting='population'/'sample_share' the ATT is a weighted estimand for " + "which raw treated exposure is not the right multiplier; derive the scoped " + "total per your weighting choice." + ), +} + # Meridian prior parameters this exporter can target, with each one's Meridian # default LogNormal(mu, sigma) per channel (verified against # meridian/model/prior_distribution.py at 1.7.0): roi_m is the return on a @@ -172,13 +229,155 @@ def _normalize_dims( return dim_cols, rows +def _extract_aggregation_rows( + aggregation_result: Any, + scale: Optional[Union[float, Sequence[float], str]], + *, + effect_name: str, + se_name: str, +) -> Tuple[List[float], List[float], List[Any]]: + """Derive per-row ``(effect, se)`` from a pinned ``AggregationResult``. + + Every per-row value (``label``, ``target``, ``att``, ``se``, ``n``) is read + from ONE ``to_dataframe()`` call, so rows arrive in the order ``summary()`` + prints (sorted by label when sortable, producer order otherwise) - the + alignment order for every per-row sequence kwarg, ``scale`` included. + Fail-closed throughout: anything the container cannot verify raises with the + remedy inline. + """ + if not isinstance(aggregation_result, AggregationResult): + if isinstance(aggregation_result, BaseResults): + raise TypeError( + f"aggregation_result must be an AggregationResult - the container " + f"returned by res.aggregate('simple') or res.aggregate('group') on " + f"estimators that produce one; got " + f"{type(aggregation_result).__name__}. EventStudyResults and " + f"estimators whose aggregate() does not return that container " + f"(WooldridgeDiDResults returns the results object itself; " + f"HeterogeneousAdoptionDiD event-study results support only " + f"'event_study') have no container-mode route - pass the explicit " + f"{effect_name}/{se_name} arguments instead" + ) + raise TypeError( + f"aggregation_result must be an AggregationResult (the return value of " + f"res.aggregate('simple') or res.aggregate('group')); got " + f"{type(aggregation_result).__name__}. Only AggregationResult is " + f"supported." + ) + level = aggregation_result.level + if level not in _CONTAINER_LEVELS: + raise ValueError( + f"aggregation_result.level must be 'simple' or 'group'; got {level!r}. " + f"Other levels ('calendar', 'dose', estimator-specific extras) have no " + f"defined MMM-experiment mapping - re-aggregate at a supported level." + ) + frame = aggregation_result.to_dataframe() + if len(frame) == 0: + raise ValueError( + "aggregation_result has no rows (the aggregation selected no cells); " + "nothing to export" + ) + if level == "simple": + target_list = list(frame["target"].tolist()) + offending = sorted({t for t in target_list if target_list.count(t) > 1}) + if offending: + raise ValueError( + f"aggregation_result has level='simple' but multiple rows for " + f"target(s) {offending!r}; producers emit a single 'overall' row " + f"per target for 'simple', so this container is out of contract - " + f"re-aggregate, or use level='group' for per-cohort rows" + ) + bad_targets = [(i, t) for i, t in enumerate(frame["target"].tolist()) if t != "att"] + if bad_targets: + idxs = [i for i, _ in bad_targets] + targets = sorted({t for _, t in bad_targets}) + raise ValueError( + f"aggregation_result row(s) {idxs} have target {targets!r} where 'att' " + f"is required; only ATT rows map to an MMM incremental outcome " + f"(ContinuousDiD's 'acrt' dose derivative, HeterogeneousAdoptionDiD's " + f"WAS estimands, and ChaisemartinDHaultfoeuille's estimand relays are " + f"not per-unit-period ATTs). The container is rejected whole - no " + f"silent row filtering." + ) + labels = frame["label"].tolist() + atts = [float(v) for v in frame["att"].tolist()] + ses = [float(v) for v in frame["se"].tolist()] + ns = [float(v) for v in frame["n"].tolist()] + for i, (att_i, se_i) in enumerate(zip(atts, ses)): + if not (math.isfinite(att_i) and math.isfinite(se_i) and se_i > 0): + raise ValueError( + f"aggregation_result row [{i}] (label {labels[i]!r}) has " + f"att={att_i!r}, se={se_i!r}; the fit carries no usable point " + f"estimate/SE for this row, so it cannot calibrate an MMM" + ) + n_rows = len(frame) + if isinstance(scale, str): + if scale != "auto": + raise ValueError( + f"scale must be a number, a sequence of numbers, or the string " + f"'auto'; got {scale!r}" + ) + estimator = aggregation_result.estimator + if estimator not in _SCALE_AUTO_ESTIMATORS: + if estimator is not None and estimator in _SCALE_HINTS: + hint = _SCALE_HINTS[estimator] + else: + hint = ( + f"estimator provenance is {estimator!r}, whose n semantics " + f"diff-diff has not audited for this derivation." + ) + raise ValueError( + f"scale='auto' is not available for this container: {hint} Pass a " + f"numeric scale=, in to_dataframe() " + f"order; auto-derivation is audited only for ImputationDiD and " + f"TwoStageDiD, whose n is the treated observations the ATT " + f"averages over." + ) + n_kind = aggregation_result.n_kind + if n_kind != "obs": + raise ValueError( + f"aggregation_result from {estimator} reports n_kind={n_kind!r}, " + f"expected 'obs'; the container schema has drifted from the " + f"audited contract - pass a numeric scale explicitly" + ) + for i, n_i in enumerate(ns): + if not (math.isfinite(n_i) and n_i > 0): + raise ValueError( + f"aggregation_result row [{i}] (label {labels[i]!r}) has " + f"n={n_i!r}; cannot auto-derive scale - pass a numeric scale" + ) + scales = ns + elif scale is not None: + scales = [ + _finite_positive("scale", v, i) + for i, v in enumerate(_broadcast("scale", scale, n_rows)) + ] + else: + raise ValueError( + f"scale is required with aggregation_result: pass a numeric " + f"scale= (scalar or one value per row in " + f"to_dataframe() order), or scale='auto' (ImputationDiD/TwoStageDiD " + f"fits only - see the {effect_name} docstring for the assumptions " + f"'auto' acknowledges)" + ) + effects = [att_i * s_i for att_i, s_i in zip(atts, scales)] + ses_out = [se_i * s_i for se_i, s_i in zip(ses, scales)] + return effects, ses_out, labels + + def to_pymc_marketing_lift_test( *, channel: Union[str, Sequence[str]], x: Union[float, Sequence[float]], delta_x: Union[float, Sequence[float]], - delta_y: Union[float, Sequence[float]], - sigma: Union[float, Sequence[float]], + delta_y: Optional[Union[float, Sequence[float]]] = None, + sigma: Optional[Union[float, Sequence[float]]] = None, + aggregation_result: Optional[AggregationResult] = None, + scale: Optional[Union[float, Sequence[float], Literal["auto"]]] = None, dims: Optional[Union[Mapping[str, str], Sequence[Mapping[str, str]]]] = None, on_wrong_sign: str = "raise", ) -> pd.DataFrame: @@ -190,15 +389,19 @@ def to_pymc_marketing_lift_test( API accepts the same shape). All values stay in original data units (PyMC-Marketing rescales internally). - **The caller supplies the scoped effect.** ``delta_y`` and ``sigma`` are the - measured incremental outcome and its standard error, already aggregated to the - population and time window ONE target-MMM row represents. This function does not - rescale an ATT, because the reconciliation needs the MMM's row granularity and - outcome scale, which it cannot see. Read the effect and SE off your fitted - result's ``summary()`` and scope them yourself: PyMC-Marketing scores one row's - ``delta_y`` against ``saturation(x + delta_x) - saturation(x)``, so ``x``, - ``delta_x``, ``delta_y``, and ``sigma`` must all describe the SAME observation - (same channel, same population, same period span, additive-level outcome). + **Two input routes.** Either the caller supplies the scoped effect explicitly - + ``delta_y`` and ``sigma``, the measured incremental outcome and its standard + error already aggregated to the population and time window ONE target-MMM row + represents - or passes ``aggregation_result=`` (the pinned container returned by + ``results.aggregate('simple')`` / ``results.aggregate('group')``) together with + ``scale=``, and this function derives ``delta_y = att * scale`` and + ``sigma = se * scale`` per container row. Rescaling is performed ONLY under that + explicit contract - reconciliation context the container cannot carry (the MMM's + row granularity, the outcome's scale) remains the caller's acknowledgement, see + ``scale``. Either way, PyMC-Marketing scores one row's ``delta_y`` against + ``saturation(x + delta_x) - saturation(x)``, so ``x``, ``delta_x``, ``delta_y``, + and ``sigma`` must all describe the SAME observation (same channel, same + population, same period span, additive-level outcome). Parameters ---------- @@ -209,12 +412,58 @@ def to_pymc_marketing_lift_test( delta_x : float or sequence of float Spend change during the experiment (nonzero; negative for go-dark/holdout tests, with ``x + delta_x >= 0``), in the same units as ``x``. - delta_y : float or sequence of float + delta_y : float or sequence of float, optional Measured incremental outcome for the SAME observation as ``x``/``delta_x``, in original outcome units. Finite; must be nonzero and share ``delta_x``'s - sign (see ``on_wrong_sign``). - sigma : float or sequence of float - Standard error of ``delta_y`` (finite, positive). + sign (see ``on_wrong_sign``). Required together with ``sigma`` unless + ``aggregation_result`` is given (the two routes are mutually exclusive). + sigma : float or sequence of float, optional + Standard error of ``delta_y`` (finite, positive). Required together with + ``delta_y`` unless ``aggregation_result`` is given. + aggregation_result : AggregationResult, optional + The pinned container returned by ``results.aggregate('simple')`` (one + experiment row) or ``results.aggregate('group')`` (one row per cohort). + Mutually exclusive with ``delta_y``/``sigma``; requires ``scale``. Rows are + consumed in ``aggregation_result.to_dataframe()`` order - the order + ``summary()`` prints - and every per-row sequence kwarg (``scale``, ``x``, + ``delta_x``, ``dims``, ...) aligns to that order. Raw results objects and + ``EventStudyResults`` are rejected; this function never calls + ``aggregate()`` itself. On a bootstrapped fit, ``aggregate('simple')`` + relays the stored percentile SE, which is used as-is. Group-container + cautions: (1) cohort rows come from ONE fit sharing controls and windows, + yet each emitted lift row is scored by PyMC-Marketing as an independent + observation with only its marginal ``sigma`` - the omitted cross-cohort + covariance can misstate the joint evidence (overstating it when the net + covariance is positive, as shared controls typically induce), and its + direction cannot be determined from marginal SEs alone; (2) a scalar + ``x`` or + ``delta_x`` replicates to every cohort row, asserting the channel's FULL + baseline spend / spend change produced only that cohort's ``delta_y`` - + pass per-cohort values; (3) the emitted frame carries NO cohort-label + column, so pass per-row ``dims`` to keep multi-cohort rows + distinguishable. + scale : float, sequence of float, or "auto", optional + Converts each container row's per-observation ATT to the row's total + incremental outcome: ``delta_y = att * scale``, ``sigma = se * scale``. + Required with ``aggregation_result`` (finite, positive; scalar or one value + per row in ``to_dataframe()`` order; e.g. treated units x treated periods + for an unweighted additive fit - an example, not a universal formula). + ``scale="auto"`` derives ``scale`` from the container's per-row ``n`` and is + accepted ONLY for ImputationDiD and TwoStageDiD fits, whose ``n`` is the + treated observations the ATT averages over. Passing ``"auto"`` acknowledges + three assumptions the container cannot verify: the outcome is in additive + levels (not log/rate/share); the fit is unweighted (on survey-weighted fits + ``att`` is a weighted average while ``n`` is a raw count); and every + treated observation's effect is identified (ImputationDiD averages finite + tau-hat only while ``n`` stays the raw count at both levels; TwoStageDiD's + 'simple' ATT support excludes rows with non-finite first-stage residuals + while ``n`` reports the pre-filter count, though its 'group' counts are + post-filter - the affected fits warn at fit time on the degenerate + branch, and ``"auto"`` there overcounts). An + ``att * scale`` (or ``se * scale``) that overflows the float range + surfaces via this function's ordinary finiteness errors - ``sigma`` is + validated before ``delta_y``, so a row where both overflow reports + ``sigma``. dims : mapping or sequence of mappings, optional Extra model-dimension columns, e.g. ``{"geo": "US-CA"}`` for a geo-level MMM built with ``MMM(dims=("geo",))``. Values must match the target model's @@ -249,7 +498,36 @@ def to_pymc_marketing_lift_test( f"on_wrong_sign must be one of {_WRONG_SIGN_POLICIES}; got {on_wrong_sign!r}" ) - n = _seq_len(channel, x, delta_x, delta_y, sigma, dims) + if aggregation_result is not None: + if delta_y is not None or sigma is not None: + raise ValueError( + "pass either aggregation_result= or delta_y=/sigma=, not both; " + "aggregation mode derives delta_y and sigma from the container " + "(delta_y = att * scale, sigma = se * scale)" + ) + delta_y, sigma, _ = _extract_aggregation_rows( + aggregation_result, scale, effect_name="delta_y", se_name="sigma" + ) + n = len(delta_y) + else: + if scale is not None: + raise ValueError( + "scale only applies with aggregation_result=; without a container, " + "pass the already-scaled delta_y and sigma directly" + ) + if delta_y is None or sigma is None: + missing = ( + "delta_y and sigma are" + if delta_y is None and sigma is None + else ("delta_y is" if delta_y is None else "sigma is") + ) + raise ValueError( + f"{missing} required when aggregation_result is not given; pass " + f"both delta_y and sigma, or pass " + f"aggregation_result=res.aggregate('simple'|'group') with scale= " + f"to derive them from a fitted result" + ) + n = _seq_len(channel, x, delta_x, delta_y, sigma, dims) channels = _broadcast("channel", channel, n) xs = _broadcast("x", x, n) delta_xs = _broadcast("delta_x", delta_x, n) @@ -499,8 +777,10 @@ def to_code( def to_meridian_roi_prior( *, - incremental_outcome: Union[float, Sequence[float]], - incremental_outcome_se: Union[float, Sequence[float]], + incremental_outcome: Optional[Union[float, Sequence[float]]] = None, + incremental_outcome_se: Optional[Union[float, Sequence[float]]] = None, + aggregation_result: Optional[AggregationResult] = None, + scale: Optional[Union[float, Sequence[float], Literal["auto"]]] = None, spend: Union[float, Sequence[float]], parameter: str = "roi_m", se_widening: float = 1.0, @@ -517,24 +797,73 @@ def to_meridian_roi_prior( independent - widen via ``se_widening`` when experiments share controls or windows). The pooled mean/sd map to lognormal ``(mu, sigma)`` via Google's closed form. - **The caller supplies the scoped estimand.** ``incremental_outcome`` is the total - incremental outcome the experiment measured, matching the estimand of the target - prior: for ``parameter="roi_m"`` that is the outcome attributable to the channel's - spend against a zero-spend counterfactual (a full holdout), divided here by - ``spend`` = the channel's total spend over the window; for ``parameter="mroi_m"`` - it is the marginal outcome of the spend change, divided by that spend change. Sign, - aggregation, and population are the caller's responsibility - diff-diff does not - rescale a headline ATT. + **Two input routes.** Either the caller supplies the scoped estimand explicitly - + ``incremental_outcome`` is the total incremental outcome the experiment measured, + matching the estimand of the target prior: for ``parameter="roi_m"`` that is the + outcome attributable to the channel's spend against a zero-spend counterfactual + (a full holdout), divided here by ``spend`` = the channel's total spend over the + window; for ``parameter="mroi_m"`` it is the marginal outcome of the spend + change, divided by that spend change - or passes ``aggregation_result=`` with + ``scale=``, and this function derives ``incremental_outcome = att * scale`` and + ``incremental_outcome_se = se * scale`` per container row. Rescaling happens ONLY + under that explicit contract; sign, estimand match, and population remain the + caller's responsibility. Parameters ---------- - incremental_outcome : float or sequence of float + incremental_outcome : float or sequence of float, optional Total incremental outcome per experiment (the estimand matching ``parameter``), in the same currency/units as the MMM. Must be finite; the pooled ROI must be - positive (lognormal support). - incremental_outcome_se : float or sequence of float + positive (lognormal support). Required together with + ``incremental_outcome_se`` unless ``aggregation_result`` is given (the two + routes are mutually exclusive). + incremental_outcome_se : float or sequence of float, optional Standard error of ``incremental_outcome`` (finite, positive), computed for the - caller's aggregation. + caller's aggregation. Required together with ``incremental_outcome`` unless + ``aggregation_result`` is given. + aggregation_result : AggregationResult, optional + The pinned container returned by ``results.aggregate('simple')`` (one + experiment) or ``results.aggregate('group')`` (one experiment per cohort, + feeding the spend-weighted pooling below). Mutually exclusive with + ``incremental_outcome``/``incremental_outcome_se``; requires ``scale``. Rows + are consumed in ``aggregation_result.to_dataframe()`` order - the order + ``summary()`` prints - and per-row sequence kwargs (``scale``, ``spend``) + align to that order. Raw results objects and ``EventStudyResults`` are + rejected; this function never calls ``aggregate()`` itself. Group-container + cautions: (1) cohort rows come from ONE fit sharing controls and windows, + and the container carries only per-row SEs (the joint covariance's + off-diagonals are discarded upstream), so the independence pooling below + can misstate the pooled uncertainty - it is anti-conservative when the + net weighted covariance is positive (shared controls typically induce + positive correlation), and the direction cannot be determined from + marginal SEs alone; ``se_widening > 1`` is a conservative heuristic, not + an exact correction; (2) a scalar + ``spend`` replicates per row, making ``total_spend = n_rows * spend`` and + the pooled prior the arithmetic MEAN of cohort ROIs, + ``sum(effect_i) / (n_rows * spend)`` - an n-fold understatement of channel + ROI if the scalar was meant as the channel's TOTAL spend; pass per-cohort + spends. + scale : float, sequence of float, or "auto", optional + Converts each container row's per-observation ATT to that experiment's total + incremental outcome: ``incremental_outcome = att * scale``, + ``incremental_outcome_se = se * scale``. Required with + ``aggregation_result`` (finite, positive; scalar or one value per row in + ``to_dataframe()`` order; e.g. treated units x treated periods for an + unweighted additive fit - an example, not a universal formula). + ``scale="auto"`` derives ``scale`` from the container's per-row ``n`` and is + accepted ONLY for ImputationDiD and TwoStageDiD fits, whose ``n`` is the + treated observations the ATT averages over. Passing ``"auto"`` acknowledges + three assumptions the container cannot verify: additive-level outcome (not + log/rate/share), an unweighted fit (survey-weighted fits pair a weighted + ``att`` with a raw-count ``n``), and fully identified effects + (ImputationDiD reports the raw treated-observation count at BOTH levels + while averaging finite effects only; TwoStageDiD does so only at + ``'simple'`` - its ``'group'`` counts are post-filter; the affected fits + warn at fit time on the degenerate branch, and ``"auto"`` there + overcounts). An overflowing ``att * scale`` surfaces via + this function's ordinary finiteness errors - ``incremental_outcome`` is + validated before ``incremental_outcome_se``, so a row where both overflow + reports ``incremental_outcome``. spend : float or sequence of float Spend the incremental outcome is divided by (finite, positive): the channel's total spend over the window for ``roi_m``, or the spend change for ``mroi_m``. @@ -575,7 +904,45 @@ def to_meridian_roi_prior( if not math.isfinite(se_w) or se_w <= 0: raise ValueError(f"se_widening must be finite and > 0; got {se_widening!r}") - n = _seq_len(incremental_outcome, incremental_outcome_se, spend) + if aggregation_result is not None: + if incremental_outcome is not None or incremental_outcome_se is not None: + raise ValueError( + "pass either aggregation_result= or " + "incremental_outcome=/incremental_outcome_se=, not both; " + "aggregation mode derives them from the container " + "(incremental_outcome = att * scale, " + "incremental_outcome_se = se * scale)" + ) + incremental_outcome, incremental_outcome_se, _ = _extract_aggregation_rows( + aggregation_result, + scale, + effect_name="incremental_outcome", + se_name="incremental_outcome_se", + ) + n = len(incremental_outcome) + else: + if scale is not None: + raise ValueError( + "scale only applies with aggregation_result=; without a container, " + "pass the already-scaled incremental_outcome and " + "incremental_outcome_se directly" + ) + if incremental_outcome is None or incremental_outcome_se is None: + missing = ( + "incremental_outcome and incremental_outcome_se are" + if incremental_outcome is None and incremental_outcome_se is None + else ( + "incremental_outcome is" + if incremental_outcome is None + else "incremental_outcome_se is" + ) + ) + raise ValueError( + f"{missing} required when aggregation_result is not given; pass " + f"both, or pass aggregation_result=res.aggregate('simple'|'group') " + f"with scale= to derive them from a fitted result" + ) + n = _seq_len(incremental_outcome, incremental_outcome_se, spend) outcomes = _broadcast("incremental_outcome", incremental_outcome, n) outcome_ses = _broadcast("incremental_outcome_se", incremental_outcome_se, n) spends = _broadcast("spend", spend, n) diff --git a/diff_diff/two_stage_results.py b/diff_diff/two_stage_results.py index e3c1125e8..1d1ac824c 100644 --- a/diff_diff/two_stage_results.py +++ b/diff_diff/two_stage_results.py @@ -385,12 +385,15 @@ def _aggregate_compute( def _aggregate_simple_result(self, kit: Any) -> AggregationResult: """One-row relay of the stored overall inference (bit-exact). - ``n = n_treated_obs`` (|Omega_1|, the D-column support the ATT - averages over) with ``n_kind="obs"``: TwoStageDiD's + ``n = n_treated_obs`` (|Omega_1|, the pre-filter D-column count) + with ``n_kind="obs"``: TwoStageDiD's ``n_treated_units``/``n_control_units`` unit sets OVERLAP (an eventually-treated unit contributes untreated observations), so the CS/EDiD disjoint-units convention cannot apply (the StackedDiD - carve-out class). + carve-out class). The ATT's actual Stage-2 support excludes rows + whose ``y_tilde`` is non-finite (their treatment indicator is + zeroed before the OLS), so on such degenerate fits - which warn at + fit time - ``n`` exceeds the averaged support. ``df`` is the kit's ``survey_df_final`` snapshot - the exact value the STORED overall ``safe_inference`` received (on a replicate fit diff --git a/docs/api/mmm.rst b/docs/api/mmm.rst index a7caa384c..34b1b0550 100644 --- a/docs/api/mmm.rst +++ b/docs/api/mmm.rst @@ -22,16 +22,24 @@ dominant Python MMM frameworks consume that evidence in different shapes: error to the prior standard deviation, converted to lognormal ``(mu, sigma)`` with Google's closed form. -**Explicit in, validated out.** These functions do not rescale a result's headline -ATT. Reconciling an experiment's estimate to a calibration input needs context -diff-diff cannot see - the target MMM's row granularity (per-geo vs national), its -time window, and the outcome's scale (additive levels vs a log/rate/share) - so the -caller supplies the already-scoped incremental outcome and its standard error (the -numbers read off a fitted result's ``summary()``, aggregated to the population and -window one MMM row represents). diff-diff does only what it can verify: assemble the -exact schema, enforce each consumer's guards, convert to the lognormal -parameterization (parity with Google's closed form), pool, and emit snippets. It is -pure numpy/pandas and imports no MMM package. +**Explicit numbers in, or the pinned aggregation contract in - validated out.** +Reconciling an experiment's estimate to a calibration input needs context diff-diff +cannot always see - the target MMM's row granularity (per-geo vs national), its +time window, and the outcome's scale (additive levels vs a log/rate/share). The +default route stays fully explicit: the caller supplies the already-scoped +incremental outcome and its standard error (the numbers read off a fitted result's +``summary()``, aggregated to the population and window one MMM row represents). +Alternatively both exporters accept ``aggregation_result=`` - the pinned +:class:`~diff_diff.AggregationResult` returned by ``results.aggregate('simple')`` +or ``results.aggregate('group')`` - with ``scale=``, deriving +``effect = att * scale`` and ``se = se * scale`` per row; ``scale="auto"`` (reading +the container's own treated-observation count) is honored only for ImputationDiD +and TwoStageDiD fits, and acknowledges assumptions the container cannot verify +(additive-level outcome, unweighted fit, fully identified effects - see the +docstrings). diff-diff does only what it can verify: assemble the exact schema, +enforce each consumer's guards, convert to the lognormal parameterization (parity +with Google's closed form), pool, and emit snippets. It is pure numpy/pandas and +imports no MMM package. to_pymc_marketing_lift_test --------------------------- @@ -45,6 +53,9 @@ Example .. code-block:: python + import numpy as np + import pandas as pd + from diff_diff import SyntheticDiD, to_pymc_marketing_lift_test # Single-treated-geo experiment (US-CA): TV spend raised there vs control @@ -53,8 +64,22 @@ Example # treated geos the pooled ATT is an average - do not assign it to one geo; # export each geo's own effect, or omit the geo dim and match aggregate # spend to an aggregate lift.) + rng = np.random.default_rng(42) + geos = [f"g{i}" for i in range(9)] + ["US-CA"] + panel = pd.DataFrame( + { + "geo": g, + "week": w, + "treated": int(g == "US-CA"), # block treatment: constant per unit + "revenue": 100.0 + 2.0 * w + rng.normal(0.0, 1.0) + + (8.0 if g == "US-CA" and w >= 8 else 0.0), + } + for g in geos + for w in range(12) + ) result = SyntheticDiD().fit(panel, outcome='revenue', treatment='treated', - unit='geo', time='week') + unit='geo', time='week', + post_periods=list(range(8, 12))) df_lift = to_pymc_marketing_lift_test( channel='tv', @@ -67,6 +92,33 @@ Example # -> columns [channel, geo, x, delta_x, delta_y, sigma], ready for # MMM.add_lift_test_measurements(df_lift) +Deriving totals from a fitted aggregation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For ImputationDiD and TwoStageDiD fits, the post-fit aggregation container can +supply the effect numbers directly - ``scale="auto"`` multiplies the per-observation +ATT by the container's treated-observation count (an explicit acknowledgement that +the outcome is in additive levels, the fit is unweighted, and every treated +observation's effect is identified; see the docstring): + +.. code-block:: python + + from diff_diff import ImputationDiD, to_pymc_marketing_lift_test + from diff_diff.prep import generate_staggered_data + + data = generate_staggered_data(n_units=60, n_periods=6, seed=11) + res = ImputationDiD().fit(data, outcome='outcome', unit='unit', + time='period', first_treat='first_treat') + + df_lift = to_pymc_marketing_lift_test( + channel='tv', + x=50_000.0, + delta_x=20_000.0, + aggregation_result=res.aggregate('simple'), # one experiment row + scale='auto', # ImputationDiD's n IS the treated unit-periods + ) + # delta_y == att * n, sigma == se * n, then the usual guards apply. + to_meridian_roi_prior --------------------- @@ -80,13 +132,17 @@ Example .. code-block:: python from diff_diff import DifferenceInDifferences, to_meridian_roi_prior + from diff_diff.prep import generate_did_data - result = DifferenceInDifferences().fit(panel, outcome='revenue', + panel = generate_did_data(n_units=60, n_periods=2, treatment_effect=5.0, + treatment_period=1, seed=7) + result = DifferenceInDifferences().fit(panel, outcome='outcome', treatment='treated', post='post') - # Caller aggregates the ATT to a total incremental outcome over the treated - # population and window (e.g. att x treated units x post periods) and supplies - # its SE - diff-diff does not rescale the headline effect. + # On the explicit route the caller aggregates the ATT to a total incremental + # outcome over the treated population and window (e.g. att x treated units x + # post periods for an unweighted additive fit) and supplies its SE; the + # aggregation-container route below derives the numbers instead. prior = to_meridian_roi_prior( incremental_outcome=180_000.0, # total incremental revenue incremental_outcome_se=45_000.0, # its standard error @@ -101,6 +157,30 @@ Example print(prior.to_code(channel='tv', media_channels=['search', 'tv'], roi_calibration_period='experiment_window_mask')) +Deriving totals from a fitted aggregation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Estimators outside the ``scale="auto"`` allowlist still hand the exporter their +effect and SE via the container - the caller supplies the numeric scale, because +those containers' ``n`` does not count treated unit-periods (CallawaySantAnna's +``simple`` container, for example, counts treated *and* control units): + +.. code-block:: python + + from diff_diff import CallawaySantAnna, to_meridian_roi_prior + from diff_diff.prep import generate_staggered_data + + data = generate_staggered_data(n_units=60, n_periods=6, seed=11) + cs = CallawaySantAnna().fit(data, outcome='outcome', unit='unit', + time='period', first_treat='first_treat') + + prior = to_meridian_roi_prior( + aggregation_result=cs.aggregate('simple'), + scale=132.0, # caller-derived treated unit-periods for THEIR scoping + spend=200_000.0, + ) + # incremental_outcome == att * 132.0, and its SE == se * 132.0. + MeridianROIPrior ---------------- diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index 8475f54ba..1379f5c7b 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -1976,7 +1976,7 @@ where `W_it(h) = 1[K_it = h]` are lead indicators, estimated on `Omega_0` only. - [x] Supports balanced and unbalanced panels (iterative Gauss-Seidel demeaning for exact FE) - [x] Event study and group aggregation -- **Note (post-fit aggregate() - rows M-021/M-118):** `fit(aggregate=, balance_e=)` is deprecated (3.9; removed 4.0; joint FutureWarning, warn-and-still-work; the `imputation_did` wrapper forwards the shared sentinel so plain wrapper calls never fire the aggregate warning; since 3.9 the wrapper itself warns per M-070) in favor of post-fit `ImputationDiDResults.aggregate(type, balance_e=)` - a PANEL-BACKED lazy recompute kit (not an EIF-payload kit: ES/group aggregation is a target-specific Theorem-3 recompute - each `balance_e` re-masks which treated observations enter every horizon and re-solves the untreated projection - so no compact influence payload can replace the frame). (a) RETAINED BUFFERS (memory contract): the kit's bookkeeping holds REFERENCES to the SAME per-fit objects `_fit_data` already retains for `pretrend_test()` - the working panel copy (all user columns plus `_tau_hat`/`_rel_time`/`_never_treated`), the Omega masks, `unit_fe`/`time_fe`/`grand_mean`/`delta_hat`/`kept_cov_mask`, the resolved survey design, and `survey_weights` - ZERO marginal memory, and pickles are unchanged via memoization (`_estimator_ref` already ships these objects); plus value SNAPSHOTS for isolation (a `treatment_groups` copy, config scalars, a `dataclasses.replace` copy of `survey_metadata`, `overall_att`, `n_treated_obs`) and TWO df-provenance scalars (`survey_df_seed`, what the analytical aggregators received; `survey_df_final`, what the stored overall inference received). Each `aggregate()` call runs on a fresh throwaway host with a call-local projection cache (the fit-local factorizations are unpicklable and never retained). (b) `balance_e` uses the BALANCED-WINDOW rule: a cohort is retained iff its observed relative-time set - checked against the FULL panel via `_build_cohort_rel_times()` - covers the contiguous window `[-balance_e, max_h]`; the SAME rule TwoStageDiD uses, divergent from CS/EfficientDiD's anchor-horizon rule. A window no cohort satisfies warns and yields the reference-marker-only dict (a legal near-empty container). (c) BOOTSTRAP fits: 'simple' RELAYS the stored overall quintet verbatim (finite safe_inference t included) with a NaN df column, while the RECOMPUTE levels fail closed (the per-target psi machinery makes exact replay tractable - a TODO row); the prior uniform fail-closed rule was superseded 2026-08-05 with the M-027 per-level convergence. (d) CONTAINER ADMISSION into `compute_honest_did`/`compute_pretrends_power` is REJECTED BY DESIGN (both terminal TypeErrors state it): the surface carries no joint event-study covariance - per-horizon conservative SEs only (container `vcov=None`; the scalar `df_survey` channel is its only df provenance, the per-row hole being the tracked M-092-completion TODO row). (e) RELAY CONVENTIONS: 'simple' relays the stored overall quintet bit-exact with `n = n_treated_obs`, `n_kind="obs"` (the treated/control UNIT sets overlap - a treated unit with pre-periods counts in both - so the CS/EDiD disjoint-units convention cannot apply; |Omega_1| is the population the ATT averages over) and `df = survey_df_final`; 'group' rows carry per-row `df_used` captured at each row's `safe_inference` (the replicate override rewrites it, the bootstrap override clears it, the all-NaN cohort branch writes no key - consumers read via `.get`); 'event_study' rides the shared `_from_relative_dict` builder via a carrier whose metadata is a copy-on-use of the KIT's fit-final metadata copy. REPLICATE-WEIGHT fits replay the extracted `_replicate_override_aggregates` with a LEVEL-MATCHED stack: `compute_replicate_refit_variance` validates replicates JOINTLY (all-finite rows), so `aggregate(L)` reproduces `fit(aggregate=L)` exactly, a `fit(aggregate='all')` surface is NOT the equivalence target when a replicate NaNs on exactly one family's targets, and - the documented migration delta - moving a replicate fit from `fit(aggregate=)` to plain fit + post-fit `aggregate()` can change the public OVERALL row's se/CI/df on such degenerate designs (each surface self-consistent; pinned in the contract tests). `pretrends=True` + replicate: post-fit `aggregate('event_study')` raises the same NotImplementedError the fit-time gate raises (per-replicate lead refits unimplemented); 'group'/'simple' still work. Recompute re-emits the fit-time warnings (LSMR, Prop-5, empty-window) with fit-tuned stacklevels - post-fit attribution lands on a library frame, an accepted verbatim-move trade-off. +- **Note (post-fit aggregate() - rows M-021/M-118):** `fit(aggregate=, balance_e=)` is deprecated (3.9; removed 4.0; joint FutureWarning, warn-and-still-work; the `imputation_did` wrapper forwards the shared sentinel so plain wrapper calls never fire the aggregate warning; since 3.9 the wrapper itself warns per M-070) in favor of post-fit `ImputationDiDResults.aggregate(type, balance_e=)` - a PANEL-BACKED lazy recompute kit (not an EIF-payload kit: ES/group aggregation is a target-specific Theorem-3 recompute - each `balance_e` re-masks which treated observations enter every horizon and re-solves the untreated projection - so no compact influence payload can replace the frame). (a) RETAINED BUFFERS (memory contract): the kit's bookkeeping holds REFERENCES to the SAME per-fit objects `_fit_data` already retains for `pretrend_test()` - the working panel copy (all user columns plus `_tau_hat`/`_rel_time`/`_never_treated`), the Omega masks, `unit_fe`/`time_fe`/`grand_mean`/`delta_hat`/`kept_cov_mask`, the resolved survey design, and `survey_weights` - ZERO marginal memory, and pickles are unchanged via memoization (`_estimator_ref` already ships these objects); plus value SNAPSHOTS for isolation (a `treatment_groups` copy, config scalars, a `dataclasses.replace` copy of `survey_metadata`, `overall_att`, `n_treated_obs`) and TWO df-provenance scalars (`survey_df_seed`, what the analytical aggregators received; `survey_df_final`, what the stored overall inference received). Each `aggregate()` call runs on a fresh throwaway host with a call-local projection cache (the fit-local factorizations are unpicklable and never retained). (b) `balance_e` uses the BALANCED-WINDOW rule: a cohort is retained iff its observed relative-time set - checked against the FULL panel via `_build_cohort_rel_times()` - covers the contiguous window `[-balance_e, max_h]`; the SAME rule TwoStageDiD uses, divergent from CS/EfficientDiD's anchor-horizon rule. A window no cohort satisfies warns and yields the reference-marker-only dict (a legal near-empty container). (c) BOOTSTRAP fits: 'simple' RELAYS the stored overall quintet verbatim (finite safe_inference t included) with a NaN df column, while the RECOMPUTE levels fail closed (the per-target psi machinery makes exact replay tractable - a TODO row); the prior uniform fail-closed rule was superseded 2026-08-05 with the M-027 per-level convergence. (d) CONTAINER ADMISSION into `compute_honest_did`/`compute_pretrends_power` is REJECTED BY DESIGN (both terminal TypeErrors state it): the surface carries no joint event-study covariance - per-horizon conservative SEs only (container `vcov=None`; the scalar `df_survey` channel is its only df provenance, the per-row hole being the tracked M-092-completion TODO row). (e) RELAY CONVENTIONS: 'simple' relays the stored overall quintet bit-exact with `n = n_treated_obs`, `n_kind="obs"` (the treated/control UNIT sets overlap - a treated unit with pre-periods counts in both - so the CS/EDiD disjoint-units convention cannot apply; |Omega_1| is the population the ATT averages over, of which only finite-tau-hat observations enter the average - `n` reports the raw count, so on partially unidentified fits `n` exceeds the averaged support) and `df = survey_df_final`; 'group' rows carry per-row `df_used` captured at each row's `safe_inference` (the replicate override rewrites it, the bootstrap override clears it, the all-NaN cohort branch writes no key - consumers read via `.get`); 'event_study' rides the shared `_from_relative_dict` builder via a carrier whose metadata is a copy-on-use of the KIT's fit-final metadata copy. REPLICATE-WEIGHT fits replay the extracted `_replicate_override_aggregates` with a LEVEL-MATCHED stack: `compute_replicate_refit_variance` validates replicates JOINTLY (all-finite rows), so `aggregate(L)` reproduces `fit(aggregate=L)` exactly, a `fit(aggregate='all')` surface is NOT the equivalence target when a replicate NaNs on exactly one family's targets, and - the documented migration delta - moving a replicate fit from `fit(aggregate=)` to plain fit + post-fit `aggregate()` can change the public OVERALL row's se/CI/df on such degenerate designs (each surface self-consistent; pinned in the contract tests). `pretrends=True` + replicate: post-fit `aggregate('event_study')` raises the same NotImplementedError the fit-time gate raises (per-replicate lead refits unimplemented); 'group'/'simple' still work. Recompute re-emits the fit-time warnings (LSMR, Prop-5, empty-window) with fit-tuned stacklevels - post-fit attribution lands on a library frame, an accepted verbatim-move trade-off. --- @@ -2065,7 +2065,7 @@ Our implementation uses multiplier bootstrap on the GMM influence function: clus - [x] Multiplier bootstrap on GMM influence function - [x] Event study and overall ATT aggregation -- **Note (post-fit aggregate() - rows M-022/M-119):** `fit(aggregate=, balance_e=)` is deprecated (3.9; removed 4.0; joint FutureWarning, warn-and-still-work; the `two_stage_did` wrapper forwards the shared sentinel so plain wrapper calls never fire the aggregate warning; since 3.9 the wrapper itself warns per M-071) in favor of post-fit `TwoStageDiDResults.aggregate(type, balance_e=)` - a PANEL-BACKED lazy recompute kit: each level is a fresh Stage-2 OLS + joint Gardner-GMM sandwich on a level-specific design, so no compact influence payload exists. (a) RETAINED BUFFERS (memory contract - the FIRST panel retention on TwoStageDiD results, a deliberate break from the CS/EDiD identifier-minimization guarantee, with a `store_kit` opt-out tracked in DEFERRED.md): a COLUMN-SUBSET COPY of the working frame - `unit`/`time`/`outcome`/`first_treat` + covariates + the cluster column (deduplicated: `cluster=` may legally name a core column) + `_never_treated`/`_rel_time`/`_y_tilde` - O(n_obs) on every results object and pickle; the Stage-1 FE model (`unit_fe`/`time_fe`/`grand_mean`/`delta_hat`/`kept_cov_mask`), the Omega masks, the full-domain `keep_mask`, the Wave-E.3-GATED `score_pad_mask`/`cluster_ids_full` values fit actually passed (None unless the always-treated pad was active), `survey_weights`, and the resolved survey design - on replicate designs that adds the O(n_obs x R) replicate matrix; plus value snapshots (`treatment_groups` copy, `ref_period`, `overall_att`, `n_treated_obs`, a `dataclasses.replace` copy of `survey_metadata`) and TWO df scalars (`survey_df_stage2`, the recompute seed; `survey_df_final`, what the stored overall inference received). (b) `balance_e` uses the BALANCED-WINDOW rule (`[-balance_e, max_h]` coverage against the full panel - the ImputationDiD rule, divergent from CS/EfficientDiD's anchor-horizon rule); zero qualifying cohorts warns and yields the reference-row-only dict with `vcov=None`. (c) BOOTSTRAP fits: 'simple' RELAYS the stored overall quintet verbatim (finite safe_inference t included) with a NaN df column, while the RECOMPUTE levels fail closed (per-level GMM scores are function-locals; replay is a TODO row) - the prior uniform rule superseded 2026-08-05 with the M-027 per-level convergence; a fit whose bootstrap FAILED (`bootstrap_results=None`, analytical inference retained) aggregates normally. (d) CONTAINER ADMISSION into `compute_honest_did`/`compute_pretrends_power` is DEFERRED, not by-design (both terminal TypeErrors state it): analytical surfaces DO carry the real joint Gardner-GMM covariance (M-092), but the pre-period coefficients are stage-1 residual MEANS - the reference horizon is dropped from the no-intercept Stage-2 design and the zero anchor row is appended mechanically - not contrasts against the advertised reference, while HonestDiD's Delta^RM/Delta^SD arithmetic hard-codes the `delta_0 = 0` normalization into its boundary/bridge constraints; admission awaits a normalization derivation (either re-estimating Stage 2 with the reference horizon in the design or deriving the residual-to-reference mapping) - the DEFERRED.md paper-gated row. (e) RELAY CONVENTIONS: 'simple' relays the stored overall quintet bit-exact with `n = n_treated_obs`, `n_kind="obs"` (overlapping unit sets - the StackedDiD carve-out class) and `df = survey_df_final` (on replicate fits that value came from the `[overall]`-only joint stack - snapshotted, never re-derived); 'group' relays a SCALAR df broadcast (deliberate divergence from ImputationDiD's per-row `df_used`: `_stage2_group` passes one immutable `survey_df` to every row's `safe_inference`, so the scalar is provenance-exact by construction and the moved method stays verbatim); 'event_study' reproduces the M-092 container contract exactly - analytical fits thread the recomputed joint vcov + `vcov_index` + the finite-and->0 df scalar through the carrier, replicate fits thread `vcov=None`/`index=None` with the REPLAYED level-matched df, and the carrier's metadata is a copy-on-use of the KIT's fit-final metadata copy. REPLICATE-WEIGHT fits replay the extracted `_replay_replicate_inference` with a LEVEL-MATCHED stack (the ImputationDiD semantics: `aggregate(L)` reproduces `fit(aggregate=L)`; `fit(aggregate='all')` is not the equivalence target on degenerate designs; the OVERALL-row migration delta on such designs is documented and pinned). Recompute re-emits fit-time warnings with fit-tuned stacklevels - an accepted verbatim-move trade-off. +- **Note (post-fit aggregate() - rows M-022/M-119):** `fit(aggregate=, balance_e=)` is deprecated (3.9; removed 4.0; joint FutureWarning, warn-and-still-work; the `two_stage_did` wrapper forwards the shared sentinel so plain wrapper calls never fire the aggregate warning; since 3.9 the wrapper itself warns per M-071) in favor of post-fit `TwoStageDiDResults.aggregate(type, balance_e=)` - a PANEL-BACKED lazy recompute kit: each level is a fresh Stage-2 OLS + joint Gardner-GMM sandwich on a level-specific design, so no compact influence payload exists. (a) RETAINED BUFFERS (memory contract - the FIRST panel retention on TwoStageDiD results, a deliberate break from the CS/EDiD identifier-minimization guarantee, with a `store_kit` opt-out tracked in DEFERRED.md): a COLUMN-SUBSET COPY of the working frame - `unit`/`time`/`outcome`/`first_treat` + covariates + the cluster column (deduplicated: `cluster=` may legally name a core column) + `_never_treated`/`_rel_time`/`_y_tilde` - O(n_obs) on every results object and pickle; the Stage-1 FE model (`unit_fe`/`time_fe`/`grand_mean`/`delta_hat`/`kept_cov_mask`), the Omega masks, the full-domain `keep_mask`, the Wave-E.3-GATED `score_pad_mask`/`cluster_ids_full` values fit actually passed (None unless the always-treated pad was active), `survey_weights`, and the resolved survey design - on replicate designs that adds the O(n_obs x R) replicate matrix; plus value snapshots (`treatment_groups` copy, `ref_period`, `overall_att`, `n_treated_obs`, a `dataclasses.replace` copy of `survey_metadata`) and TWO df scalars (`survey_df_stage2`, the recompute seed; `survey_df_final`, what the stored overall inference received). (b) `balance_e` uses the BALANCED-WINDOW rule (`[-balance_e, max_h]` coverage against the full panel - the ImputationDiD rule, divergent from CS/EfficientDiD's anchor-horizon rule); zero qualifying cohorts warns and yields the reference-row-only dict with `vcov=None`. (c) BOOTSTRAP fits: 'simple' RELAYS the stored overall quintet verbatim (finite safe_inference t included) with a NaN df column, while the RECOMPUTE levels fail closed (per-level GMM scores are function-locals; replay is a TODO row) - the prior uniform rule superseded 2026-08-05 with the M-027 per-level convergence; a fit whose bootstrap FAILED (`bootstrap_results=None`, analytical inference retained) aggregates normally. (d) CONTAINER ADMISSION into `compute_honest_did`/`compute_pretrends_power` is DEFERRED, not by-design (both terminal TypeErrors state it): analytical surfaces DO carry the real joint Gardner-GMM covariance (M-092), but the pre-period coefficients are stage-1 residual MEANS - the reference horizon is dropped from the no-intercept Stage-2 design and the zero anchor row is appended mechanically - not contrasts against the advertised reference, while HonestDiD's Delta^RM/Delta^SD arithmetic hard-codes the `delta_0 = 0` normalization into its boundary/bridge constraints; admission awaits a normalization derivation (either re-estimating Stage 2 with the reference horizon in the design or deriving the residual-to-reference mapping) - the DEFERRED.md paper-gated row. (e) RELAY CONVENTIONS: 'simple' relays the stored overall quintet bit-exact with `n = n_treated_obs`, `n_kind="obs"` (overlapping unit sets - the StackedDiD carve-out class; the pre-filter |Omega_1| count, while the ATT's Stage-2 support excludes rows whose `y_tilde` is non-finite - on such degenerate fits `n` exceeds the averaged support) and `df = survey_df_final` (on replicate fits that value came from the `[overall]`-only joint stack - snapshotted, never re-derived); 'group' relays a SCALAR df broadcast (deliberate divergence from ImputationDiD's per-row `df_used`: `_stage2_group` passes one immutable `survey_df` to every row's `safe_inference`, so the scalar is provenance-exact by construction and the moved method stays verbatim); 'event_study' reproduces the M-092 container contract exactly - analytical fits thread the recomputed joint vcov + `vcov_index` + the finite-and->0 df scalar through the carrier, replicate fits thread `vcov=None`/`index=None` with the REPLAYED level-matched df, and the carrier's metadata is a copy-on-use of the KIT's fit-final metadata copy. REPLICATE-WEIGHT fits replay the extracted `_replay_replicate_inference` with a LEVEL-MATCHED stack (the ImputationDiD semantics: `aggregate(L)` reproduces `fit(aggregate=L)`; `fit(aggregate='all')` is not the equivalence target on degenerate designs; the OVERALL-row migration delta on such designs is documented and pinned). Recompute re-emits fit-time warnings with fit-tuned stacklevels - an accepted verbatim-move trade-off. --- @@ -6319,10 +6319,16 @@ estimator-focused: **Module:** `diff_diff/mmm.py` -**Scope:** Interop builders (not estimators), **explicit in / validated out**. Reconciling an experiment's estimate to a calibration input requires context diff-diff cannot see - the target MMM's row granularity (per-geo vs national), its time window, and the outcome's scale (additive levels vs log/rate/share) - so the module does NOT rescale a result's headline ATT. The caller supplies the already-scoped incremental outcome and its SE (the numbers read off `summary()`, aggregated to the population and window their MMM row represents); diff-diff assembles the target schema, enforces each consumer's guards, converts to the lognormal parameterization, pools, and emits snippets. Pure numpy/pandas; no MMM package imported; no result introspection, so the module is purely additive (does not touch any estimator). Deriving totals from a fitted result is deferred to the post-4.0 `results.aggregate()` layer, where the estimator - which owns its aggregation weights, balance, and survey masses - can compute its own total effect and SE. +**Scope:** Interop builders (not estimators), **explicit numbers in, or the pinned aggregation contract in - validated out**. Reconciling an experiment's estimate to a calibration input requires context diff-diff cannot always see - the target MMM's row granularity (per-geo vs national), its time window, and the outcome's scale (additive levels vs log/rate/share). The default route stays fully explicit: the caller supplies the already-scoped incremental outcome and its SE (the numbers read off `summary()`, aggregated to the population and window their MMM row represents). The 3.9 post-fit `results.aggregate()` layer - the route this section previously deferred to - now exists, and both exporters consume its pinned `AggregationResult` container via `aggregation_result=` + `scale=` (the exporters never call `aggregate()` themselves and reject raw results objects and `EventStudyResults`; the module remains purely additive and touches no estimator). diff-diff assembles the target schema, enforces each consumer's guards, converts to the lognormal parameterization, pools, and emits snippets. Pure numpy/pandas; no MMM package imported. **Key implementation requirements:** +*Container mode (`aggregation_result=` + `scale=`, both exporters):* +- **Note:** Scale contract and row order. Derivation is `effect_i = att_i * scale_i`, `se_out_i = se_i * scale_i` per container row, with rows consumed in `to_dataframe()` order (sorted by label when sortable, producer order otherwise - the order `summary()` prints); every per-row sequence kwarg (`scale`, `spend`, `x`, `delta_x`, `dims`) aligns to that order, and the helper reads all per-row values from one `to_dataframe()` call. `scale` is always required with a container: a number (finite, positive; scalar or per-row) or the literal `"auto"`. Accepted levels are `simple` (one row; a multi-row simple container is out-of-contract and rejected) and `group` (one experiment per cohort); every row's `target` must be `"att"` (ContinuousDiD's `acrt`, HAD's WAS estimands, and dCDH's estimand relays reject the container WHOLE - no silent row filtering), and every row needs finite `att` and finite positive `se`. +- **Note:** `scale="auto"` allowlist and routing. Auto-derivation (`scale_i = n_i`) is honored ONLY for container `estimator` provenance in `{ImputationDiD, TwoStageDiD}`, the audited producers whose `n` is the treated observations (unit-periods) the ATT averages over at both supported levels. Routing keys on the `estimator` field, never on `n_kind` alone - CallawaySantAnna repeated-cross-section fits report `n_kind="obs"` with `n` = treated+control observations - and `n_kind == "obs"` serves only as a schema-drift sanity guard on the allowlist path. CallawaySantAnna (`simple`: treated+control units/obs; `group`: contributing (g, t) cells) and EfficientDiD (units/cells) are refused with hints naming their `n` semantics; StackedDiD is refused because its `n` is a deduplicated distinct-treated-UNIT count (no per-observation mass) and its `weighting="population"`/`"sample_share"` estimands make raw treated exposure the wrong multiplier. +- **Note:** `scale="auto"` is an explicit caller acknowledgement of three assumptions the container cannot verify: the outcome is in additive levels (no outcome-scale provenance exists on the container); the fit is unweighted (survey-weighted fits pair a weighted `att` with a raw-count `n` and carry no survey marker - pass a numeric `scale` there); and every treated observation's effect is identified. On the identification point, ImputationDiD averages finite tau-hat only while `n` stays the raw treated-obs count at BOTH levels, and TwoStageDiD's `simple` ATT support excludes rows whose first-stage residual is non-finite while `n` reports the pre-filter count (its `group` `n` IS post-filter) - both estimators warn at fit time on the degenerate branch, and `"auto"` on such a fit overcounts (behaviorally pinned in `tests/test_mmm.py::TestAggregationWorkflow`). +- **Note:** `group` containers carry correlated cohorts, both exporters: all cohort rows come from ONE fit sharing controls/windows, and the container stores only per-row SEs (joint-covariance off-diagonals are discarded upstream). Meridian pooling treats rows as independent (see the pooling Note below) and PyMC-Marketing scores each lift row independently with its marginal `sigma`; the omitted cross-cohort covariance can misstate the joint/pooled uncertainty - anti-conservatively when the net weighted covariance is positive (shared controls typically induce positive correlation) - and its direction cannot be determined from marginal SEs alone. `se_widening > 1` is a conservative heuristic, not an exact correction (docstring-noted on both exporters). + *Lift-test frame (`to_pymc_marketing_lift_test`):* - Assembles one row per experiment with the exact schema `[channel, *dims, x, delta_x, delta_y, sigma]` from caller-supplied values. `delta_y`/`sigma` are the scoped incremental outcome and its SE; all four numeric columns must describe the SAME calibration observation (same channel, population, period span, additive-level outcome) - stated in the docstring, not machine-checkable, since spend and scope are user input. - Sign/zero policy (`on_wrong_sign={raise,drop,keep}`, default raise): PyMC-Marketing rejects `sign(delta_y) != sign(delta_x)` with `NonMonotonicError`; `delta_y == 0` is a distinct class (degenerate for its strictly-positive Gamma lift likelihood, which its own monotonicity check does not catch). Both share one disposition; `drop` raises rather than emit an empty frame; `keep` warns the frame is not valid PyMC-Marketing input. @@ -6333,7 +6339,7 @@ estimator-focused: - **Estimand (`parameter={roi_m,mroi_m}`, default roi_m):** `roi_m` is the return on the channel's full spend (a zero-spend/full-holdout estimand; `spend` = total channel spend); `mroi_m` is the marginal return of a spend change (`spend` = the spend change). Under saturation these differ, so the caller declares which their `incremental_outcome` measures; the returned prior and `.to_code()` target that parameter with its own Meridian default for non-experiment channels (roi_m `LogNormal(0.2, 0.9)`, mroi_m `LogNormal(0.0, 0.5)`, both verified at 1.7.0). Sign, aggregation, and population are the caller's responsibility - a go-dark contract's numerator is `Y_exposed - Y_zero`, which the caller supplies with the correct sign. - Lognormal conversion matches Meridian's `lognormal_dist_from_mean_std` exactly: `sigma = sqrt(log1p((s/m)^2))`, `mu = ln(m) - log1p((s/m)^2)/2` (`log1p` keeps a ~1e-8 relative SE from rounding `sigma` to 0). - A non-positive pooled ROI mean raises (lognormal positivity), pointing to pooling, wider priors, or Meridian's contribution/coefficient parameterizations. `spend`, `incremental_outcome_se`, and `se_widening` must be finite and positive; `incremental_outcome` finite. -- **Note:** Pooled `roi_sd = sqrt(sum((w_i * sd_i)^2))` treats experiments as independent (no covariance term). Experiments sharing control units or overlapping windows are positively correlated and the pooled sd is then anti-conservative; the docstring instructs users to widen via `se_widening`. +- **Note:** Pooled `roi_sd = sqrt(sum((w_i * sd_i)^2))` treats experiments as independent (no covariance term). Ignoring covariance can misstate the pooled sd - it is anti-conservative when the net weighted covariance is positive, which experiments sharing control units or overlapping windows typically induce - and the direction cannot be determined from marginal SEs alone; the docstring instructs users to widen via `se_widening` (a conservative heuristic, not an exact correction). - **Channel scope in `.to_code()`**: roi_m/mroi_m have batch shape `n_media_channels` and a scalar distribution broadcasts to EVERY channel, so the snippet helper requires explicit scope - `channel=`+`media_channels=` emits a vector prior in model channel order (non-experiment channels keep the parameter's Meridian default), `single_channel=True` emits a scalar snippet marked single-channel-only, and neither raises. It also requires the prior's TIME scope (`roi_calibration_period=` or `full_model_window=True`), because Meridian's default applies an experiment-window prior over all model times; and it sets `media_prior_type="roi"`/`"mroi"` on `ModelSpec` (Meridian ignores a supplied `roi_m`/`mroi_m` unless the matching prior type is selected). Snippets use the TensorFlow substrate; a comment points JAX users at `tensorflow_probability.substrates.jax`. *Outputs:* All scaled and pooled values (`roi`, `roi_sd`, pooled moments, lognormal `mu`/`sigma`) are validated finite-and-positive after arithmetic; overflow/underflow raises rather than emitting non-finite or zero-uncertainty calibration data. diff --git a/tests/test_doc_snippets.py b/tests/test_doc_snippets.py index e3fed6abb..3f200fd3e 100644 --- a/tests/test_doc_snippets.py +++ b/tests/test_doc_snippets.py @@ -39,6 +39,7 @@ "api/pretrends.rst", "api/power.rst", "api/changes_in_changes.rst", + "api/mmm.rst", "python_comparison.rst", "r_comparison.rst", ] diff --git a/tests/test_mmm.py b/tests/test_mmm.py index aaa8bb7dc..4c777aa49 100644 --- a/tests/test_mmm.py +++ b/tests/test_mmm.py @@ -1,21 +1,29 @@ """Tests for diff_diff.mmm - MMM calibration input assembly (interop). -The exporters are explicit-in / validated-out: the caller supplies the already-scoped -incremental outcome and its SE, and the module assembles the target schema, enforces -each consumer's guards, converts to the lognormal parameterization, and pools. These -tests cover the schema/guards/math directly, plus a realistic workflow test that reads -att/se off fitted estimators and feeds them in (the module never introspects results). +The exporters take either explicit already-scoped numbers or the pinned +``AggregationResult`` container (``aggregation_result=`` + ``scale=``, with +``scale="auto"`` honored only for the audited ImputationDiD/TwoStageDiD +producers); the module assembles the target schema, enforces each consumer's +guards, converts to the lognormal parameterization, and pools. These tests cover +the schema/guards/math directly, the container-mode extraction/routing matrix, +end-to-end workflows on fitted estimators, and a contract pin of exactly the +container surface the module consumes. """ import math import warnings +import numpy as np +import pandas as pd import pytest from scipy import stats from diff_diff import ( + AggregationResult, CallawaySantAnna, DifferenceInDifferences, + ImputationDiD, + TwoStageDiD, to_meridian_roi_prior, to_pymc_marketing_lift_test, ) @@ -443,12 +451,13 @@ def test_did_workflow(self): assert prior.roi_mean == pytest.approx(total / 100_000.0) def test_callaway_santanna_workflow(self): - # CS gives a headline ATT; the caller is responsible for turning it into a - # total incremental outcome using the estimator's own aggregation (not a - # naive att x count, which need not reproduce CS's cohort weights or - # variance - that estimator-owned aggregation is the post-4.0 follow-up). - # Here we simply confirm the fitted numbers flow through the exporter once - # the caller has supplied a total and its SE. + # CS gives a headline ATT; the container route now exists + # (aggregation_result= + a NUMERIC scale=, see TestAggregationWorkflow), + # but CS's own estimator-owned aggregation producing a total incremental + # outcome remains undelivered (DEFERRED.md remainder row) - a naive + # att x count need not reproduce CS's cohort weights or variance. Here we + # simply confirm the fitted numbers flow through the exporter once the + # caller has supplied a total and its SE. data = generate_staggered_data(n_units=80, n_periods=8, seed=7) cs = CallawaySantAnna().fit( data, outcome="outcome", unit="unit", time="period", first_treat="first_treat" @@ -463,6 +472,652 @@ def test_callaway_santanna_workflow(self): assert prior.roi_mean == pytest.approx(150_000.0 / 200_000.0) +def _make_agg( + level="simple", + estimator="ImputationDiD", + n_kind="obs", + labels=("overall",), + atts=(2.0,), + ses=(0.5,), + ns=(30.0,), + targets=None, +): + """Hand-built AggregationResult mirroring the producers' field conventions.""" + k = len(labels) + return AggregationResult( + level=level, + label=np.array(labels, dtype=object), + target=np.array(targets if targets is not None else ["att"] * k, dtype=object), + att=np.array(atts, dtype=float), + se=np.array(ses, dtype=float), + t_stat=np.full(k, 1.0), + p_value=np.full(k, 0.5), + conf_int_lower=np.full(k, 0.0), + conf_int_upper=np.full(k, 1.0), + n=np.array(ns, dtype=float), + df=np.full(k, np.nan), + n_kind=n_kind, + estimator=estimator, + ) + + +class TestAggregationExtraction: + """Container-mode routing/rejection matrix, on hand-built containers.""" + + def test_rejects_results_object_with_remedy(self): + result = DifferenceInDifferences().fit( + generate_did_data(n_units=20, n_periods=2, treatment_period=1, seed=3), + outcome="outcome", + treatment="treated", + post="post", + ) + with pytest.raises(TypeError, match="have no container-mode route"): + to_pymc_marketing_lift_test(channel="tv", x=1.0, delta_x=1.0, aggregation_result=result) + + def test_event_study_container_gets_results_object_message(self): + from diff_diff import EventStudyResults + + es = EventStudyResults( + event_time=np.array([-1, 0, 1]), + att=np.array([0.0, 1.0, 1.5]), + se=np.array([np.nan, 0.2, 0.3]), + t_stat=np.array([np.nan, 5.0, 5.0]), + p_value=np.array([np.nan, 0.01, 0.01]), + conf_int_lower=np.array([np.nan, 0.6, 0.9]), + conf_int_upper=np.array([np.nan, 1.4, 2.1]), + is_reference=np.array([True, False, False]), + n=np.array([np.nan, 10.0, 10.0]), + ) + # EventStudyResults subclasses BaseResults, so it must get the 1a + # results-object message, never the generic 1b one. + with pytest.raises(TypeError, match="EventStudyResults and estimators"): + to_pymc_marketing_lift_test(channel="tv", x=1.0, delta_x=1.0, aggregation_result=es) + + def test_rejects_non_container_type(self): + from diff_diff.results_base import Diagnostic + + class _SomeDiagnostic(Diagnostic): + """Stand-in for the Diagnostic-marker roster (not BaseResults).""" + + for bad in (object(), _SomeDiagnostic()): + with pytest.raises(TypeError, match="Only AggregationResult is supported"): + to_pymc_marketing_lift_test( + channel="tv", x=1.0, delta_x=1.0, aggregation_result=bad + ) + + def test_results_object_without_scale_still_type_error(self): + # The type check runs before the scale-required check. + result = DifferenceInDifferences().fit( + generate_did_data(n_units=20, n_periods=2, treatment_period=1, seed=3), + outcome="outcome", + treatment="treated", + post="post", + ) + with pytest.raises(TypeError, match="must be an AggregationResult"): + to_meridian_roi_prior(aggregation_result=result, spend=10.0) + + def test_rejects_unsupported_level(self): + agg = _make_agg(level="calendar") + with pytest.raises(ValueError, match="level must be 'simple' or 'group'"): + to_pymc_marketing_lift_test( + channel="tv", x=1.0, delta_x=1.0, aggregation_result=agg, scale=1.0 + ) + + def test_rejects_empty_container(self): + agg = _make_agg(level="group", labels=(), atts=(), ses=(), ns=()) + with pytest.raises(ValueError, match="has no rows"): + to_pymc_marketing_lift_test( + channel="tv", x=1.0, delta_x=1.0, aggregation_result=agg, scale=1.0 + ) + + def test_rejects_multi_row_simple_container(self): + agg = _make_agg( + level="simple", + labels=("overall", "overall2"), + atts=(1.0, 2.0), + ses=(0.1, 0.1), + ns=(5.0, 5.0), + ) + with pytest.raises(ValueError, match="out of contract"): + to_pymc_marketing_lift_test( + channel="tv", x=1.0, delta_x=1.0, aggregation_result=agg, scale=1.0 + ) + + def test_rejects_non_att_target_whole_container(self): + # ContinuousDiD's simple container carries att + acrt rows; the acrt row + # is a dose derivative, so the WHOLE container is rejected (no filtering). + agg = _make_agg( + level="simple", + estimator="ContinuousDiD", + n_kind="units", + labels=("overall", "overall"), + atts=(1.0, 0.2), + ses=(0.1, 0.05), + ns=(40.0, 40.0), + targets=("att", "acrt"), + ) + with pytest.raises(ValueError, match="rejected whole"): + to_pymc_marketing_lift_test( + channel="tv", x=1.0, delta_x=1.0, aggregation_result=agg, scale=1.0 + ) + + def test_rejects_unusable_inference(self): + for atts, ses in ( + ((math.nan,), (0.5,)), + ((math.inf,), (0.5,)), + ((1.0,), (0.0,)), + ((1.0,), (-1.0,)), + ((1.0,), (math.nan,)), + ((1.0,), (math.inf,)), + ): + agg = _make_agg(atts=atts, ses=ses) + with pytest.raises(ValueError, match="no usable point estimate/SE"): + to_pymc_marketing_lift_test( + channel="tv", + x=1.0, + delta_x=1.0, + aggregation_result=agg, + scale=1.0, + ) + + def test_auto_scale_allowlisted_estimators(self): + for estimator in ("ImputationDiD", "TwoStageDiD"): + agg = _make_agg(estimator=estimator, atts=(2.0,), ses=(0.5,), ns=(30.0,)) + df = to_pymc_marketing_lift_test( + channel="tv", + x=100.0, + delta_x=50.0, + aggregation_result=agg, + scale="auto", + ) + assert df.loc[0, "delta_y"] == 2.0 * 30.0 + assert df.loc[0, "sigma"] == 0.5 * 30.0 + + def test_auto_scale_rejected_off_allowlist(self): + cases = [ + ("CallawaySantAnna", "treated\\+control"), + ("EfficientDiD", "disjoint treated\\+control"), + ("StackedDiD", "deduplicated distinct-treated-unit"), + ("SomethingNew", "not audited"), + (None, "not audited"), + ] + for estimator, fragment in cases: + agg = _make_agg(estimator=estimator) + with pytest.raises(ValueError, match=fragment): + to_pymc_marketing_lift_test( + channel="tv", + x=1.0, + delta_x=1.0, + aggregation_result=agg, + scale="auto", + ) + + def test_auto_scale_n_kind_drift_guard(self): + agg = _make_agg(estimator="ImputationDiD", n_kind="units") + with pytest.raises(ValueError, match="schema has drifted"): + to_pymc_marketing_lift_test( + channel="tv", x=1.0, delta_x=1.0, aggregation_result=agg, scale="auto" + ) + + def test_auto_scale_rejects_bad_n(self): + for n in (math.nan, 0.0, -3.0, math.inf, -math.inf): + agg = _make_agg(ns=(n,)) + with pytest.raises(ValueError, match="cannot auto-derive scale"): + to_pymc_marketing_lift_test( + channel="tv", + x=1.0, + delta_x=1.0, + aggregation_result=agg, + scale="auto", + ) + + def test_missing_scale_names_both_routes(self): + agg = _make_agg() + with pytest.raises(ValueError, match="scale is required with aggregation_result"): + to_pymc_marketing_lift_test(channel="tv", x=1.0, delta_x=1.0, aggregation_result=agg) + + def test_non_auto_string_scale_rejected(self): + agg = _make_agg() + with pytest.raises(ValueError, match="or the string 'auto'"): + to_pymc_marketing_lift_test( + channel="tv", x=1.0, delta_x=1.0, aggregation_result=agg, scale="AUTO" + ) + + def test_explicit_scale_scalar_and_per_row(self): + agg = _make_agg( + level="group", + estimator="CallawaySantAnna", + n_kind="cells", + labels=(2001, 2002), + atts=(1.0, 2.0), + ses=(0.1, 0.2), + ns=(4.0, 5.0), + ) + df = to_pymc_marketing_lift_test( + channel="tv", x=1.0, delta_x=1.0, aggregation_result=agg, scale=10.0 + ) + assert df["delta_y"].tolist() == [10.0, 20.0] + df2 = to_pymc_marketing_lift_test( + channel="tv", x=1.0, delta_x=1.0, aggregation_result=agg, scale=[10.0, 100.0] + ) + assert df2["delta_y"].tolist() == [10.0, 200.0] + assert df2["sigma"].tolist() == [pytest.approx(1.0), pytest.approx(20.0)] + + def test_scale_validation_reuses_existing_messages(self): + agg = _make_agg( + level="group", + labels=(1, 2), + atts=(1.0, 2.0), + ses=(0.1, 0.2), + ns=(4.0, 5.0), + ) + with pytest.raises(ValueError, match="scale has length 3 but 2"): + to_pymc_marketing_lift_test( + channel="tv", + x=1.0, + delta_x=1.0, + aggregation_result=agg, + scale=[1.0, 2.0, 3.0], + ) + for bad in (0.0, -1.0, math.nan, math.inf): + with pytest.raises(ValueError, match="scale must be finite and > 0"): + to_pymc_marketing_lift_test( + channel="tv", + x=1.0, + delta_x=1.0, + aggregation_result=agg, + scale=bad, + ) + + def test_row_order_is_to_dataframe_order(self): + # Producer order 2003, 2001; to_dataframe() sorts sortable labels, so the + # 2001 row comes first and per-row scale aligns to the SORTED order. + agg = _make_agg( + level="group", + labels=(2003, 2001), + atts=(3.0, 1.0), + ses=(0.3, 0.1), + ns=(7.0, 5.0), + ) + df = to_pymc_marketing_lift_test( + channel="tv", + x=1.0, + delta_x=1.0, + aggregation_result=agg, + scale=[10.0, 100.0], + ) + # Sorted order: label 2001 (att=1.0) gets scale 10, label 2003 gets 100. + assert df["delta_y"].tolist() == [10.0, 300.0] + + +class TestLiftTestFromAggregation: + def test_simple_container_one_row(self): + agg = _make_agg(atts=(2.5,), ses=(0.4,), ns=(12.0,)) + df = to_pymc_marketing_lift_test( + channel="tv", + x=500.0, + delta_x=250.0, + aggregation_result=agg, + scale="auto", + dims={"geo": "US"}, + ) + assert list(df.columns) == ["channel", "geo", "x", "delta_x", "delta_y", "sigma"] + assert df.loc[0, "delta_y"] == 2.5 * 12.0 + assert df.loc[0, "sigma"] == pytest.approx(0.4 * 12.0) + + def test_group_container_broadcast_interplay(self): + agg = _make_agg( + level="group", + labels=(1, 2, 3), + atts=(1.0, 2.0, 3.0), + ses=(0.1, 0.2, 0.3), + ns=(10.0, 10.0, 10.0), + ) + df = to_pymc_marketing_lift_test( + channel="tv", + x=[100.0, 200.0, 300.0], + delta_x=50.0, + aggregation_result=agg, + scale="auto", + ) + assert len(df) == 3 + assert df["x"].tolist() == [100.0, 200.0, 300.0] + assert df["delta_x"].tolist() == [50.0, 50.0, 50.0] + with pytest.raises(ValueError, match="x has length 2 but 3"): + to_pymc_marketing_lift_test( + channel="tv", + x=[100.0, 200.0], + delta_x=50.0, + aggregation_result=agg, + scale="auto", + ) + + def test_mode_exclusivity(self): + agg = _make_agg() + with pytest.raises(ValueError, match="not both"): + to_pymc_marketing_lift_test( + channel="tv", + x=1.0, + delta_x=1.0, + delta_y=5.0, + sigma=1.0, + aggregation_result=agg, + scale=1.0, + ) + # Container plus exactly ONE lone member is also "not both" (or-joined). + with pytest.raises(ValueError, match="not both"): + to_pymc_marketing_lift_test( + channel="tv", + x=1.0, + delta_x=1.0, + delta_y=5.0, + aggregation_result=agg, + scale=1.0, + ) + with pytest.raises(ValueError, match="delta_y and sigma are required"): + to_pymc_marketing_lift_test(channel="tv", x=1.0, delta_x=1.0) + with pytest.raises(ValueError, match="sigma is required"): + to_pymc_marketing_lift_test(channel="tv", x=1.0, delta_x=1.0, delta_y=5.0) + with pytest.raises(ValueError, match="delta_y is required"): + to_pymc_marketing_lift_test(channel="tv", x=1.0, delta_x=1.0, sigma=1.0) + with pytest.raises(ValueError, match="scale only applies with"): + to_pymc_marketing_lift_test( + channel="tv", x=1.0, delta_x=1.0, delta_y=5.0, sigma=1.0, scale=2.0 + ) + + def test_sign_policy_applies_to_derived_rows(self): + agg = _make_agg(atts=(-2.0,), ses=(0.5,), ns=(10.0,)) + with pytest.raises(ValueError, match="NonMonotonicError"): + to_pymc_marketing_lift_test( + channel="tv", + x=100.0, + delta_x=50.0, + aggregation_result=agg, + scale="auto", + ) + with pytest.warns(UserWarning, match="Keeping invalid"): + df = to_pymc_marketing_lift_test( + channel="tv", + x=100.0, + delta_x=50.0, + aggregation_result=agg, + scale="auto", + on_wrong_sign="keep", + ) + assert df.loc[0, "delta_y"] == -20.0 + + +class TestMeridianFromAggregation: + def test_simple_container_prior_math(self): + agg = _make_agg(atts=(2.0,), ses=(0.5,), ns=(30.0,)) + prior = to_meridian_roi_prior(aggregation_result=agg, scale="auto", spend=1_000.0) + total, total_se = 2.0 * 30.0, 0.5 * 30.0 + assert prior.roi_mean == pytest.approx(total / 1_000.0) + assert prior.roi_sd == pytest.approx(total_se / 1_000.0) + # Lognormal closed-form roundtrip (Google's parameterization). + m, s = prior.roi_mean, prior.roi_sd + assert prior.sigma == pytest.approx(math.sqrt(math.log(1 + (s / m) ** 2))) + assert prior.mu == pytest.approx(math.log(m) - 0.5 * math.log(1 + (s / m) ** 2)) + + def test_group_container_per_cohort_spend_pooling(self): + agg = _make_agg( + level="group", + estimator="TwoStageDiD", + labels=(2001, 2002), + atts=(1.0, 2.0), + ses=(0.1, 0.2), + ns=(10.0, 20.0), + ) + spends = [500.0, 1_500.0] + prior = to_meridian_roi_prior(aggregation_result=agg, scale="auto", spend=spends) + rois = [1.0 * 10.0 / 500.0, 2.0 * 20.0 / 1_500.0] + sds = [0.1 * 10.0 / 500.0, 0.2 * 20.0 / 1_500.0] + weights = [500.0 / 2_000.0, 1_500.0 / 2_000.0] + assert prior.roi_mean == pytest.approx(sum(w * r for w, r in zip(weights, rois))) + assert prior.roi_sd == pytest.approx(math.hypot(*(w * s for w, s in zip(weights, sds)))) + assert [e.spend for e in prior.per_experiment] == spends + + def test_mode_exclusivity(self): + agg = _make_agg() + with pytest.raises(ValueError, match="not both"): + to_meridian_roi_prior( + incremental_outcome=10.0, + incremental_outcome_se=1.0, + aggregation_result=agg, + scale=1.0, + spend=100.0, + ) + with pytest.raises(ValueError, match="not both"): + to_meridian_roi_prior( + incremental_outcome_se=1.0, + aggregation_result=agg, + scale=1.0, + spend=100.0, + ) + with pytest.raises(ValueError, match="incremental_outcome and incremental_outcome_se are"): + to_meridian_roi_prior(spend=100.0) + with pytest.raises(ValueError, match="incremental_outcome_se is required"): + to_meridian_roi_prior(incremental_outcome=10.0, spend=100.0) + with pytest.raises(ValueError, match="incremental_outcome is required"): + to_meridian_roi_prior(incremental_outcome_se=1.0, spend=100.0) + with pytest.raises(ValueError, match="scale only applies with"): + to_meridian_roi_prior( + incremental_outcome=10.0, + incremental_outcome_se=1.0, + scale=2.0, + spend=100.0, + ) + + def test_negative_pooled_att_hits_positivity_guard(self): + agg = _make_agg(atts=(-2.0,), ses=(0.5,), ns=(30.0,)) + with pytest.raises(ValueError, match="positive support"): + to_meridian_roi_prior(aggregation_result=agg, scale="auto", spend=1_000.0) + + +class TestAggregationWorkflow: + """End-to-end: fit -> aggregate -> export, with container-independent oracles + (treated unit-period counts computed from the input frame, never agg.n).""" + + def test_imputation_auto_scale(self): + data = generate_staggered_data(n_units=60, n_periods=6, seed=11) + res = ImputationDiD().fit( + data, outcome="outcome", unit="unit", time="period", first_treat="first_treat" + ) + agg = res.aggregate("simple") + assert agg.estimator == "ImputationDiD" + # Frame-derived treated-obs count (never-treated coded first_treat == 0). + treated_obs = int( + ((data["first_treat"] > 0) & (data["period"] >= data["first_treat"])).sum() + ) + spend = 50_000.0 + prior = to_meridian_roi_prior(aggregation_result=agg, scale="auto", spend=spend) + manual = to_meridian_roi_prior( + incremental_outcome=float(agg.att[0]) * treated_obs, + incremental_outcome_se=float(agg.se[0]) * treated_obs, + spend=spend, + ) + assert prior.roi_mean == pytest.approx(manual.roi_mean) + assert prior.roi_sd == pytest.approx(manual.roi_sd) + + def test_two_stage_group_auto_scale(self): + data = generate_staggered_data(n_units=60, n_periods=6, seed=11) + res = TwoStageDiD().fit( + data, outcome="outcome", unit="unit", time="period", first_treat="first_treat" + ) + agg = res.aggregate("group") + frame = agg.to_dataframe() + df = to_pymc_marketing_lift_test( + channel="tv", + x=100.0, + delta_x=50.0, + aggregation_result=agg, + scale="auto", + on_wrong_sign="keep", + ) + assert len(df) == len(frame) + for i, (_, row) in enumerate(frame.iterrows()): + cohort = row["label"] + cohort_obs = int( + ((data["first_treat"] == cohort) & (data["period"] >= data["first_treat"])).sum() + ) + assert df.loc[i, "delta_y"] == pytest.approx(row["att"] * cohort_obs) + + def test_callaway_explicit_scale_path(self): + data = generate_staggered_data(n_units=60, n_periods=6, seed=11) + cs = CallawaySantAnna().fit( + data, outcome="outcome", unit="unit", time="period", first_treat="first_treat" + ) + agg = cs.aggregate("simple") + with pytest.raises(ValueError, match="treated\\+control"): + to_meridian_roi_prior(aggregation_result=agg, scale="auto", spend=1_000.0) + scale = 40.0 # caller-derived treated unit-periods for THEIR scoping + prior = to_meridian_roi_prior(aggregation_result=agg, scale=scale, spend=1_000.0) + assert prior.roi_mean == pytest.approx(float(agg.att[0]) * scale / 1_000.0) + + def test_imputation_nonfinite_tau_divergence(self): + # tests/test_imputation.py recipe: drop never-treated obs at t=5 so that + # period's time FE is unidentified -> non-finite tau-hat there, while the + # container's n stays the raw |Omega_1| count. This is the documented + # scale="auto" overcount scenario, pinned behaviorally. + rng = np.random.default_rng(42) + rows = [] + for i in range(40): + ft = 2 if i < 20 else 99 + for t in range(6): + if ft == 99 and t == 5: + continue + y = rng.standard_normal() + i * 0.1 + t * 0.05 + if t >= ft: + y += 1.0 + rows.append({"unit": i, "time": t, "outcome": y, "first_treat": ft}) + data = pd.DataFrame(rows) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + res = ImputationDiD().fit( + data, outcome="outcome", unit="unit", time="time", first_treat="first_treat" + ) + agg = res.aggregate("simple") + full_omega_1 = 20 * 4 # 20 treated units, periods 2..5 + identified_obs = 20 * 3 # (cohort, t=5) cell is unidentified + assert float(agg.n[0]) == full_omega_1 + assert float(agg.n[0]) > identified_obs # auto-scale would overcount + + def test_two_stage_nonfinite_ytilde_divergence(self): + # Analogous construction: with no untreated observations at t=5, stage 1 + # (fit on untreated obs only) cannot identify that period's time FE, so + # treated rows there get non-finite y_tilde and drop from the ATT's + # support - while the simple container's n stays the pre-filter |Omega_1|. + rng = np.random.default_rng(42) + rows = [] + for i in range(40): + ft = 2 if i < 20 else 99 + for t in range(6): + if ft == 99 and t == 5: + continue + y = rng.standard_normal() + i * 0.1 + t * 0.05 + if t >= ft: + y += 1.0 + rows.append({"unit": i, "time": t, "outcome": y, "first_treat": ft}) + data = pd.DataFrame(rows) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + res = TwoStageDiD().fit( + data, outcome="outcome", unit="unit", time="time", first_treat="first_treat" + ) + agg = res.aggregate("simple") + full_omega_1 = 20 * 4 + post_filter_support = 20 * 3 # treated rows at t=5 lose their y_tilde + assert float(agg.n[0]) == full_omega_1 + assert float(agg.n[0]) > post_filter_support + + def test_overflow_surfaces_downstream_messages(self): + # Only the effect product overflows (huge att, tiny se), so the message + # names the effect, per each exporter's own validation order. + agg = _make_agg(atts=(1e300,), ses=(1e-300,), ns=(30.0,)) + with pytest.raises(ValueError, match="delta_y must be finite"): + to_pymc_marketing_lift_test( + channel="tv", + x=1.0, + delta_x=1.0, + aggregation_result=agg, + scale=1e10, + ) + with pytest.raises(ValueError, match="incremental_outcome must be finite"): + to_meridian_roi_prior(aggregation_result=agg, scale=1e10, spend=100.0) + + +class TestAggregationContractPin: + """Pin exactly the AggregationResult surface mmm consumes, so container schema + drift fails loudly here (independent of tests/test_aggregate_contract.py, + which compares producer frames AGAINST the constant, not the constant itself). + Containers at any alpha behave identically: alpha/CI/df/p are not consumed.""" + + _CONSUMED_COLUMNS = {"label", "target", "att", "se", "n"} + _NOT_CONSUMED_COLUMNS = { + "level", # consumed as the ATTRIBUTE, not the frame column + "t_stat", + "p_value", + "conf_int_lower", + "conf_int_upper", + "weight", + "df", + } + + def test_aggregation_schema_literal_pin(self): + from diff_diff.aggregation import AGGREGATION_SCHEMA + + # mmm consumes to_dataframe() rows; a schema change must be consciously + # synced with _extract_aggregation_rows. + assert AGGREGATION_SCHEMA == ( + "level", + "label", + "target", + "att", + "se", + "t_stat", + "p_value", + "conf_int_lower", + "conf_int_upper", + "n", + "weight", + "df", + ) + + def test_consumed_surface_is_closed(self): + from diff_diff.aggregation import AGGREGATION_SCHEMA + + # The consumed + deliberately-not-consumed sets jointly cover the schema, + # so this pin is a closed definition of mmm's read surface. + assert self._CONSUMED_COLUMNS | self._NOT_CONSUMED_COLUMNS == set(AGGREGATION_SCHEMA) + agg = _make_agg() + for attr in ("level", "n_kind", "estimator"): + assert hasattr(agg, attr) + assert callable(agg.to_dataframe) + frame = agg.to_dataframe() + for col in self._CONSUMED_COLUMNS: + assert col in frame.columns + + def test_public_annotations_runtime_resolvable(self): + # AggregationResult is imported at module runtime (not TYPE_CHECKING), + # so annotation consumers can resolve the exporters' signatures. + import typing + + for fn in (to_pymc_marketing_lift_test, to_meridian_roi_prior): + hints = typing.get_type_hints(fn) + assert "aggregation_result" in hints and "scale" in hints + + def test_allowlist_strings_match_provenance_derivation(self): + from diff_diff.imputation_results import ImputationDiDResults + from diff_diff.mmm import _SCALE_AUTO_ESTIMATORS + from diff_diff.two_stage_results import TwoStageDiDResults + + derived = { + cls.__name__.replace("Results", "") + for cls in (ImputationDiDResults, TwoStageDiDResults) + } + assert derived == set(_SCALE_AUTO_ESTIMATORS) + + def test_public_exports(): import diff_diff diff --git a/tests/test_naming_guard.py b/tests/test_naming_guard.py index 08b991ac8..2b50716ce 100644 --- a/tests/test_naming_guard.py +++ b/tests/test_naming_guard.py @@ -969,6 +969,11 @@ def _token_family_code_refs(tok): ("estimator", "diff_diff/aggregation.py"): ( "AggregationResult.estimator - independent field holding a CLASS NAME" ), + ("estimator", "diff_diff/mmm.py"): ( + "reads AggregationResult.estimator (the container's own provenance " + "field, a CLASS NAME) to route container-mode scale='auto' - not a " + "read of CiC's renamed results field" + ), ("estimator", "diff_diff/business_report.py"): ( 'report-schema "estimator" keys holding class names / native tags' ),