Skip to content

feat: MMM exporters container mode (aggregation_result= + scale=) on the 3.9 aggregation surface - #765

Merged
igerber merged 2 commits into
mainfrom
feature/mmm-aggregation-container-mode
Aug 15, 2026
Merged

igerber merged 2 commits into
mainfrom
feature/mmm-aggregation-container-mode

Conversation

@igerber

@igerber igerber commented Aug 15, 2026

Copy link
Copy Markdown
Owner

Summary

  • Both MMM exporters (to_pymc_marketing_lift_test, to_meridian_roi_prior) gain a container mode: aggregation_result= accepts the pinned AggregationResult returned by post-fit results.aggregate('simple') (one experiment row) or 'group' (one row per cohort), together with the new required scale=, deriving effect = att * scale and se = se * scale per row in to_dataframe() order. The explicit-numbers route is unchanged; the two routes are mutually exclusive.
  • scale="auto" reads the container's per-row n and is honored ONLY for ImputationDiD/TwoStageDiD provenance — the audited producers whose n matches the ATT's averaged support on unweighted, fully identified fits. Passing "auto" acknowledges the three assumptions the container cannot verify (additive-level outcome, unweighted fit, fully identified effects); everything else fails closed with the remedy inline (raw results objects and EventStudyResults, unsupported levels, non-"att" targets rejected whole, unusable inference, non-allowlisted "auto" with per-estimator hints keyed on the container's estimator field — never on n_kind, which serves only as a drift guard).
  • Docs sweep: REGISTRY MMM section reworded to the new posture with container-mode **Note:** bullets; docs/api/mmm.rst Overview and examples rewritten (now self-contained and executed by tests/test_doc_snippets.py); both LLM guides updated; producer-side n docstrings (imputation_results.py, two_stage_results.py) and their REGISTRY relay notes gain the finite-support qualifier so estimator sections and the MMM section agree; DEFERRED.md row rewritten to the estimator-owned-totals remainder (CS/EfficientDiD/StackedDiD, whose containers report units/cells).
  • tests/test_naming_guard.py: one CONSUMER_ALLOWLIST entry ("estimator", "diff_diff/mmm.py") mirroring aggregation.py's existing entry for the same independent container field.

Methodology references (required if estimator / math changes)

  • Method name(s): No estimator/math changes — interop scaling of existing ImputationDiD (Borusyak–Jaravel–Spiess 2024) and TwoStageDiD (Gardner 2022) aggregation outputs into PyMC-Marketing lift tests and Google Meridian lognormal ROI priors.
  • Paper / source link(s): PyMC-Marketing lift-test calibration (add_lift_test_measurements schema); Google Meridian "Set custom prior distributions using past experiments" + lognormal_dist_from_mean_std (Meridian 1.7.0). See the REGISTRY "MMM Calibration Export (interop)" section.
  • Any intentional deviations from the source (and why): The scale="auto" assumptions (additive outcome, unweighted fit, fully identified effects) and the correlated-group-container caveats are documented as REGISTRY **Note:** bullets; no pooling/lognormal math changed.

Validation

  • Tests added/updated: tests/test_mmm.py (extraction/routing matrix, container-mode exporter tests, end-to-end workflows with frame-derived oracles, non-finite identification divergence pins for both allowlisted estimators, overflow pin, AggregationResult contract pin, runtime-resolvable annotations pin), tests/test_naming_guard.py (allowlist entry), tests/test_doc_snippets.py (api/mmm.rst added to the executed roster).
  • Backtest / simulation / notebook evidence (if applicable): N/A — deterministic scaling; behavioral pins assert att * n against counts derived independently from the input frames.

Security / privacy

  • Confirm no secrets/PII in this PR: Yes

…the 3.9 aggregation surface

Both MMM exporters now accept the pinned AggregationResult container from
post-fit results.aggregate('simple'|'group') via aggregation_result=, with a
required scale= (numeric, or "auto" for the audited ImputationDiD/TwoStageDiD
producers whose n matches the ATT's averaged support on unweighted, fully
identified fits). Derivation is effect = att * scale, se = se * scale, per row
in to_dataframe() order; everything the container cannot verify fails closed
(raw results objects, EventStudyResults, unsupported levels, non-"att" targets,
unusable inference, non-allowlisted "auto" with per-estimator hints keyed on
the container's estimator provenance).

Docs: REGISTRY MMM section reworded to the new posture with container-mode
Notes; api/mmm.rst examples made self-contained and added to the doc-snippet
executor; both LLM guides updated; producer-side n docstrings and REGISTRY
relay notes gain the finite-support qualifier; DEFERRED.md row rewritten to
the estimator-owned-totals remainder (CS/EfficientDiD/StackedDiD).

Tests: extraction/routing matrix on hand-built containers, container-mode
exporter tests, end-to-end workflows with frame-derived oracles, non-finite
identification divergence pins for both allowlisted estimators, overflow pin,
and a contract pin of exactly the AggregationResult surface consumed. One
CONSUMER_ALLOWLIST entry (estimator, diff_diff/mmm.py) for the container's
provenance field read.
@github-actions

Copy link
Copy Markdown

Overall assessment: ✅ Looks good

Executive summary

  • No unmitigated P0/P1 findings.
  • Scaling and SE propagation match the Registry contract.
  • Empty, non-finite, unsupported-target, and invalid-scale cases fail closed.
  • Documented weighting, identification, and cohort-covariance limitations are P3 informational.
  • One minor docstring inconsistency remains.

Methodology

  • P3 — Documented deviations. scale="auto" assumptions and group-cohort covariance limitations are explicitly noted in REGISTRY.md:L6327-L6342. ImputationDiD and TwoStageDiD count semantics agree with their producers.
    Impact: Known, disclosed limitations only.
    Concrete fix: None required.

Code Quality

No findings.

Performance

No findings.

Maintainability

No findings; schema and provenance contracts have explicit regression pins.

Tech Debt

  • P3 — Tracked. Estimator-owned totals for CS/EfficientDiD/StackedDiD remain tracked in DEFERRED.md:L75.
    Impact: No release blocker.
    Concrete fix: None required.

Security

No findings or apparent secrets.

Documentation/Tests

  • P2 — Meridian docstring overstates pre-filter counts. diff_diff/mmm.py:L852-L860 says both estimators use pre-filter counts, but TwoStageDiD group rows use post-filter support (diff_diff/two_stage_results.py:L442-L456), as correctly stated in REGISTRY.md:L6329.
    Impact: Users may incorrectly believe valid TwoStageDiD group scaling overcounts.
    Concrete fix: State that ImputationDiD uses raw counts at both levels, while TwoStageDiD does so only for simple; its group count is post-filter.

Changed Python files passed syntax compilation. Runtime tests could not be executed because the review environment lacks pytest/numpy/pandas/scipy.

@github-actions

Copy link
Copy Markdown

🔁 AI review rerun (requested by @igerber)

Head SHA: 11123f7c66f9d2c6b1b0f4dbd09dac9de9956af2


Overall assessment: ✅ Looks good

Executive summary

  • No unmitigated P0/P1 findings.
  • The previous Meridian docstring inconsistency is resolved.
  • Scaling and SE propagation match BJS ATT weighting, Gardner’s treated-observation estimand, and the Registry contract.
  • Empty, invalid-inference, unsupported-target, and overflow cases fail closed.
  • Full tests could not run because pytest/numpy/pandas/scipy are unavailable; all changed Python files passed AST parsing.

Methodology

  • P3 — Documented limitations. scale="auto" assumptions and group-cohort covariance caveats are explicitly labeled in docs/methodology/REGISTRY.md:L6327-L6330.
    Impact: Known limitations only.
    Concrete fix: None required.

Code Quality

No findings.

Performance

No findings.

Maintainability

No findings; provenance and aggregation-schema contracts have regression coverage.

Tech Debt

  • P3 — Tracked. Estimator-owned totals for CS/EfficientDiD/StackedDiD remain tracked in DEFERRED.md:L75.
    Impact: No release blocker.
    Concrete fix: None required.

Security

No findings or apparent secrets.

Documentation/Tests

No findings. The previous P2 is fixed: diff_diff/mmm.py:L859-L862 now correctly distinguishes TwoStageDiD’s pre-filter simple count from its post-filter group count, consistent with diff_diff/two_stage_results.py:L388-L396 and L442-L456.

@igerber igerber added the ready-for-ci Triggers CI test workflows label Aug 15, 2026
@igerber
igerber merged commit a16e72c into main Aug 15, 2026
39 of 40 checks passed
@igerber
igerber deleted the feature/mmm-aggregation-container-mode branch August 15, 2026 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-ci Triggers CI test workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant