fix(sdid): shape-only pre-treatment fit RMSE, level-gap field, placebo-anchored poor-fit warning

TODO.md

@@ -77,6 +77,7 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m
 
 | Issue | Location | Origin | Effort | Priority |
 |-------|----------|--------|--------|----------|
+| Geo-experiments tutorial: the SDiD fits on the `generate_factor_data(...)` panel (conversions ~1,500, sigma-hat ~310) hit the Frank-Wolfe iteration cap - the placebo fit warns once and the bootstrap cross-check warns on 100/100 draws - yet the narrative presents the bootstrap SE as a paper-faithful robustness check. Pre-existing on `main` under BOTH backends (surfaced when the notebook outputs were refreshed for the shape-only fit diagnostic); the committed April outputs predate the aggregated non-convergence warning. Either rescale / re-tune the DGP (or raise `max_iter` / loosen `min_decrease`) so the refits converge and re-execute, or label the bootstrap result unreliable and drop it from the "Validity evidence" claim. | `docs/tutorials/18_geo_experiments.ipynb` | local codex review of the SDiD shape-only fit-RMSE PR | Mid | Low |
 | Replicate Chang (2020) §4.2.1 ML-design RCS DGP — requires a penalized (Lasso-class) propensity learner or a maintainer fixture via the duck-typed `propensity_learner=` object route; native logit+linear verified pure noise at (N=500, p=100) across ~20 review seeds (att ~ −23..+16, SE ~4-10, EPV ~1.4-1.6 vs threshold 10, fitted out-of-fold clipping ~5-29%, mean ~14%); at (N=200, p=100) `outcome_learner="linear"` fails closed on control-fold rank deficiency even under an oracle propensity (use ridge/sieve there); §4.2.2 is replicated | `tests/test_methodology_dml_did.py`, `diff_diff/_learners.py` | DML PR-B2 | Mid | Low |
 | Optional scheduled end-to-end execution gate for the MMM tutorials (29/30): a cron-only workflow (or extension of `mmm-interop.yml`) that executes both notebooks in isolated exact-pin environments, so a stale/invalid committed posterior cannot stay green indefinitely - today the hybrid posture (deliberate: notebooks execute locally with committed outputs; CI smoke-tests the exporters without sampling; drift tests pin source + committed-output needles) leaves the MCMC claims un-re-executed in CI | `.github/workflows/mmm-interop.yml`, `docs/tutorials/29_mmm_calibration_pymc.ipynb`, `docs/tutorials/30_mmm_calibration_meridian.ipynb` | mmm-interop | Mid | Low |
 | Committed `fixest::feols` event-study golden for TWFE `event_study=True` (within + pooled specs, unbalanced + covariate panels, matched CR1 cluster convention, per-period effects + vcov block) - the in-suite gates are shared-core cross-checks (TWFE-within == MPD-absorb, pooled == MPD bit-exact), so a defect common to the shared core would pass; the live-R harness (`benchmarks/R/benchmark_multiperiod.R`, `feols(y ~ treated * time_f \| unit)`) validated the within design in `docs/benchmarks.rst` but is not a committed regression test - follow the `fixest_did_twfe_golden.json` committed-golden pattern (pytest.skip when absent) | `tests/test_fixest_did_twfe_parity.py`, `benchmarks/R/` | 3(a) R2 | Mid | Medium |

changelog.d/20260904-sdid-shape-only-fit-rmse.md

@@ -0,0 +1,44 @@
+### Fixed
+- **SyntheticDiD `pre_treatment_fit` is now shape-only**: the reported
+  pre-treatment RMSE (and the "Pre-treatment fit is poor" warning) previously
+  measured the raw level residual between the treated mean and the synthetic
+  control, while the Frank-Wolfe unit weights are fit on column-centered
+  outcomes (`intercept=True`, matching R `synthdid`) and deliberately leave a
+  constant level gap to the DiD step. A parallel treated series sitting at a
+  different level therefore reported a large RMSE and a false poor-fit
+  warning even when the ATT was recovered exactly. The RMSE is now taken on
+  the pre-period residual after removing its mean, which is the data-fit
+  component of the centered Frank-Wolfe objective, computed on the
+  normalized outcome scale and rescaled (so a large common outcome level
+  cannot perturb it); `in_time_placebo()` and `sensitivity_to_zeta_omega()`
+  report the same shape-only `pre_fit_rmse`. Results pickled before this
+  release are migrated on load: the shape RMSE and the level gap are
+  recomputed from the stored trajectories and replace the stale level RMSE
+  (which is cleared when no trajectories were stored), never relabeled. Estimates, standard errors
+  and weights are unchanged.
+
+### Behavioral Changes
+- **SyntheticDiD pre-fit diagnostic redefinition**: `pre_treatment_fit` and
+  the `pre_fit_rmse` diagnostic columns drop the level gap, so their values
+  fall for any design with a treated-vs-synthetic level offset, and are NaN
+  with a single pre-period. The poor-fit warning is now anchored to an
+  in-space placebo fit reference (Abadie, Diamond & Hainmueller 2010; Abadie
+  2021): the treated fit is compared with the same statistic over up to 20
+  placebo fits of control units treated as if treated (Algorithm 4 draws at
+  the fit-time zeta), and the warning fires when the placebo p-value is at
+  or below 0.05 (at the 20-draw default: worse than every placebo draw),
+  replacing the unreachable `1 x std(treated pre-outcomes)` rule. The reference is computed for every variance method (one extra
+  Frank-Wolfe solve per draw) from a private RNG stream, so SE draws are
+  unchanged; it needs at least 19 successful draws (`n_bootstrap >= 19`) to
+  fire and is absent when no pseudo-control remains. A treated unit far
+  noisier than every control still fits worse than every placebo and warns;
+  the warning text says so. `summary()` labels the values
+  `Pre-fit RMSE (shape)`, `Pre-fit level gap` and `Pre-fit placebo p-value`.
+
+### Added
+- **`SyntheticDiDResults.pre_treatment_level_gap`**: signed mean pre-period
+  gap (treated minus synthetic), the constant offset absorbed by the DiD
+  step, reported in `summary()` and `to_dict()` for inspection.
+- **`SyntheticDiDResults.pre_fit_placebo_rmse` / `pre_fit_placebo_pvalue`**:
+  the placebo pre-fit reference distribution and the treated fit's placebo
+  p-value behind the poor-fit warning (p-value also in `to_dict()`).

diff_diff/business_report.py

@@ -2348,8 +2348,8 @@ def _render_summary(schema: Dict[str, Any]) -> str:
             else:
                 sentences.append(
                     "The synthetic control is designed to match the treated "
-                    "group's pre-period trajectory (SDiD's weighted-parallel-"
-                    "trends analogue)."
+                    "group's pre-period trend (SDiD's weighted-parallel-"
+                    "trends analogue; a constant level gap is differenced out)."
                 )
         elif verdict == "inconclusive":
             # Round-35 P1 CI review on PR #318: a ``verdict=="inconclusive"``

diff_diff/diagnostic_report.py

@@ -3647,8 +3647,16 @@ def _pt_synthetic_fit(self) -> Dict[str, Any]:
 
         SDiD's design-enforced fit quality substitutes for a standard PT test:
         the synthetic control is explicitly constructed to match the treated
-        group's pre-period trajectory, so small ``pre_treatment_fit`` RMSE
+        group's pre-period trend, so small ``pre_treatment_fit`` RMSE
         means the weighted-PT analogue is satisfied.
+
+        ``pre_treatment_fit`` is SHAPE-ONLY (the constant pre-period level
+        gap is removed; SDID differences it out), whereas classic SCM's
+        ``pre_rmspe`` in ``_pt_scm_fit`` is level-inclusive by design. Both
+        are emitted under the shared ``pre_treatment_fit_rmse`` key as a
+        fit-quality RMSE in each estimator's own metric. NaN (a single
+        pre-period) is reported as ``skipped`` so the narrative never renders
+        ``RMSE = nan``.
         """
         r = self._results
         fit = _to_python_float(getattr(r, "pre_treatment_fit", None))
@@ -3658,6 +3666,15 @@ def _pt_synthetic_fit(self) -> Dict[str, Any]:
                 "method": "synthetic_fit",
                 "reason": "SyntheticDiDResults.pre_treatment_fit is not populated " "on this fit.",
             }
+        if not np.isfinite(fit):
+            return {
+                "status": "skipped",
+                "method": "synthetic_fit",
+                "reason": (
+                    "SyntheticDiDResults.pre_treatment_fit is not defined on this "
+                    "fit (fewer than 2 pre-periods)."
+                ),
+            }
         # Proxy verdict: unlike a classical PT p-value, this is a fit-quality
         # metric. Classify conservatively — phrasing in BR will explain that
         # this is SDiD's design-enforced analogue, not a PT hypothesis test.
@@ -4576,10 +4593,10 @@ def _render_overall_interpretation(schema: Dict[str, Any], labels: Dict[str, str
                 )
             else:
                 sentences.append(
-                    f"The synthetic control matches the treated group's "
-                    f"pre-period trajectory with RMSE = "
+                    f"The synthetic control tracks the treated group's "
+                    f"pre-period trend with shape-only RMSE = "
                     f"{rmse:.3g} (SDiD's design-enforced analogue of parallel "
-                    f"trends)."
+                    f"trends; a constant level gap is differenced out by SDID)."
                     if isinstance(rmse, (int, float))
                     else "SDiD's synthetic control is designed to satisfy the "
                     "weighted parallel-trends analogue."

diff_diff/guides/llms-full.txt

@@ -1880,7 +1880,10 @@ Returned by `SyntheticDiD.fit()`.
 | `noise_level` | `float` | Estimated noise level |
 | `zeta_omega` | `float` | Unit weight regularization |
 | `zeta_lambda` | `float` | Time weight regularization |
-| `pre_treatment_fit` | `float` | Pre-treatment RMSE |
+| `pre_treatment_fit` | `float` | Shape-only pre-treatment RMSE (constant level gap removed: the data-fit component of the centered Frank-Wolfe objective; NaN with one pre-period) |
+| `pre_treatment_level_gap` | `float` | Signed mean pre-period gap, treated minus synthetic (absorbed by the DiD step; diagnostic only) |
+| `pre_fit_placebo_rmse` | `np.ndarray` | Reference distribution: shape-only pre-fit RMSE of up to 20 placebo (control-as-treated) fits; `None` when no placebo fit is possible |
+| `pre_fit_placebo_pvalue` | `float` | Plus-one-adjusted rank of the treated fit among the placebo fits, `(1 + #{placebo >= treated}) / (1 + n_draws)`; a fit diagnostic, not a treatment-effect p-value; the poor-fit warning fires at <= 0.05 (Abadie, Diamond & Hainmueller 2010 in-space placebo fit assessment) |
 
 **Methods:** `summary()`, `print_summary()`, `to_dict()`, `to_dataframe()`, `get_unit_weights_df()`, `get_time_weights_df()`
 

diff_diff/practitioner.py

@@ -798,7 +798,8 @@ def _handle_synthetic(results: Any):
                 "counterfactual well."
             ),
             code=(
-                "print(f'Pre-treatment fit (RMSE): {results.pre_treatment_fit:.4f}')\n"
+                "print(f'Pre-fit RMSE (shape): {results.pre_treatment_fit:.4f}')\n"
+                "print(f'Pre-fit level gap: {results.pre_treatment_level_gap:.4f}')\n"
                 "concentration = results.get_weight_concentration()\n"
                 "print(f\"Effective N: {concentration['effective_n']:.1f}\")\n"
                 "print(f\"Top-5 weight share: {concentration['top_k_share']:.2%}\")"

diff_diff/results.py

@@ -1149,6 +1149,31 @@ class SyntheticDiDResults(BaseResults):
         (for ``"jackknife"``). The ``variance_method`` field disambiguates
         the contents. (The deprecated read-only alias ``placebo_effects``
         returns this array and is removed in v4.0.0.)
+    pre_treatment_fit : float, optional
+        Shape-only pre-treatment fit: the RMSE of the pre-period residual
+        (treated mean minus synthetic control) after removing its mean. The
+        Frank-Wolfe unit weights are fit on column-centered outcomes, so a
+        constant level gap is not a fit failure (SDID differences it out)
+        and is excluded here. NaN with a single pre-period.
+    pre_treatment_level_gap : float, optional
+        Signed mean pre-period gap, treated minus synthetic control. This is
+        the constant offset absorbed by the DiD step; it is reported for
+        inspection and never enters the poor-fit warning.
+    pre_fit_placebo_rmse : np.ndarray, optional
+        Reference distribution for ``pre_treatment_fit``: the shape-only
+        pre-fit RMSE of each placebo fit in which a random set of
+        ``n_treated`` control units is treated as if treated and the unit
+        weights are re-estimated on the remaining controls (Algorithm 4
+        draws, fit-time zeta). At most 20 draws (``min(n_bootstrap, 20)``);
+        ``None`` when no placebo fit is possible (no pseudo-control left, or
+        a single pre-period).
+    pre_fit_placebo_pvalue : float, optional
+        ``(1 + #{placebo RMSE >= treated RMSE}) / (1 + n_draws)``: the
+        plus-one-adjusted rank of the treated units' fit among the placebo
+        fits — a fit diagnostic, not a treatment-effect p-value. The
+        poor-fit warning fires at ``<= 0.05`` (in-space placebo fit
+        assessment, Abadie, Diamond & Hainmueller 2010; Abadie 2021), which
+        needs at least 19 successful draws to be reachable.
     synthetic_pre_trajectory : np.ndarray, optional
         Synthetic control trajectory in pre-treatment periods, shape
         ``(n_pre,)``. Equal to ``Y_pre_control @ omega_eff`` where
@@ -1186,6 +1211,9 @@ class SyntheticDiDResults(BaseResults):
     zeta_omega: Optional[float] = field(default=None)
     zeta_lambda: Optional[float] = field(default=None)
     pre_treatment_fit: Optional[float] = field(default=None)
+    pre_treatment_level_gap: Optional[float] = field(default=None)
+    pre_fit_placebo_rmse: Optional[np.ndarray] = field(default=None)
+    pre_fit_placebo_pvalue: Optional[float] = field(default=None)
     variance_effects: Optional[np.ndarray] = field(default=None)
     n_bootstrap: Optional[int] = field(default=None)
     # Survey design metadata (SurveyMetadata instance from diff_diff.survey)
@@ -1256,6 +1284,33 @@ def __setstate__(self, state: Dict[str, Any]) -> None:
         if "placebo_effects" in state and "variance_effects" not in state:
             state = dict(state)
             state["variance_effects"] = state.pop("placebo_effects")
+        # Pre-v3.11.2 pickles: ``pre_treatment_fit`` was the LEVEL-inclusive
+        # RMSE and the shape-only / placebo-reference fields did not exist.
+        # Never relabel the stale value as shape-only: recompute both
+        # statistics from the stored trajectories when they are present
+        # (retained on results since v3.8), otherwise clear it. The placebo
+        # reference cannot be rebuilt without the panel, so it stays None
+        # (the poor-fit warning is a fit-time event and is not replayed).
+        if "pre_treatment_level_gap" not in state:
+            state = dict(state)
+            treated_pre = state.get("treated_pre_trajectory")
+            synthetic_pre = state.get("synthetic_pre_trajectory")
+            if treated_pre is not None and synthetic_pre is not None:
+                resid = np.asarray(treated_pre, dtype=np.float64) - np.asarray(
+                    synthetic_pre, dtype=np.float64
+                )
+                level_gap = float(np.mean(resid))
+                state["pre_treatment_level_gap"] = level_gap
+                state["pre_treatment_fit"] = (
+                    float(np.sqrt(np.mean((resid - level_gap) ** 2)))
+                    if resid.shape[0] >= 2
+                    else float("nan")
+                )
+            else:
+                state["pre_treatment_level_gap"] = None
+                state["pre_treatment_fit"] = None
+            state.setdefault("pre_fit_placebo_rmse", None)
+            state.setdefault("pre_fit_placebo_pvalue", None)
         self.__dict__.update(state)
 
     @property
@@ -1326,7 +1381,11 @@ def summary(self, alpha: Optional[float] = None) -> str:
             lines.append(f"{'Noise level:':<25} {self.noise_level:>10.4f}")
 
         if self.pre_treatment_fit is not None:
-            lines.append(f"{'Pre-treatment fit (RMSE):':<25} {self.pre_treatment_fit:>10.4f}")
+            lines.append(f"{'Pre-fit RMSE (shape):':<25} {self.pre_treatment_fit:>10.4f}")
+        if self.pre_treatment_level_gap is not None:
+            lines.append(f"{'Pre-fit level gap:':<25} {self.pre_treatment_level_gap:>10.4f}")
+        if self.pre_fit_placebo_pvalue is not None:
+            lines.append(f"{'Pre-fit placebo p-value:':<25} {self.pre_fit_placebo_pvalue:>10.3f}")
 
         # Variance method info
         lines.append(f"{'Variance method:':<25} {self.variance_method:>10}")
@@ -1416,6 +1475,8 @@ def to_dict(self) -> Dict[str, Any]:
             "zeta_omega": self.zeta_omega,
             "zeta_lambda": self.zeta_lambda,
             "pre_treatment_fit": self.pre_treatment_fit,
+            "pre_treatment_level_gap": self.pre_treatment_level_gap,
+            "pre_fit_placebo_pvalue": self.pre_fit_placebo_pvalue,
         }
         if self.n_bootstrap is not None:
             result["n_bootstrap"] = self.n_bootstrap
@@ -1613,7 +1674,9 @@ def in_time_placebo(
             Columns:
                 - ``fake_treatment_period`` — the shifted date
                 - ``att`` — placebo ATT (ideally near 0)
-                - ``pre_fit_rmse`` — RMSE on the fake pre-window
+                - ``pre_fit_rmse`` — shape-only RMSE on the fake pre-window
+                  (pre-period mean gap removed, matching
+                  ``pre_treatment_fit``)
                 - ``n_pre_fake`` — periods before the fake date
                 - ``n_post_fake`` — periods from the fake date onward
 
@@ -1749,7 +1812,10 @@ def in_time_placebo(
                 lambda_fake,
             )
             synthetic_pre_fake_n = Y_pre_c_n @ omega_eff_fake
-            pre_fit_n = float(np.sqrt(np.mean((y_pre_t_mean_n - synthetic_pre_fake_n) ** 2)))
+            # Shape-only RMSE (pre-window mean gap removed), matching the
+            # fit-time ``pre_treatment_fit`` definition. n_pre_fake >= 2 here.
+            resid_fake_n = y_pre_t_mean_n - synthetic_pre_fake_n
+            pre_fit_n = float(np.sqrt(np.mean((resid_fake_n - resid_fake_n.mean()) ** 2)))
             # ATT is scale-equivariant and shift-invariant in Y; RMSE is
             # scale-equivariant. Rescale back to original-Y units.
             row["att"] = float(att_fake_n * Y_scale)
@@ -1789,7 +1855,9 @@ def sensitivity_to_zeta_omega(
             Columns:
                 - ``zeta_omega`` — the regularization value evaluated
                 - ``att`` — resulting ATT
-                - ``pre_fit_rmse`` — RMSE on the original pre-period
+                - ``pre_fit_rmse`` — shape-only RMSE on the original
+                  pre-period (pre-period mean gap removed, matching
+                  ``pre_treatment_fit``; NaN with a single pre-period)
                 - ``max_unit_weight`` — max element of the composed
                   ``omega_eff`` (sensitivity indicator: close to 1 means
                   near-one-hot solutions; close to ``1/n_control`` means
@@ -1895,7 +1963,14 @@ def sensitivity_to_zeta_omega(
                 time_weights,
             )
             synthetic_pre_n = Y_pre_control_n @ omega_eff
-            pre_fit_n = float(np.sqrt(np.mean((y_pre_t_mean_n - synthetic_pre_n) ** 2)))
+            # Shape-only RMSE (pre-period mean gap removed), matching the
+            # fit-time ``pre_treatment_fit`` definition; undefined (NaN) with
+            # a single pre-period, which is a legal fit.
+            resid_n = y_pre_t_mean_n - synthetic_pre_n
+            if resid_n.shape[0] >= 2:
+                pre_fit_n = float(np.sqrt(np.mean((resid_n - resid_n.mean()) ** 2)))
+            else:
+                pre_fit_n = float("nan")
             herf = float(np.sum(omega_eff**2))
             rows.append(
                 {