feat(v4): item 8b of the 4.0 cut - unified to_dataframe(level='event_study') schema on every results class (M-093 done) (#883)

Author: igerber

TODO.md

@@ -86,7 +86,8 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m
 
 | Issue | Location | Origin | Effort | Priority |
 |-------|----------|--------|--------|----------|
-| Re-execute the nine tutorials whose committed outputs still show the 3.x residual-df p-values/CIs after the `df_convention` default moved to `"cluster"` (t(G-1)): `01_basic_did`, `02_staggered_did`, `04_parallel_trends`, `05_honest_did`, `07_pretrends_power`, `09_real_world_examples`, `11_imputation_did`, `13_stacked_did`, `16_wooldridge_etwfe` (not `16_survey_did`: the survey df wins). Run under `DIFF_DIFF_BACKEND=rust` with the per-worktree kernel, reconcile any prose that restates a moved number, and `--nbmake` each; scheduled for the 4.0 release PR (`docs/v4-design.md` section 9 item 10) together with any item-8/9 movers (24 was refreshed in #879; 23 in 7b, re-validated in 7c - it prints no p-values or CIs). 7b's label changes also drift any committed analytical DiD/TWFE summary (", G=" on the variance line) and every SunAbraham summary (the new "Std. errors:" line) - covered by the same re-execution. | `docs/tutorials/*.ipynb` | #879 | Mid | Medium |
+| Relay `to_dataframe(level='group')` to `aggregate('group').to_dataframe()` on CallawaySantAnna/DMLDiD, EfficientDiD and TripleDifference (staggered) - the level still raises with the post-fit pointer while `level='event_study'` was rewired to the aggregate route at the 4.0 cut (item 8b, [M-093]); ImputationDiD/TwoStageDiD keep their native `group` frames and need the same decision | `diff_diff/staggered_results.py`, `diff_diff/efficient_did_results.py`, `diff_diff/triple_diff_results.py` | #883 | Quick | Low |
+| Re-execute the eight tutorials whose committed outputs still show the 3.x residual-df p-values/CIs after the `df_convention` default moved to `"cluster"` (t(G-1)): `01_basic_did`, `02_staggered_did`, `04_parallel_trends`, `05_honest_did`, `07_pretrends_power`, `09_real_world_examples`, `11_imputation_did`, `13_stacked_did` (not `16_survey_did`: the survey df wins; `16_wooldridge_etwfe` was re-executed and reconciled in the 8b PR). Run under `DIFF_DIFF_BACKEND=rust` with the per-worktree kernel, reconcile any prose that restates a moved number, and `--nbmake` each; scheduled for the 4.0 release PR (`docs/v4-design.md` section 9 item 10) together with any item-8/9 movers (24 was refreshed in #879; 23 in 7b, re-validated in 7c - it prints no p-values or CIs). 7b's label changes also drift any committed analytical DiD/TWFE summary (", G=" on the variance line) and every SunAbraham summary (the new "Std. errors:" line) - covered by the same re-execution. | `docs/tutorials/*.ipynb` | #879 | Mid | Medium |
 | TWFE event-study one-way HC2-BM absorbed-FE R parity against the `mpd_absorbed_fe_did` golden (`clubSandwich::vcovCR(cluster=1:n, type="CR2")`): since 4.0 (item 7b) `cluster=False` + hc2_bm reaches the one-way lane directly (no singleton-cluster column needed; `tests/test_v4_cluster_policy.py` pins finiteness and the static-surface LinearRegression parity only), so the open item is the R pin itself; the hc2 lane is pinned by `TestTWFEEventStudyAbsorbedFERParity` and the clustered hc2_bm per-period lane by `test_twfe_event_study_cluster_hc2_bm_per_period_df_matches_clubsandwich`; the MultiPeriodDiD lanes died at the 4.0 cut (PR 4c-ii) | `tests/test_estimators_vcov_type.py` | #v4-4c-i | Mid | Medium |
 | TWFE event-study power profile: `simulate_power`/`simulate_mde`/`simulate_sample_size` have no `_EstimatorProfile` for `TwoWayFixedEffects.fit(event_study=True)` since the MultiPeriodDiD registry entry went with the class (PR 4c-ii); `EventStudyResults` carries no scalar headline, so the profile needs a design (post-period average via the `tests/helpers/event_study_average.py` arithmetic, or a named period) plus its `_SURVEY_FIT_BUILDERS` twin | `diff_diff/power.py` | #v4-4c-ii | Mid | Low |
 | Clustered hc2_bm Bell-McCaffrey DOF guard is batch-relative (`linalg._cr2_bm_dof_inner`: a contrast is NaN'd when its scaled max\|B\| falls below 1e-10 of the batch maximum), so the event-study core keeps the otherwise-unused post-period-average contrast column as the anchor to hold the 3.x per-period NaN masks bit-identical (PR 4c-ii, user decision 2026-10-25). Reproduction: `x_nc = treated * 1[period >= 4] + 3e-6 * default_rng(seed).normal(size=n)` as a covariate on the oracle panel flips the mask between `[eye(k), avg]` and `eye(k)` on 20/20 seeds (cond(X) ~ 9e5; 1e-7 never flips, <= 3e-8 rank-drops) - too BLAS-sensitive for a committed oracle, hence the structural pin `test_hc2_bm_contrast_batch_keeps_post_average_anchor`. Decide whether the guard should use a batch-independent anchor (the treatment contrast, or the absolute floor only) and then drop the column | `diff_diff/estimators.py`, `diff_diff/linalg.py` | #v4-4c-ii | Mid | Medium |

changelog.d/20261031-v4-pr8b-m093-event-study-schema.md

@@ -0,0 +1,7 @@
+### Breaking Changes
+
+- Every results class's `to_dataframe(level="event_study")` now emits the unified `EVENT_STUDY_SCHEMA` ([M-093], item 8b of the 4.0 cut): `event_time, att, se, t_stat, p_value, conf_int_lower, conf_int_upper, cband_lower, cband_upper, n, df, is_reference, estimand` - the frame `results.aggregate("event_study").to_dataframe()` returns (`CallawaySantAnna`/`DMLDiD`, `EfficientDiD`, `ImputationDiD`, `TwoStageDiD`, `StackedDiD`, `ContinuousDiD`, `TripleDifference` staggered, `ChaisemartinDHaultfoeuille`, HAD event-study results, `DurationDiD`, `LWDiD`) or the builder frame (`SunAbraham`, `LPDiD`, `WooldridgeDiD`, `SpilloverDiD`). The 3.x bespoke frames are gone: `relative_period` -> `event_time`; `effect` / `att_glob` / LPDiD's `coefficient` -> `att`; `conf_int_lo`/`hi`, `conf_low`/`high`, HAD's `conf_int_low`/`high` -> `conf_int_lower`/`conf_int_upper`; `n_obs` -> `n` (NaN on the `is_reference` row); dCDH's `horizon` -> `event_time` on the paper's `l1_first_switch` convention, its per-horizon `estimand` labels (`DID^pl_l` / `ref` / `DID_l`) are not carried (`estimand` is `"att"`) and its reference row is marked `is_reference` instead of `n_obs 0`. `WooldridgeDiD` and `LWDiD` append their fit-configuration columns after the schema (`cohort_trends`, `aggregation_weights`; `rolling`, `estimation_method`, `vcov_type`), and `WooldridgeDiD.to_dataframe(level="event_study")` (its default level) now RAISES before `aggregate("event_study", weights=)` instead of returning an empty frame. `CallawaySantAnna`, `EfficientDiD`, `TripleDifference` and analytical `ImputationDiD` / `TwoStageDiD` / `ContinuousDiD` no longer raise "not stored on 4.0 fits" - the level recomputes the surface from the retained kit (bootstrapped `CallawaySantAnna` / `EfficientDiD` / `TripleDifference` fits replay the fit-time bootstrap on every call; an `ImputationDiD` replicate-weight survey fit with `pretrends=True` raises `NotImplementedError` as `aggregate()` does); a kit-less pre-3.9 pickle, or a bootstrapped pickle predating the replay state, that still carries a stored dict now raises where it returned a frame. `level="group"` on those three classes still raises with the post-fit pointer. New levels: `SpilloverDiDResults.to_dataframe(level="simple" | "event_study")` (default `"simple"`, the former one-row frame; aggregate-mode fits raise on `"event_study"`), `HeterogeneousAdoptionDiDEventStudyResults.to_dataframe(level="event_study")`, `DurationDiDResults.to_dataframe(level="event_study")`. An `LPDiD` fit whose stored event-study frame is empty raises instead of returning the empty frame.
+
+### Internal
+
+- The routing rule is the M-153 LWDiD precedent (`aggregate(level).to_dataframe()` on the `AggregationMixin` classes, `build_event_study_surface(self).to_dataframe()` elsewhere); `results_base._ABSENT_SURFACE_HINTS` gains a `SunAbrahamResults` entry. Ledger: M-093 flips to `done` with `test_ref` `tests/test_v4_event_study_schema.py` (the cross-producer schema-prefix / route-equality / error-contract pins; `tests/test_v4_sentinel_retirement.py` keeps the 8a pins) and a tenth amendment note; `docs/v4-design.md` section 5 now states the PREFIX rule and item 8 is complete. REGISTRY (CS, EfficientDiD, Imputation/TwoStage/Stacked relay and reference-row Notes, DDD, HAD, plus new to_dataframe-schema Notes for dCDH, SunAbraham, WooldridgeDiD, LPDiD, DurationDiD and SpilloverDiD), the migration guide, `llms-full.txt`, the practitioner guide, `docs/api/wooldridge_etwfe.rst` and `docs/api/spillover.rst` describe the frame; notebooks 16 (re-executed, outputs refreshed and prose reconciled - it leaves the TODO.md tutorial re-execution row), 19 (reads `event_time`/`att`/`is_reference`) and 14 (export prose) updated. TODO.md gains the `level='group'` relay row.

diff_diff/chaisemartin_dhaultfoeuille_results.py

@@ -1693,9 +1693,19 @@ def to_dataframe(self, level: str = "overall") -> pd.DataFrame:
             - ``"per_period"``: one row per time period with
               ``did_plus_t``, ``did_minus_t``, switching cell counts, and
               the A11-zeroed flags.
-            - ``"event_study"``: one row per horizon (positive and
-              negative/placebo), including a reference period at
-              horizon 0. Available when ``L_max >= 1``.
+            - ``"event_study"``: the unified event-study table -
+              ``self.aggregate("event_study").to_dataframe()`` - with the
+              pinned ``EVENT_STUDY_SCHEMA`` columns (``event_time, att, se,
+              t_stat, p_value, conf_int_lower, conf_int_upper, cband_lower,
+              cband_upper, n, df, is_reference, estimand``), identical across
+              estimators (4.0, ledger row M-093). One row per horizon on the
+              paper's ``l1_first_switch`` convention: placebos at negative
+              ``event_time``, the reference marked ``is_reference`` at 0,
+              ``DID_l`` at ``l >= 1``. The per-horizon estimand labels
+              (``DID^pl_l`` / ``ref`` / ``DID_l``) are not carried - the
+              ``estimand`` column is the ATT contract's ``"att"``;
+              ``summary()`` keeps the paper's labels. Available when
+              ``L_max >= 1``.
             - ``"normalized"``: one row per horizon for the normalized
               effects ``DID^n_l``. Available when ``L_max >= 1``.
             - ``"twfe_weights"``: per-(group, time) TWFE decomposition
@@ -1847,64 +1857,7 @@ def to_dataframe(self, level: str = "overall") -> pd.DataFrame:
             return pd.DataFrame(rows)
 
         elif level == "event_study":
-            rows = []
-            # Placebo horizons (negative keys)
-            if self.placebo_event_study:
-                for h in sorted(self.placebo_event_study.keys()):
-                    entry = self.placebo_event_study[h]
-                    cband = entry.get("cband_conf_int", (np.nan, np.nan))
-                    rows.append(
-                        {
-                            "horizon": h,
-                            "estimand": f"DID^pl_{abs(h)}",
-                            "effect": entry["effect"],
-                            "se": entry["se"],
-                            "t_stat": entry["t_stat"],
-                            "p_value": entry["p_value"],
-                            "conf_int_lower": entry["conf_int"][0],
-                            "conf_int_upper": entry["conf_int"][1],
-                            "n_obs": entry["n_obs"],
-                            "cband_lower": cband[0] if cband else np.nan,
-                            "cband_upper": cband[1] if cband else np.nan,
-                        }
-                    )
-            # Reference period (horizon 0)
-            rows.append(
-                {
-                    "horizon": 0,
-                    "estimand": "ref",
-                    "effect": 0.0,
-                    "se": np.nan,
-                    "t_stat": np.nan,
-                    "p_value": np.nan,
-                    "conf_int_lower": np.nan,
-                    "conf_int_upper": np.nan,
-                    "n_obs": 0,
-                    "cband_lower": np.nan,
-                    "cband_upper": np.nan,
-                }
-            )
-            # Positive horizons
-            if self.event_study_effects:
-                for h in sorted(self.event_study_effects.keys()):
-                    entry = self.event_study_effects[h]
-                    cband = entry.get("cband_conf_int", (np.nan, np.nan))
-                    rows.append(
-                        {
-                            "horizon": h,
-                            "estimand": self._horizon_label(h),
-                            "effect": entry["effect"],
-                            "se": entry["se"],
-                            "t_stat": entry["t_stat"],
-                            "p_value": entry["p_value"],
-                            "conf_int_lower": entry["conf_int"][0],
-                            "conf_int_upper": entry["conf_int"][1],
-                            "n_obs": entry["n_obs"],
-                            "cband_lower": cband[0] if cband else np.nan,
-                            "cband_upper": cband[1] if cband else np.nan,
-                        }
-                    )
-            return pd.DataFrame(rows)
+            return self.aggregate("event_study").to_dataframe()
 
         elif level == "normalized":
             if not self.normalized_effects:

diff_diff/continuous_did_results.py

@@ -494,6 +494,15 @@ def to_dataframe(self, level: str = "dose_response") -> pd.DataFrame:
         ----------
         level : str, default="dose_response"
             ``"dose_response"``, ``"group_time"``, or ``"event_study"``.
+            ``level="event_study"`` returns the unified event-study table -
+            ``self.aggregate("event_study").to_dataframe()`` - with the pinned
+            ``EVENT_STUDY_SCHEMA`` columns (``event_time, att, se, t_stat, p_value,
+            conf_int_lower, conf_int_upper, cband_lower, cband_upper, n, df,
+            is_reference, estimand``), identical across estimators (4.0, ledger
+            row M-093).
+            Analytical fits recompute the binarized event study from the
+            retained kit; bootstrapped fits relay the fit-time percentile
+            surface.
         """
         if level == "dose_response":
             att_df = self.dose_response_att.to_dataframe()
@@ -526,29 +535,7 @@ def to_dataframe(self, level: str = "dose_response") -> pd.DataFrame:
                 )
             return pd.DataFrame(rows)
         elif level == "event_study":
-            if self.event_study_effects is None:
-                raise ValueError(
-                    "Event study effects not stored (only bootstrapped fits "
-                    "materialize the surface at fit). Call "
-                    "results.aggregate('event_study') for the unified "
-                    "post-fit container - analytic fits recompute it from "
-                    "the retained kit; a result unpickled from an older "
-                    "release must be re-fit with diff-diff >= 3.9."
-                )
-            rows = []
-            for rel_t, data in sorted(self.event_study_effects.items()):
-                rows.append(
-                    {
-                        "relative_period": rel_t,
-                        "att_glob": data["effect"],
-                        "se": data["se"],
-                        "t_stat": data["t_stat"],
-                        "p_value": data["p_value"],
-                        "conf_int_lower": data["conf_int"][0],
-                        "conf_int_upper": data["conf_int"][1],
-                    }
-                )
-            return pd.DataFrame(rows)
+            return self.aggregate("event_study").to_dataframe()
         else:
             raise ValueError(
                 f"Unknown level: {level}. Use 'dose_response', 'group_time', or 'event_study'."

diff_diff/duration_did_results.py

@@ -445,11 +445,21 @@ def to_dataframe(self, level: str = "periods") -> pd.DataFrame:
 
         Parameters
         ----------
-        level : {"periods", "att"}
+        level : {"periods", "att", "event_study"}
             ``"periods"``: one row per post-treatment date with the effect,
             pointwise inference, simultaneous band and validity status.
             ``"att"``: the single headline row.
+            ``level="event_study"`` returns the unified event-study table -
+            ``self.aggregate("event_study").to_dataframe()`` - with the pinned
+            ``EVENT_STUDY_SCHEMA`` columns (``event_time, att, se, t_stat, p_value,
+            conf_int_lower, conf_int_upper, cband_lower, cband_upper, n, df,
+            is_reference, estimand``), identical across estimators (4.0, ledger
+            row M-093).
+            Relative event time (``-1`` is the reference row, ``0..`` the
+            post-treatment dates in order).
         """
+        if level == "event_study":
+            return self.aggregate("event_study").to_dataframe()
         if level == "att":
             return pd.DataFrame(
                 [
@@ -482,7 +492,7 @@ def to_dataframe(self, level: str = "periods") -> pd.DataFrame:
                     "status": list(self.period_status),
                 }
             )
-        raise ValueError(f"level must be 'periods' or 'att', got {level!r}")
+        raise ValueError(f"level must be 'periods', 'att' or 'event_study', got {level!r}")
 
     # ------------------------------------------------------------------ #
     # Summary                                                              #

diff_diff/efficient_did_results.py

@@ -810,6 +810,15 @@ def to_dataframe(self, level: str = "group_time") -> pd.DataFrame:
         ----------
         level : str
             ``"group_time"``, ``"event_study"``, or ``"group"``.
+            ``level="event_study"`` returns the unified event-study table -
+            ``self.aggregate("event_study").to_dataframe()`` - with the pinned
+            ``EVENT_STUDY_SCHEMA`` columns (``event_time, att, se, t_stat, p_value,
+            conf_int_lower, conf_int_upper, cband_lower, cband_upper, n, df,
+            is_reference, estimand``), identical across estimators (4.0, ledger
+            row M-093).
+            Bootstrapped fits replay the fit-time multiplier bootstrap on
+            every call; ``"group"`` raises on 4.0 fits with a pointer to
+            ``aggregate('group')``.
         """
         if level == "group_time":
             rows = []
@@ -829,29 +838,7 @@ def to_dataframe(self, level: str = "group_time") -> pd.DataFrame:
             return pd.DataFrame(rows)
 
         elif level == "event_study":
-            if self.event_study_effects is None:
-                raise ValueError(
-                    "Event study effects are not stored on 4.0 fits; use "
-                    "results.aggregate('event_study').to_dataframe() "
-                    "(bootstrapped fits replay the fit-time multiplier "
-                    "bootstrap - no refit needed); a result unpickled from "
-                    "a pre-3.9 release carries no aggregation kit and must "
-                    "be refit."
-                )
-            rows = []
-            for rel_t, data in sorted(self.event_study_effects.items()):
-                rows.append(
-                    {
-                        "relative_period": rel_t,
-                        "effect": data["effect"],
-                        "se": data["se"],
-                        "t_stat": data["t_stat"],
-                        "p_value": data["p_value"],
-                        "conf_int_lower": data["conf_int"][0],
-                        "conf_int_upper": data["conf_int"][1],
-                    }
-                )
-            return pd.DataFrame(rows)
+            return self.aggregate("event_study").to_dataframe()
 
         elif level == "group":
             if self.group_effects is None: