feat(v4): item 7b of the 4.0 cut - SpilloverDiD auto-cluster, uniform cluster=False, cluster metadata (M-080, M-155) (#880)

Author: igerber

TODO.md

@@ -21,6 +21,7 @@ Related tracking surfaces:
 
 | Issue | Location | Origin | Effort | Priority |
 |-------|----------|--------|--------|----------|
+| One-way (unclustered) Bell-McCaffrey DOF for `SunAbraham` / `WooldridgeDiD` `vcov_type="hc2_bm"` + `cluster=False`: both reject the combination at 4.0 (item 7b) because their BM contrast helper `_compute_cr2_bm_contrast_dof` is cluster-only; port the one-way lane through `linalg._compute_bm_dof_from_contrasts` (the TWFE event-study core's helper) with a clubSandwich `vcovCR(cluster=1:n, type="CR2")` parity pin, then lift the two rejects and their tests | `diff_diff/sun_abraham.py`, `diff_diff/wooldridge.py` | #v4-7b | Mid | Low |
 | `decompose_twfe_weights(method="aipw")`: port upstream `implicit_aipw_weights` (covariate-adjusted only; `twfe_weights.py` rejects the value today and lists `"fwl"`). The R goldens already carry the `decompose.aipw` / `balance.aipw` / `two_period.*` blocks under `meta.reserved_blocks`, so no R re-run is needed | `diff_diff/twfe_weights.py` | #812 | Mid | Medium |
 | Consolidate the remaining estimator-entangled DR/logit score variants (`staggered.py::_doubly_robust` + RC twins, `triple_diff.py`, `lwdid.py`, `wooldridge.py`) onto the shared `_dr_scores.py` module, each migration with its own committed oracle capture (the ContinuousDiD lift's two-tier pattern in `tests/test_dr_scores.py`); and add a ridge vcov path to `solve_ridge` if an estimator ever needs analytical ridge inference | `diff_diff/_dr_scores.py` | dml-b0 | Mid | Low |
 | Numeric between-period cohorts (e.g. `first_treat=4.5` with integer times) are rejected by LWDiD while CallawaySantAnna estimates them and LWDiD's own datetime/Period cohorts map to the next observed period — close the dtype asymmetry by adopting the next-observed-period mapping for numeric cohorts too (contract documented in REGISTRY cohort-encodings Note + `docs/api/lwdid.rst` Input Contract). Lands only after PR #588 merges | `diff_diff/lwdid.py` | #588 | Quick | Low |
@@ -85,8 +86,8 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m
 
 | Issue | Location | Origin | Effort | Priority |
 |-------|----------|--------|--------|----------|
-| Re-execute the nine tutorials whose committed outputs still show the 3.x residual-df p-values/CIs after the `df_convention` default moved to `"cluster"` (t(G-1)): `01_basic_did`, `02_staggered_did`, `04_parallel_trends`, `05_honest_did`, `07_pretrends_power`, `09_real_world_examples`, `11_imputation_did`, `13_stacked_did`, `16_wooldridge_etwfe` (not `16_survey_did`: the survey df wins). Run under `DIFF_DIFF_BACKEND=rust` with the per-worktree kernel, reconcile any prose that restates a moved number, and `--nbmake` each; scheduled for the 4.0 release PR (`docs/v4-design.md` section 9 item 10) together with any item-7b/8/9 movers (24 was refreshed in #879; 23 belongs to 7b). | `docs/tutorials/*.ipynb` | #879 | Mid | Medium |
-| TWFE event-study one-way HC2-BM absorbed-FE R parity against the `mpd_absorbed_fe_did` golden (`clubSandwich::vcovCR(cluster=1:n, type="CR2")`): the within spec with hc2_bm auto-clusters at unit, so the one-way lane needs an explicit singleton-cluster column (`cluster=<row id>` + hc2_bm = CR2 with G=n, PT2018 §3.3) once the G=n CR2 path is verified; the hc2 lane is pinned by `TestTWFEEventStudyAbsorbedFERParity` and the clustered hc2_bm per-period lane by `test_twfe_event_study_cluster_hc2_bm_per_period_df_matches_clubsandwich`; the MultiPeriodDiD lanes died at the 4.0 cut (PR 4c-ii) | `tests/test_estimators_vcov_type.py` | #v4-4c-i | Mid | Medium |
+| Re-execute the nine tutorials whose committed outputs still show the 3.x residual-df p-values/CIs after the `df_convention` default moved to `"cluster"` (t(G-1)): `01_basic_did`, `02_staggered_did`, `04_parallel_trends`, `05_honest_did`, `07_pretrends_power`, `09_real_world_examples`, `11_imputation_did`, `13_stacked_did`, `16_wooldridge_etwfe` (not `16_survey_did`: the survey df wins). Run under `DIFF_DIFF_BACKEND=rust` with the per-worktree kernel, reconcile any prose that restates a moved number, and `--nbmake` each; scheduled for the 4.0 release PR (`docs/v4-design.md` section 9 item 10) together with any item-7c/8/9 movers (24 was refreshed in #879; 23 in 7b and again in 7c). 7b's label changes also drift any committed analytical DiD/TWFE summary (", G=" on the variance line) and every SunAbraham summary (the new "Std. errors:" line) - covered by the same re-execution. | `docs/tutorials/*.ipynb` | #879 | Mid | Medium |
+| TWFE event-study one-way HC2-BM absorbed-FE R parity against the `mpd_absorbed_fe_did` golden (`clubSandwich::vcovCR(cluster=1:n, type="CR2")`): since 4.0 (item 7b) `cluster=False` + hc2_bm reaches the one-way lane directly (no singleton-cluster column needed; `tests/test_v4_cluster_policy.py` pins finiteness and the static-surface LinearRegression parity only), so the open item is the R pin itself; the hc2 lane is pinned by `TestTWFEEventStudyAbsorbedFERParity` and the clustered hc2_bm per-period lane by `test_twfe_event_study_cluster_hc2_bm_per_period_df_matches_clubsandwich`; the MultiPeriodDiD lanes died at the 4.0 cut (PR 4c-ii) | `tests/test_estimators_vcov_type.py` | #v4-4c-i | Mid | Medium |
 | TWFE event-study power profile: `simulate_power`/`simulate_mde`/`simulate_sample_size` have no `_EstimatorProfile` for `TwoWayFixedEffects.fit(event_study=True)` since the MultiPeriodDiD registry entry went with the class (PR 4c-ii); `EventStudyResults` carries no scalar headline, so the profile needs a design (post-period average via the `tests/helpers/event_study_average.py` arithmetic, or a named period) plus its `_SURVEY_FIT_BUILDERS` twin | `diff_diff/power.py` | #v4-4c-ii | Mid | Low |
 | Clustered hc2_bm Bell-McCaffrey DOF guard is batch-relative (`linalg._cr2_bm_dof_inner`: a contrast is NaN'd when its scaled max\|B\| falls below 1e-10 of the batch maximum), so the event-study core keeps the otherwise-unused post-period-average contrast column as the anchor to hold the 3.x per-period NaN masks bit-identical (PR 4c-ii, user decision 2026-10-25). Reproduction: `x_nc = treated * 1[period >= 4] + 3e-6 * default_rng(seed).normal(size=n)` as a covariate on the oracle panel flips the mask between `[eye(k), avg]` and `eye(k)` on 20/20 seeds (cond(X) ~ 9e5; 1e-7 never flips, <= 3e-8 rank-drops) - too BLAS-sensitive for a committed oracle, hence the structural pin `test_hc2_bm_contrast_batch_keeps_post_average_anchor`. Decide whether the guard should use a batch-independent anchor (the treatment contrast, or the absolute floor only) and then drop the column | `diff_diff/estimators.py`, `diff_diff/linalg.py` | #v4-4c-ii | Mid | Medium |
 | Re-validate the plan-review dual engine on `gpt-6.1-sol` with `tools/plan-review-eval`: re-pin every arm's composite `cli_version` to the live Claude Code + codex CLI pair, add a gpt-6.1-sol arm, run the harness against the gpt-5.6-sol baseline (the 2026-10 swap carried the model over from the code-review campaign 2 verdict without a plan-review campaign). | `tools/plan-review-eval/config/configs.json` | reviewer-eval gpt61sol PR-2 | Heavy | Low |

changelog.d/20261029-v4-pr7b-m080-cluster-policy.md

@@ -0,0 +1,10 @@
+### Breaking Changes
+
+- `SpilloverDiD` auto-clusters at `unit` when `cluster=` is omitted ([M-080], item 7b of the 4.0 cut), mirroring `TwoWayFixedEffects`' carve-outs (explicit `cluster=` wins; the auto-cluster is dropped on the Conley path and never injected as a survey PSU). Its default standard errors, p-values and CIs move; `cluster=False` reproduces the 3.x observation-level HC1 numbers. Results report `cluster_name="unit"` / `n_clusters`. Inference stays on the residual df until the `df_convention` knob lands ([M-154], the next PR).
+- `cluster=False` is the uniform explicit off-switch ([M-080]): the plain heteroskedasticity-robust sandwich with residual-df t inference on `TwoWayFixedEffects` (static + event-study; one-way CR2 Bell-McCaffrey under `hc2_bm`), `SunAbraham`, `WooldridgeDiD` (OLS within-transform lane with the absorbed-FE full-K rescale; logit/poisson plain QMLE sandwich) and `LPDiD` (per-horizon HC1; `if_robust` unclustered IF variance on the regression-adjustment path; implicit per-observation PSUs under a survey design); an accepted bit-identical no-op on the unit-level-by-construction estimators (`CallawaySantAnna` incl. its RC auto-PSU path, `EfficientDiD` incl. `hausman_pretest`, `DMLDiD`, `ChaisemartinDHaultfoeuille`, staggered `TripleDifference`, `LWDiD` incl. the sensitivity helpers, `HeterogeneousAdoptionDiD`, `ImputationDiD`, `TwoStageDiD`) and on the 2x2 `DifferenceInDifferences` / 2x2x2 `TripleDifference` surfaces; still rejected on `StackedDiD`. Rejected with `inference="wild_bootstrap"` (DiD/TWFE), `WooldridgeDiD(n_bootstrap > 0)` and `vcov_type="hc2_bm"` on `SunAbraham`/`WooldridgeDiD` (cluster-only Bell-McCaffrey DOF; tracked in TODO.md).
+- `cluster` accepts `str | None | False` only ([M-155], `diff_diff.utils.validate_cluster`): `True` and non-`str` column labels raise one shared `ValueError` at construction, through `set_params` and at `fit()` on every estimator exposing the parameter (DMLDiD's existing rule generalized; integer-labelled cluster columns must be renamed).
+- Cluster metadata: `DifferenceInDifferences` / `TwoWayFixedEffects` report `n_clusters` from the regression's effective cluster count on non-survey analytical fits (previously `None` unless the wild bootstrap ran); `SunAbrahamResults` gains an appended `n_clusters` field and reports `cluster_name` on clustered analytical fits (previously `None` off the Conley path), and its `summary()` renders the variance-family label on analytical non-survey fits; `TwoWayFixedEffects` under a survey design with no explicit `cluster=` reports `cluster_name=None` (the auto-cluster is never injected as a PSU).
+
+### Internal
+
+- Ledger: M-080 and the new M-155 (shared cluster contract) -> `done`; new M-154 (SpilloverDiD `df_convention`, item 7c) -> `planned`; pinned by `tests/test_v4_cluster_policy.py`. Tutorial 23 section 6 shows the four-row SE table (CR1 at unit, HC1 via `cluster=False`, Conley lag 0 / lag 1) and was re-executed; `tests/test_t23_spillover_tva_drift.py` pins the new default and the table's source.

diff_diff/chaisemartin_dhaultfoeuille.py

@@ -45,7 +45,7 @@
     DCDHBootstrapResults,
 )
 from diff_diff.linalg import solve_ols
-from diff_diff.utils import safe_inference, validate_n_bootstrap
+from diff_diff.utils import ClusterArg, safe_inference, validate_cluster, validate_n_bootstrap
 
 __all__ = [
     "ChaisemartinDHaultfoeuille",
@@ -407,12 +407,14 @@ class ChaisemartinDHaultfoeuille(ChaisemartinDHaultfoeuilleBootstrapMixin, BaseE
     ----------
     alpha : float, default=0.05
         Significance level for confidence intervals.
-    cluster : str, optional, default=None
-        Must be ``None`` (the default). User-specified clustering via
-        this kwarg is not supported — passing any non-``None`` value
-        raises ``NotImplementedError`` at construction time (and the
-        same gate fires from ``set_params``). The effective clustering
-        depends on how you call ``fit()``:
+    cluster : str, None or False, optional, default=None
+        ``None`` (the default) and ``False`` (4.0, M-080: an accepted explicit
+        no-op - the variance is group-level by construction) both select the
+        default path. User-specified clustering via this kwarg is not
+        supported — a column name raises ``NotImplementedError`` at
+        construction time (and the same gate fires from ``set_params``);
+        other values raise the shared ``ValueError`` (labels must be ``str``).
+        The effective clustering depends on how you call ``fit()``:
 
         - **Default (no survey_design)**: clustered at the group level
           via the cohort-recentered influence-function plug-in
@@ -731,7 +733,7 @@ class ChaisemartinDHaultfoeuille(ChaisemartinDHaultfoeuilleBootstrapMixin, BaseE
     def __init__(
         self,
         alpha: float = 0.05,
-        cluster: Optional[str] = None,
+        cluster: ClusterArg = None,
         n_bootstrap: int = 0,
         bootstrap_weights: str = "rademacher",
         seed: Optional[int] = None,
@@ -778,7 +780,12 @@ def __init__(
                 "paths_of_interest=[(...), ...] for explicit user-"
                 "specified paths. Set one and leave the other as None."
             )
-        if cluster is not None:
+        # Shared cluster contract first ([M-155]): True / non-str labels raise the
+        # shared ValueError; False is an accepted explicit no-op ([M-080] category
+        # b: the variance is unit-level by construction); a column name is
+        # the unsupported custom clustering below.
+        cluster = validate_cluster(cluster)
+        if cluster is not None and cluster is not False:
             raise NotImplementedError(
                 f"cluster={cluster!r}: user-specified clustering is not "
                 f"supported in ChaisemartinDHaultfoeuille. dCDH clusters at "
@@ -787,8 +794,9 @@ def __init__(
                 f"multiplier bootstrap. Under survey_design with strictly-"
                 f"coarser PSUs, bootstrap clustering automatically upgrades "
                 f"to PSU-level Hall-Mammen wild. To use the default path, "
-                f"pass cluster=None (the "
-                f"default). Custom clustering is reserved for a future "
+                f"pass cluster=None (the default) or cluster=False (an accepted "
+                f"no-op: the variance is unit-level by construction). Custom "
+                f"clustering is reserved for a future "
                 f"phase. See REGISTRY.md ChaisemartinDHaultfoeuille section "
                 f"for the full contract."
             )
@@ -853,14 +861,16 @@ def _validate_invariants(self) -> None:
                 "paths_of_interest=[(...), ...] for explicit user-"
                 "specified paths. Set one and leave the other as None."
             )
-        if self.cluster is not None:
+        validate_cluster(self.cluster)
+        if self.cluster is not None and self.cluster is not False:
             raise NotImplementedError(
                 f"cluster={self.cluster!r}: user-specified clustering is "
                 f"not supported in ChaisemartinDHaultfoeuille. dCDH clusters "
                 f"at the group level by default; under survey_design with "
                 f"strictly-coarser PSUs the bootstrap automatically upgrades "
                 f"to PSU-level Hall-Mammen wild clustering. Pass cluster=None "
-                f"(the default) to use this path. User-specified custom "
+                f"(the default) or cluster=False (accepted no-op) to use this "
+                f"path. User-specified custom "
                 f"clustering is reserved for a future phase. See REGISTRY.md "
                 f"ChaisemartinDHaultfoeuille section for the full contract."
             )
@@ -1041,6 +1051,9 @@ def fit(
         # public parameters are unit/covariates.
         group = unit
         controls = covariates
+        # Fit-time re-check of the shared cluster contract ([M-155]; the
+        # custom-clustering NotImplementedError lives in __init__).
+        validate_cluster(self.cluster)
         # ------------------------------------------------------------------
         # Step 1: Column validation
         # ------------------------------------------------------------------