Simultaneous Bands for Heterogeneous Effects

Why Pointwise Intervals Are Not Enough

A conditional average treatment effect (CATE) describes how the average effect varies with covariates. In applications, a researcher may report estimated effects at several prespecified covariate profiles, across prespecified subgroups, or along a curve such as age. An interval at one profile addresses uncertainty there. It does not, by itself, address the chance that at least one interval in a collection misses its target.

That distinction matters when results are read together. If a researcher scans many pointwise intervals and highlights a profile whose interval excludes zero, the apparent finding may reflect the number of profiles examined, not just the evidence at that one profile. Simultaneous inference targets the collection as a whole: it aims to cover every true effect in the prespecified collection at once with a stated confidence level.

The word simultaneous refers to the joint event that all effects in the specified collection are covered. It does not mean that each effect has a 95% probability of being inside its particular interval after the data have been observed. Coverage is a repeated-sampling property of the procedure, and the target set must be stated: four named subgroups and a continuum of ages are different inferential targets.

Finite Collections of Prespecified Effects

Suppose the target consists of \(m\) effects \(\tau_1,\ldots,\tau_m\), such as CATEs for four subgroups defined in advance. Let \(\hat\tau_j\) estimate \(\tau_j\), and let \(SE_j\) be its standard error. A familiar pointwise normal interval is \(\hat\tau_j\pm z_{1-\alpha/2}SE_j\), where \(z_q\) is the \(q\) quantile of the standard normal distribution. Using this interval for each of \(m\) effects does not generally give \(1-\alpha\) coverage for all \(m\) effects together.

One transparent solution is the Bonferroni method. For a familywise error target \(\alpha\), use a two-sided interval with error probability \(\alpha/m\) for each effect. Equivalently, use the critical value \(z_{1-\alpha/(2m)}\). The term familywise error here refers to the probability that at least one of the \(m\) intervals fails to cover its corresponding effect.

This guarantee can be conservative, particularly when the estimated effects are strongly correlated. Bonferroni does not use the dependence structure: it protects against the worst case permitted by the marginal error bounds. A max-statistic method can use the joint distribution of the estimates to obtain a critical value calibrated to that dependence.

Simultaneous Bands for a CATE Curve

For a curve, the index \(x\) ranges over a prespecified set \(\mathcal X\). A common band has lower and upper limits \(\hat\tau(x)-c\,\widehat{SE}(x)\) and \(\hat\tau(x)+c\,\widehat{SE}(x)\). The critical value \(c\) must be chosen for the joint coverage target, rather than borrowed automatically from a pointwise interval. If \(\mathcal X\) contains only a finite grid, the band covers the effects at that grid. To claim coverage over a continuum, the procedure must calibrate the supremum over that continuum (or justify a sufficiently accurate approximation to it).

A max-statistic approach calibrates the largest standardized estimation error across the target set. Define the ideal statistic \[ T=\sup_{x\in\mathcal X}\left|\frac{\hat\tau(x)-\tau(x)}{\widehat{SE}(x)}\right|. \] If \(c_{1-\alpha}\) is the \(1-\alpha\) quantile of the sampling distribution of \(T\), then the band using that critical value is designed so that all effects are covered together with probability \(1-\alpha\). In practice, \(\tau(x)\) is unknown, so a valid bootstrap or multiplier procedure approximates the distribution of the centered estimation process and takes the maximum over \(x\) in each replicate. The quantile of those replicate maxima supplies the critical value.

This calibration requires more than drawing many bootstrap curves. The estimator and its standard errors must have a valid joint large-sample approximation over the target set, and the resampling scheme must consistently approximate that joint process. If effects are estimated with flexible nuisance models, the inference procedure must account for their estimation and any sample splitting or cross-fitting used. The exact regularity conditions depend on the estimator; a pointwise standard error alone does not establish validity of a whole-curve band.

Justifying the Max-Statistic Band

The simultaneous-band construction follows directly from the definition of a maximum. If \(T\leq c_{1-\alpha}\), then every standardized error is no larger than \(c_{1-\alpha}\) in absolute value. Rearranging that statement at each \(x\) gives the band condition. This short argument explains why the critical value must reflect the maximum rather than the distribution of one standardized error.

Thus, if the distribution of \(T\) is approximated well and \(c_{1-\alpha}\) is its appropriate quantile, the probability of the all-\(x\) coverage event is approximately \(1-\alpha\). In a bootstrap implementation, each replicate generates a standardized centered error process over the same target set, and the largest absolute value in that replicate is recorded. Taking the empirical \(1-\alpha\) quantile of those maxima estimates the required cutoff. The procedure is only as reliable as the joint approximation and the implementation of the target set.

What the Band Does and Does Not Establish

A simultaneous band addresses multiplicity from examining a defined collection of estimates. It does not repair bias, confounding, poor overlap, misspecified models, or failure of the estimator's inferential assumptions. For the band to describe causal effects, the causal estimand must be identified under the study's design or assumptions; for example, consistency, conditional exchangeability, and positivity are relevant in an observational setting. The band quantifies sampling uncertainty conditional on the validity of that causal interpretation.

Prespecification is also important. A band calibrated for four named groups does not automatically cover additional groups selected after examining the results. Nor does a band on a finite grid automatically cover every point on a continuous curve. The family of targets and the calibration method need to match the claims being made.

A max-statistic band may be narrower than Bonferroni intervals when the joint dependence is used effectively, but this is not guaranteed. Its validity depends on a sound approximation of the joint process. Bonferroni is often easier to audit, while max-statistic methods can be more efficient when the dependence structure and resampling method are reliable.

Common Mistakes

• Reading many pointwise intervals as a simultaneous result. A 95% interval at each profile does not imply 95% probability that all intervals cover. Use a familywise adjustment or a valid joint critical value.
• Claiming that all intervals include zero without checking their limits. In the four-subgroup example, the first lower limit is \(0.0506\), which is positive. It excludes zero; the other three intervals include zero.
• Calling a grid band a continuous-curve band. Coverage at selected points does not ensure coverage between them. Calibrate the supremum over the intended domain or limit the claim to the grid.
• Treating a simultaneous band as proof of causality. Simultaneous coverage concerns estimation uncertainty; causal interpretation still depends on identification assumptions and the design.