Confidence Intervals for Subgroup Contrasts

Why a Contrast Needs Its Own Interval

A subgroup analysis may report a treatment effect for each subgroup, but the scientific question is often whether those effects differ. That is a question about a contrast between estimates, not about whether one subgroup-specific confidence interval excludes zero while another includes it. The uncertainty in the contrast depends on how the estimates vary together.

Let \(G\) denote a pretreatment subgroup, with two categories labelled 0 and 1, and let \(Y(1)\) and \(Y(0)\) denote the potential outcomes under treatment and control. For a specified target population within subgroup \(g\), define the average treatment effect as \(\tau_g=E[Y(1)-Y(0)\mid G=g]\). The contrast considered here is \(\Delta=\tau_1-\tau_0\). It measures the difference in treatment effects on the chosen outcome scale and in the chosen target populations.

A confidence interval for \(\Delta\) concerns sampling uncertainty in an estimate of this contrast. Its causal interpretation also depends on identifying the subgroup effects: for example, a randomized study with consistent treatment assignment and adequate treatment representation in both subgroups can identify them under the trial’s target-population definition. In an observational study, identification additionally requires suitable control of confounding. An interval cannot correct bias from a failure of causal identification.

Variance and Confidence Interval for a Difference

Suppose the two estimates are approximately jointly normally distributed around their target values, with estimated standard errors \(SE_1\) and \(SE_0\), and estimated covariance \(\widehat{\operatorname{Cov}}(\hat{\tau}_1,\hat{\tau}_0)\). The variance of a difference is the sum of the two variances minus twice their covariance:

The minus sign matters. If estimates tend to move together across repeated samples, their positive covariance reduces the variability of their difference. If they tend to move in opposite directions, negative covariance increases the variability of the difference. Treating the estimates as independent when they are not can therefore make the interval too narrow or too wide.

For a large-sample Wald interval with approximately normal sampling distribution, a two-sided 95% confidence interval is:

The multiplier 1.96 is the 97.5th percentile of the standard normal distribution. Under the approximation, intervals constructed this way cover the target contrast in about 95% of repeated samples. The interval is about \(\Delta\), not about either subgroup effect alone.

Why the Covariance Term Appears

The variance formula follows directly from the variance of a linear combination. Write the centered estimates as \(\hat{\tau}_1-\tau_1\) and \(\hat{\tau}_0-\tau_0\). Since \(\hat{\Delta}-\Delta=(\hat{\tau}_1-\tau_1)-(\hat{\tau}_0-\tau_0)\), expanding the squared centered difference gives:

The final term comes from the two cross-products in the square; omitting it assumes the covariance is zero. That assumption is justified only when the estimates are uncorrelated under the sampling procedure, not merely because they refer to different subgroups.

Estimates can be correlated when they use shared observations, common fitted model parameters, or the same estimated standardization weights. A covariance estimate can be obtained from a joint sandwich variance calculation or from paired resampling: in each bootstrap replicate, resample the data according to the study design, re-estimate both subgroup effects, and retain the pair. The covariance across these paired estimates estimates their joint sampling covariance. Resampling the two effects independently would discard the dependence being estimated.

Covariance Depends on the Estimation Procedure

There is no universal rule that subgroup estimates are independent. If they are estimated from disjoint observations using separate procedures, independence may be reasonable under an appropriate sampling model. But that is a property to establish from the design and estimation procedure. A shared control group, common regression coefficients, or weights estimated from the full sample can link the estimates even when subgroup membership itself is disjoint.

For a fitted model, covariance can also be obtained by writing the contrast as a linear combination of model coefficients. If the estimated coefficient vector is \(\hat{\boldsymbol{\beta}}\), its estimated covariance matrix is \(\widehat{\mathbf V}\), and the contrast is \(\mathbf c^\mathsf{T}\hat{\boldsymbol{\beta}}\), then its estimated variance is \(\mathbf c^\mathsf{T}\widehat{\mathbf V}\mathbf c\). This is the same variance rule, expressed for multiple coefficients. It is particularly useful when subgroup effects are standardized predictions rather than single model coefficients.

Reading a Regression Interaction as a Contrast

With a binary subgroup and an additive linear model, a treatment-by-subgroup coefficient can equal the difference between subgroup treatment effects. Consider a model for a continuous outcome:

where \(A\) is coded 0 for control and 1 for treatment, and \(G\) is coded 0 or 1. Under this model, the fitted treatment effect for subgroup 0 is \(\beta_A\). For subgroup 1 it is \(\beta_A+\beta_{AG}\). Therefore their difference is \(\beta_{AG}\). The identity concerns the model’s additive outcome scale; interpreting it causally requires the relevant design or confounding-control assumptions, as well as a model appropriate for the target effect.

Common Mistakes and Interpretation

A frequent error is to reason that a difference must be present because one subgroup’s interval excludes zero while the other subgroup’s interval includes zero. That compares two decisions about zero, not the subgroup effects with each other. Conversely, overlapping subgroup intervals do not demonstrate that the effects are equal. The appropriate uncertainty calculation is a confidence interval, or a test, for \(\Delta\) itself, using the joint covariance.

Another error is to set the covariance to zero just because the effects refer to different subgroups. Before doing so, check whether the estimates share observations, fitted parameters, standardization weights, or other estimated quantities. If the covariance is unavailable, the contrast interval cannot be fully justified by pretending it is zero; obtain a joint variance estimate appropriate to the estimator and study design.

Finally, a confidence interval describes uncertainty under the stated sampling and estimation assumptions. A narrow interval does not rule out bias from confounding, selection, or a mismatched target population, and a wide interval does not establish that effects are equal. Keep the scale and units explicit: a contrast of risk differences is measured in risk-difference units, while a contrast of mean differences is measured in outcome units.