Uncertainty for Conditional Group Differences

Why Uncertainty Depends on the Predictor Values

When a regression model includes interactions, the fitted difference between two groups can vary with the predictor values at which the groups are compared. A point estimate gives the estimated difference at a particular profile, but it does not show how precisely that difference has been estimated. To quantify uncertainty, we need the variability of the relevant combination of estimated coefficients—not just the standard errors printed beside the individual coefficients.

The key is to write the conditional difference as a linear combination of coefficients. Its variance depends on both the variances of those coefficients and their covariances. This tutorial develops that calculation and uses it to form confidence intervals at specified predictor values.

From an Interaction Model to a Difference

Begin with a two-group model that allows the groups to have different slopes for a quantitative predictor \(X\). Let \(G=0\) identify the reference group and \(G=1\) the comparison group:

At a chosen value \(x\), the fitted mean for group 1 minus the fitted mean for group 0 is

Thus, the estimated difference is \(\hat{\Delta}(x)=\hat{\beta}_2+x\hat{\beta}_3\). Its uncertainty depends on the joint estimation of \(\hat{\beta}_2\) and \(\hat{\beta}_3\). In particular, their covariance matters: the two estimates may tend to vary together or in opposite directions.

For a model with two binary group indicators and a quantitative predictor, a three-way interaction can make the difference between the levels of \(G\) depend on both the other group indicator \(S\) and \(X\). Consider the model

Here \(G\) and \(S\) are coded 0 or 1. Subtracting the fitted mean for \(G=0\) from the fitted mean for \(G=1\), while holding \(x\) and \(S=s\) fixed, gives

The difference is therefore a linear combination of four coefficients. The same principle applies to other interaction models: calculate the difference between the two fitted means, then treat that expression as a contrast of coefficients.

Standard Error and Confidence Interval

Write the coefficient estimates in a vector \(\hat{\boldsymbol{\beta}}\), and let \(\widehat{\operatorname{Cov}}(\hat{\boldsymbol{\beta}})\) be their estimated covariance matrix. If the desired difference is \(c^\mathsf{T}\boldsymbol{\beta}\), where \(c\) is a vector of known weights, its estimate and estimated variance are

For the three-way model above, using coefficient order \((\beta_0,\beta_1,\beta_2,\beta_3,\beta_4,\beta_5,\beta_6,\beta_7)\), the contrast vector for the difference between \(G=1\) and \(G=0\) at \(X=x,S=s\) is

If the regression model has independent, normally distributed errors with mean zero and constant variance, and its mean is correctly specified, the usual linear-model inference uses a \(t\) distribution with the model’s residual degrees of freedom \(df\). The estimated covariance matrix is obtained from the fitted model; in the standard linear model it is \(s^2(X^\mathsf{T}X)^{-1}\). A two-sided pointwise confidence interval with confidence level \(1-\alpha\) is

“Pointwise” means the interval is for the difference at the particular predictor values used to form \(c\). It is not automatically a confidence band that covers the differences at every possible value of \(X\).

Why Covariance Appears: A Short Derivation

For two estimated coefficients, the variance of their weighted sum follows by expanding the variance:

The cross-term occurs because, in general, the two estimates are not independent. For the difference \(\hat{\beta}_2+x\hat{\beta}_3\), take \(a=1\) and \(b=x\):

This expansion is also what the covariance-matrix expression \(c^\mathsf{T}\widehat{\operatorname{Cov}}(\hat{\boldsymbol{\beta}})c\) calculates for a longer contrast. Omitting the covariance terms is justified only when the relevant covariances are zero, not merely because software displays separate standard errors for each coefficient.

Using the Full Contrast in a Three-Way Model

With a three-way interaction, the covariance information must cover every coefficient contributing to the conditional difference. The following example uses the order \((\hat{\beta}_2,\hat{\beta}_3,\hat{\beta}_6,\hat{\beta}_7)\), because the other coefficients cancel when subtracting the two fitted means.

The Same Model Can Give Different Uncertainty at Different Profiles

Changing the predictor values changes the contrast vector. Consequently, both the estimated difference and its standard error can change, even though the fitted model and its covariance matrix remain the same. For the three-way model and covariance matrix in the previous example, consider the difference at \(x=3,s=0\). The contrast is now \(c=(1,3,0,0)^\mathsf{T}\), and the estimated difference is

Only the first two coefficients contribute. Their variances and covariance give

Using the same \(df=40\) critical value, the 95% interval is \(1.6\pm2.021(0.5292)=1.6\pm1.069\), or \((0.531,\ 2.669)\) outcome units. This is a pointwise interval for \(S=0,X=3\), whereas the interval in the preceding example was for \(S=1,X=3\). Both must be interpreted at their stated profiles.

Common Mistakes and Practical Checks

• Leaving out covariance. Adding only the weighted coefficient variances treats estimates as if they were uncorrelated. Use the full covariance matrix, or include every cross-term in the expanded variance.
• Using the wrong contrast. A group difference at \(x\) includes the interaction coefficient multiplied by \(x\). In a three-way model it also includes terms involving the second group indicator and its product with \(x\). Subtract the two fitted means to verify the weights.
• Confusing coefficient standard errors with the contrast standard error. The standard error of a conditional difference is calculated from its contrast vector and the covariance matrix; it is generally not the sum of the individual coefficient standard errors.
• Interpreting a pointwise interval too broadly. An interval at one chosen profile does not guarantee coverage for differences across an entire range of predictor values.
• Interpreting the interval as individual variation or causation. This interval concerns a fitted mean difference under the regression model. By itself, it neither describes the spread of individual outcomes nor establishes a causal effect.