Uncertainty in Differences Between Predicted Means

Why a Difference Needs Its Own Uncertainty Calculation

Two fitted means from the same regression model are usually related: they are calculated using the same estimated coefficients. As a result, the uncertainty in their difference is not generally found by treating the two predictions as independent. The covariance between them is part of the calculation.

The target here is the difference in the model’s conditional mean outcome for two specified predictor profiles. It is not the difference between two future individual outcomes, which also includes individual outcome variation. The direction of the comparison matters: “profile B minus profile A” is the negative of “profile A minus profile B.”

Definition: Let $x_A$ and $x_B$ be the predictor vectors for two profiles in a regression model, and let $\hat{\mu}_A$ and $\hat{\mu}_B$ be their fitted conditional means. The estimated difference, in the stated direction, is $\hat{\Delta}=\hat{\mu}_B-\hat{\mu}_A$. Its target is the corresponding difference between the two conditional means in the population model.

The Model and the Difference Formula

For a linear regression model, write the observations as $Y=X\beta+\varepsilon$, where $X$ is the model matrix, $\beta$ is the coefficient vector, and $\varepsilon$ is the error vector. For the usual standard error and t-based confidence interval, assume the model’s mean structure is correctly specified, the errors are independent with mean zero and common variance $\sigma^2$, and the model matrix has full column rank. Normal errors, $\varepsilon\sim N(0,\sigma^2 I)$, give the usual exact t-based interval. In applications, the linear mean structure, independence, and roughly constant variance should be judged using the study design and appropriate diagnostic checks; the normality assumption is especially relevant for exact small-sample inference.

Let $p$ be the number of fitted coefficients. The fitted mean at profile $A$ is $x_A^\mathsf{T}\hat{\beta}$, and at profile $B$ it is $x_B^\mathsf{T}\hat{\beta}$. Thus their estimated difference is a linear combination of the same coefficient estimates. With $s^2$ denoting the residual mean square, the estimated coefficient covariance matrix is $s^2(X^\mathsf{T}X)^{-1}$. The standard error follows by applying that covariance matrix to the difference between the profile vectors.

Theorem: For the difference $\hat{\Delta}=\hat{\mu}_B-\hat{\mu}_A$, let $c=x_B-x_A$. Under the stated linear model, $$ \widehat{\operatorname{Var}}(\hat{\Delta}) =s^2c^\mathsf{T}(X^\mathsf{T}X)^{-1}c =\widehat{\operatorname{Var}}(\hat{\mu}_B) +\widehat{\operatorname{Var}}(\hat{\mu}_A) -2\widehat{\operatorname{Cov}}(\hat{\mu}_A,\hat{\mu}_B), \qquad SE(\hat{\Delta})=\sqrt{\widehat{\operatorname{Var}}(\hat{\Delta})}. $$

The covariance term is not an optional adjustment: it is part of the variance of a difference. Positive covariance reduces the variance compared with the sum of the two prediction variances; negative covariance increases it. The covariance can be positive, zero, or negative, depending on the profiles and fitted model.

Why the Covariance Appears

The variance rule for two random quantities $U$ and $V$ is $\operatorname{Var}(U-V)=\operatorname{Var}(U)+\operatorname{Var}(V)-2\operatorname{Cov}(U,V)$. This rule applies to the two fitted means because both depend on $\hat{\beta}$. More explicitly, the difference estimate is $c^\mathsf{T}\hat{\beta}$. Its variance is

\operatorname{Var}(c^\mathsf{T}\hat{\beta}) =c^\mathsf{T}\operatorname{Var}(\hat{\beta})c =\sigma^2c^\mathsf{T}(X^\mathsf{T}X)^{-1}c.

Replacing the unknown $\sigma^2$ with $s^2$ gives the estimated variance in the theorem. To see the role of covariance directly, expand $\hat{\Delta}=\hat{\mu}_B-\hat{\mu}_A$: the two cross-products in the expansion together contribute $-2\widehat{\operatorname{Cov}}(\hat{\mu}_A,\hat{\mu}_B)$. Omitting them is equivalent to assuming that the two fitted means have zero covariance, which is generally unjustified.

When the model errors are normal and the model matrix has $p$ columns, the usual t interval uses $df=n-p$. A $100(1-\alpha)\%$ confidence interval for the population difference is

\hat{\Delta}\ \pm\ t_{1-\alpha/2,\,df}\,SE(\hat{\Delta}).

This interval concerns the difference in conditional means for the two specified profiles. Its repeated-sampling interpretation is that the procedure produces intervals containing that population difference in $100(1-\alpha)\%$ of repeated samples under the model assumptions. It does not say there is a $100(1-\alpha)\%$ probability that the fixed difference lies in this particular calculated interval.

Worked Example: Two Nearby Profiles in Simple Regression

Suppose a hypothetical regression relates exam score to hours of practice. The fitted slope is 2.0 points per hour. For a hypothetical sample, $n=10$, $\bar{x}=5$ hours, $S_{xx}=10$ hours squared, and the residual standard deviation is $s=2$ points. Compare the fitted mean at 4 hours with the fitted mean at 5.5 hours, defining the difference as the latter minus the former. Assume the linear mean relationship and the regression error assumptions are reasonable, the profiles lie within the observed practice-hour range, and residual checks do not show a major departure from constant variance or normality.

In simple regression, the estimated variance of a fitted mean at $x$ is $s^2[1/n+(x-\bar{x})^2/S_{xx}]$, and the covariance of two fitted means at $x_A$ and $x_B$ is $s^2[1/n+(x_A-\bar{x})(x_B-\bar{x})/S_{xx}]$. Therefore, for 4 and 5.5 hours,

\widehat{\operatorname{Var}}(\hat{\mu}_A) =4\left(\frac{1}{10}+\frac{(4-5)^2}{10}\right)=0.8\text{ points}^2, \qquad \widehat{\operatorname{Var}}(\hat{\mu}_B) =4\left(\frac{1}{10}+\frac{(5.5-5)^2}{10}\right)=0.5\text{ points}^2.

Their covariance is

\widehat{\operatorname{Cov}}(\hat{\mu}_A,\hat{\mu}_B) =4\left(\frac{1}{10}+\frac{(4-5)(5.5-5)}{10}\right) =0.2\text{ points}^2.

The fitted difference is the slope multiplied by the 1.5-hour separation: $\hat{\Delta}=2.0(5.5-4)=3.0$ points. Its estimated variance and standard error are

\widehat{\operatorname{Var}}(\hat{\Delta})=0.8+0.5-2(0.2)=0.9\text{ points}^2, \qquad SE(\hat{\Delta})=\sqrt{0.9}=0.9487\text{ points}.

Here $df=10-2=8$. Using $t_{0.975,8}=2.306$, the 95% interval is $3.0\pm2.306\sqrt{0.9}$, or approximately $(0.8123,5.188)$ points. Under the model assumptions, the data are consistent with a population mean exam score at 5.5 hours that is about 0.81 to 5.19 points higher than at 4 hours. This describes a difference in fitted means, not a guaranteed change for an individual student.

Worked Example: Negative Covariance Increases the Uncertainty

Consider another hypothetical simple regression relating a measured outcome to a quantitative predictor. The sample has $n=10$, $\bar{x}=5$, $S_{xx}=120$, and $s=4$ outcome units. Suppose the fitted slope is $-1.5$ outcome units per predictor unit. Compare the fitted mean at $x=9$ with that at $x=1$, defining the difference as the mean at 9 minus the mean at 1. Assume both profiles are within the observed range, the linear form is appropriate there, and the errors are independent, have approximately constant variance, and are sufficiently normal for t inference.

The fitted difference is $\hat{\Delta}=-1.5(9-1)=-12$ outcome units. Each prediction has estimated variance

16\left(\frac{1}{10}+\frac{(9-5)^2}{120}\right) =16\left(\frac{1}{10}+\frac{16}{120}\right) =\frac{56}{15} \approx3.7333\text{ outcome units}^2.

The covariance between the two predictions is negative because the profiles lie on opposite sides of $\bar{x}$:

16\left(\frac{1}{10}+\frac{(1-5)(9-5)}{120}\right) =16\left(\frac{1}{10}-\frac{16}{120}\right) =-\frac{8}{15} \approx-0.5333\text{ outcome units}^2.

Consequently,

\widehat{\operatorname{Var}}(\hat{\Delta}) =\frac{56}{15}+\frac{56}{15}-2\left(-\frac{8}{15}\right) =\frac{128}{15} \approx8.5333\text{ outcome units}^2, \qquad SE(\hat{\Delta})=\sqrt{\frac{128}{15}}\approx2.9212\text{ outcome units}.

With $df=8$, a 95% interval uses $t_{0.975,8}=2.306$. Using the unrounded standard error and critical value, the interval is approximately

-12\pm2.306\sqrt{\frac{128}{15}} \approx(-18.736,-5.264)\text{ outcome units}.

The entire interval is below zero. For these profiles, the fitted mean at $x=9$ is estimated to be lower than the fitted mean at $x=1$; the interval gives plausible values for that population mean difference under the model. The negative covariance increases the difference’s variance: subtracting a negative covariance adds to the sum of the prediction variances.

Worked Example: Calculating a Difference From a Coefficient Covariance Matrix

Suppose a hypothetical multiple regression predicts blood pressure in mmHg from an intercept, a centered baseline score, and an indicator for a comparison group. A profile $A$ is in the reference group with centered baseline score 0, and profile $B$ is in the comparison group with centered baseline score 2. Their design vectors are $x_A=(1,0,0)^\mathsf{T}$ and $x_B=(1,2,1)^\mathsf{T}$. The fitted means are 50 and 57 mmHg, respectively. Suppose the estimated covariance matrix of the three fitted coefficients is

\widehat{\operatorname{Var}}(\hat{\beta})= \begin{pmatrix} 1&0.2&0.1\\ 0.2&0.5&0.1\\ 0.1&0.1&0.4 \end{pmatrix}.

This matrix is assumed to have come from a correctly specified regression fit with $n=25$ observations and $p=3$ coefficients. Assume the profiles are supported by the observed predictor range and diagnostic checks are reasonably consistent with independent, constant-variance, approximately normal errors.

For $B-A$, the fitted difference is $57-50=7$ mmHg. The variance for profile $A$ is 1 mmHg squared. For $B$, multiplying its design vector through the covariance matrix gives

\widehat{\operatorname{Var}}(\hat{\mu}_B) =1+4(0.5)+0.4+2(2)(0.2)+2(1)(0.1)+2(2)(1)(0.1) =4.8\text{ mmHg}^2.

The covariance is $x_A^\mathsf{T}\widehat{\operatorname{Var}}(\hat{\beta})x_B=1+2(0.2)+0.1=1.5\text{ mmHg}^2$. Thus,

\widehat{\operatorname{Var}}(\hat{\Delta})=4.8+1-2(1.5)=2.8\text{ mmHg}^2, \qquad SE(\hat{\Delta})=\sqrt{2.8}=1.6733\text{ mmHg}.

Here $df=25-3=22$. With $t_{0.975,22}\approx2.074$, a 95% confidence interval is $7\pm2.074\sqrt{2.8}$, approximately $(3.530,10.47)$ mmHg. For these specified profiles, the comparison-group profile has a higher fitted mean blood pressure; the interval quantifies uncertainty in that difference, not in an individual’s blood pressure.

A Practical Calculation Sequence

For a reported difference, make the profile definitions and subtraction direction explicit first. Then use the same fitted model for both predictions and calculate their covariance along with their variances. The sequence below keeps the target and its uncertainty aligned.

Specify the contrast.
Write the target as profile B minus profile A, and identify every predictor value or category in both profiles.

Calculate the fitted difference.
Subtract the fitted mean for A from the fitted mean for B, keeping the outcome units.

Calculate its standard error.
Use the difference vector and coefficient covariance matrix, or combine both prediction variances and their covariance.

Form and interpret the interval.
Use the appropriate residual degrees of freedom and t critical value, then state the interval in context and in the specified direction.

Common Mistake: Treating Predictions as Independent

A tempting but generally wrong calculation is $SE(\hat{\Delta})=\sqrt{SE(\hat{\mu}_A)^2+SE(\hat{\mu}_B)^2}$, justified by saying that the profiles are different. Different profiles do not make their fitted means independent: both predictions use the same estimated regression coefficients. The correct expression includes $-2\widehat{\operatorname{Cov}}(\hat{\mu}_A,\hat{\mu}_B)$. Check its sign carefully. If covariance is negative, subtracting twice that covariance increases the variance; if it is positive, the variance is reduced.

Another reporting pitfall is to give a positive difference without saying which profile was subtracted from which. Reversing the order changes the estimate and interval signs, though it does not change the standard error. Always name the profiles and state the direction of comparison.

Key takeaway: The uncertainty in a difference between fitted means depends on both prediction variances and their covariance. Define the subtraction direction, calculate the covariance from the shared model fit, and interpret the resulting interval as uncertainty about a population difference in conditional means.

Check Your Understanding

Answer each question using the stated difference direction.

Why is the variance of a difference between fitted means not generally just the sum of their variances?
If two fitted means have variances 4 and 9 and covariance 1.5, calculate the variance and standard error of their difference.
If the covariance between two fitted means is negative, does it increase or decrease the variance of their difference? Explain using the formula.
A 95% confidence interval for profile B minus profile A is $(-4,-1)$ outcome units. What does its sign indicate about the fitted population means for these profiles?
Explain why a confidence interval for a difference in fitted means is not an interval for the difference between two future individual outcomes.