Deriving the Prediction Variance at a New Design Point

What Varies When We Predict a New Outcome?

A fitted regression model gives a predicted mean response at a specified design point. Predicting an individual future outcome involves an additional source of variation: even if the mean at that point were known exactly, a new outcome would still differ from the mean because of its random error. The prediction-error variance keeps these two sources distinct.

We work conditionally on the design matrix. Let $X$ be a fixed $n\times p$ matrix with full column rank, let $\beta$ be a $p$-vector, and suppose the observed response vector follows the normal linear model. At a specified new design vector $x_0$, the future response is generated by the same model and has an error independent of the errors in the observed responses.

Model and target: Assume $$ Y=X\beta+\varepsilon,\qquad \varepsilon\sim N(0,\sigma^2 I_n), $$ where $\sigma^2>0$ and $X$ has full column rank. For a fixed $x_0\in\mathbb R^p$, let $$ Y_0=x_0^T\beta+\varepsilon_0,\qquad \varepsilon_0\sim N(0,\sigma^2), $$ with $\varepsilon_0$ independent of $\varepsilon$. The least-squares estimator is $\hat\beta=(X^TX)^{-1}X^TY$, and the prediction error for the new response is $Y_0-x_0^T\hat\beta$.

The full-rank assumption ensures that $X^TX$ is invertible. The common variance $\sigma^2$, independent errors, and normal distributions are assumptions of this exact normal-theory setting. We will derive the variance conditional on $X$ and $x_0$; thus these quantities are treated as fixed throughout the calculation.

Deriving the Two Contributions

First, the fitted mean at $x_0$ is $x_0^T\hat\beta$, while the true mean response there is $x_0^T\beta$. Their difference depends only on the errors in the original sample. Substituting $Y=X\beta+\varepsilon$ into the estimator gives

\hat\beta-\beta=(X^TX)^{-1}X^T\varepsilon.

The future response adds a separate error $\varepsilon_0$. Therefore the prediction error can be written as

Y_0-x_0^T\hat\beta =\varepsilon_0-x_0^T(\hat\beta-\beta).

The minus sign does not affect the variance of the fitted-mean term, but it is important when writing the error decomposition. The two terms are independent because the future error is independent of the entire observed error vector.

Theorem: Under the model assumptions above, $$ \operatorname{Var}(Y_0-x_0^T\hat\beta\mid X,x_0) =\sigma^2\left[1+x_0^T(X^TX)^{-1}x_0\right]. $$ The first term, $\sigma^2$, is the variance of the new observation’s own error. The second term, $\sigma^2x_0^T(X^TX)^{-1}x_0$, is the variance of the estimated mean at $x_0$.

To prove the result, we first obtain the covariance matrix of $\hat\beta$. For any fixed matrix $A$ and random vector $Z$, $\operatorname{Cov}(AZ)=A\operatorname{Cov}(Z)A^T$. Applying this to the displayed expression for $\hat\beta-\beta$ yields

\begin{aligned} \operatorname{Cov}(\hat\beta) &=(X^TX)^{-1}X^T\operatorname{Cov}(\varepsilon) X(X^TX)^{-1}\\ &=\sigma^2(X^TX)^{-1}X^TX(X^TX)^{-1}\\ &=\sigma^2(X^TX)^{-1}. \end{aligned}

Consequently,

\operatorname{Var}\{x_0^T(\hat\beta-\beta)\} =x_0^T\operatorname{Cov}(\hat\beta)x_0 =\sigma^2x_0^T(X^TX)^{-1}x_0.

Now take the variance of the prediction-error decomposition. Independence makes the covariance between $\varepsilon_0$ and $x_0^T(\hat\beta-\beta)$ zero. Thus

\begin{aligned} \operatorname{Var}(Y_0-x_0^T\hat\beta\mid X,x_0) &=\operatorname{Var}(\varepsilon_0) +\operatorname{Var}\{x_0^T(\hat\beta-\beta)\}\\ &=\sigma^2+\sigma^2x_0^T(X^TX)^{-1}x_0\\ &=\sigma^2\left[1+x_0^T(X^TX)^{-1}x_0\right]. \end{aligned}

This also establishes that the prediction error has mean zero: both $\varepsilon_0$ and $\hat\beta-\beta$ have mean zero. Hence its mean squared error equals the variance just derived. $\square$

Why the Prediction Error Is Normal

The variance calculation does not require normality; it follows from the covariance assumptions and independence. Normality supplies a further exact distributional statement, useful when constructing normal-theory prediction procedures.

Result: Under the stated normal model and independence assumption, $$ Y_0-x_0^T\hat\beta \sim N\!\left(0,\, \sigma^2\left[1+x_0^T(X^TX)^{-1}x_0\right]\right), $$ conditional on $X$ and $x_0$.

Indeed, $\hat\beta-\beta=(X^TX)^{-1}X^T\varepsilon$ is a linear transformation of a multivariate normal vector, so it is multivariate normal. Thus $x_0^T(\hat\beta-\beta)$ is normal. The error $\varepsilon_0$ is also normal and independent of that term. The difference of independent normal random variables is normal; its mean and variance are the values calculated above. This proves the result.

Worked Design-Matrix Calculations

Worked Example: An Intercept-Only Model

Suppose $n=4$ observations follow an intercept-only model, so $X$ is the $4\times1$ vector of ones. Find the prediction-error variance for a new response, taking $\sigma^2=9$.

Here $X^TX=4$, so $(X^TX)^{-1}=1/4$. The new design vector is $x_0=1$. Therefore

x_0^T(X^TX)^{-1}x_0=1\left(\frac14\right)1=\frac14.

The fitted-mean contribution to the variance is $9(1/4)=9/4$. The future response contributes its own error variance, $9$. Altogether,

\operatorname{Var}(Y_0-\hat\beta) =9\left(1+\frac14\right) =\frac{45}{4}=11.25.

The result separates the variability of a new observation around the common mean from the sampling variability of the estimated common mean.

Worked Example: A Linear Trend at a New Design Point

Let the model have an intercept and a linear trend, with observed predictor values $z=-1,0,1,2$. Thus the rows of $X$ are $(1,z)$. Find the prediction-error variance at $z_0=3$ when $\sigma^2=4$.

The required cross-products are $\sum 1=4$, $\sum z=2$, and $\sum z^2=6$, so

X^TX= \begin{pmatrix} 4&2\\ 2&6 \end{pmatrix}, \qquad (X^TX)^{-1} =\frac1{20} \begin{pmatrix} 6&-2\\ -2&4 \end{pmatrix} = \begin{pmatrix} 0.3&-0.1\\ -0.1&0.2 \end{pmatrix}.

At $z_0=3$, the design vector is $x_0=(1,3)^T$. Multiplying explicitly,

\begin{aligned} x_0^T(X^TX)^{-1}x_0 &=(1,3) \begin{pmatrix} 0.3&-0.1\\ -0.1&0.2 \end{pmatrix} \begin{pmatrix}1\\3\end{pmatrix}\\ &=(1,3)\begin{pmatrix}0\\0.5\end{pmatrix} =1.5. \end{aligned}

Thus the fitted-mean contribution is $4(1.5)=6$, and the new observation contributes $4$. The total prediction-error variance is $4(1+1.5)=10$. The calculation uses the specified new design vector; it does not require that this point be one of the observed predictor values.

Worked Example: Predicting a Treatment-Group Outcome

Suppose six observations come from a model with an intercept and a treatment indicator. Three observations have indicator $0$, and three have indicator $1$. For a new treated observation, use $x_0=(1,1)^T$ and suppose $\sigma^2=9$. Calculate the prediction-error variance.

The design matrix has three rows $(1,0)$ and three rows $(1,1)$. Hence

X^TX= \begin{pmatrix} 6&3\\ 3&3 \end{pmatrix}, \qquad \det(X^TX)=6(3)-3(3)=9,

and therefore

(X^TX)^{-1} =\frac19 \begin{pmatrix} 3&-3\\ -3&6 \end{pmatrix} = \begin{pmatrix} 1/3&-1/3\\ -1/3&2/3 \end{pmatrix}.

The design-point factor is

x_0^T(X^TX)^{-1}x_0 =(1,1) \begin{pmatrix} 1/3&-1/3\\ -1/3&2/3 \end{pmatrix} \begin{pmatrix}1\\1\end{pmatrix} =(1,1)\begin{pmatrix}0\\1/3\end{pmatrix} =\frac13.

The fitted-mean contribution is $9(1/3)=3$. The future error contributes $9$, so the prediction-error variance is $9+3=12$, equivalently $9(1+1/3)=12$.

Interpretation and a Common Pitfall

The factor $x_0^T(X^TX)^{-1}x_0$ describes how uncertainty in the estimated coefficients is carried to the particular linear combination $x_0^T\hat\beta$. Its value depends on both the observed design matrix and the new design vector. The multiplier $\sigma^2$ scales the coefficient-estimation contribution, while the separate $\sigma^2$ accounts for the random deviation of the future outcome from its mean.

A common mistake is to use only $\sigma^2x_0^T(X^TX)^{-1}x_0$ as the variance for predicting a future individual response. That expression is the variance of the estimated mean response $x_0^T\hat\beta$ around $x_0^T\beta$; it does not include the future observation’s error. For a future outcome, the prediction error contains both sources, and their variances add because the errors are independent. Conversely, the additional $\sigma^2$ should not be added when the target is only the mean response.

The independence assumption is consequential. If the future error were correlated with the observed errors, the covariance term in the variance of $\varepsilon_0-x_0^T(\hat\beta-\beta)$ would not generally be zero, and the displayed formula would need an additional covariance contribution. The formula also concerns the true variance when $\sigma^2$ is specified; replacing it with an estimate is a separate step in inference.

Key takeaway: A future-response prediction error is the future observation’s error minus the estimation error in its fitted mean. Under independent normal errors, the two components have zero covariance, giving the variance $\sigma^2[1+x_0^T(X^TX)^{-1}x_0]$.

Check Your Understanding

Use the model assumptions and derivation above to answer the following questions.

Which assumption allows the two variance contributions in the prediction error to add without a covariance term?
In an intercept-only model with $n$ observations, what is $x_0^T(X^TX)^{-1}x_0$ for a new observation?
What does $\sigma^2x_0^T(X^TX)^{-1}x_0$ measure, and what additional source of variation is needed for a future individual outcome?
Why does full column rank of $X$ matter for the derivation using $(X^TX)^{-1}$?
Which part of the variance derivation uses normality, and which part requires only the stated covariance and independence assumptions?