Prediction Intervals for Linear Combinations of Future Outcomes

Why a Weighted Prediction Target Needs Its Own Variance

Several future responses may be reported together as a sum, average, difference, or another weighted combination. Predicting each response separately and combining the intervals does not generally give an exact interval for that target. Instead, calculate the prediction error of the weighted combination directly. Its variance includes the future observation errors and uncertainty in the estimated regression mean; the latter can make different prediction errors correlated.

Assume the fixed-design normal linear model for the observed responses, with a specified set of future design points. The future responses are independent of the observed sample, and their errors are independent of one another and have the same variance as the observed errors. The weights are fixed, not chosen after observing the future outcomes.

Definition: Let $Y$ be the $n$-vector of observed responses, $X$ the $n \times p$ design matrix of full column rank, and suppose $Y=X\beta+\varepsilon$, where $\varepsilon\sim N(0,\sigma^2 I_n)$. Let $X_f$ be the $m \times p$ matrix whose rows are the specified future design vectors. Write $Y_f=X_f\beta+\varepsilon_f$, where $\varepsilon_f\sim N(0,\sigma^2 I_m)$ is independent of $\varepsilon$. For a fixed weight vector $w\in\mathbb R^m$, the target is $w^\mathsf{T}Y_f$, and its predictor is $w^\mathsf{T}X_f\hat\beta$, where $\hat\beta=(X^\mathsf{T}X)^{-1}X^\mathsf{T}Y$.

The target is a future random quantity, not merely the weighted mean response $w^\mathsf{T}X_f\beta$. Consequently, its prediction error includes the contribution $w^\mathsf{T}\varepsilon_f$. Define the residual variance estimate by $s^2=\mathrm{RSS}/(n-p)$, where $\mathrm{RSS}=(Y-X\hat\beta)^\mathsf{T}(Y-X\hat\beta)$.

Deriving the Prediction-Error Variance

Subtracting the predictor from the future target gives

w^\mathsf{T}Y_f-w^\mathsf{T}X_f\hat\beta = w^\mathsf{T}\varepsilon_f-w^\mathsf{T}X_f(\hat\beta-\beta).

The two terms on the right are independent and have mean zero. The first has variance $\sigma^2 w^\mathsf{T}w$. Since $\operatorname{Var}(\hat\beta)=\sigma^2(X^\mathsf{T}X)^{-1}$, the second has variance $\sigma^2 w^\mathsf{T}X_f(X^\mathsf{T}X)^{-1}X_f^\mathsf{T}w$. Thus the prediction-error variance is $\sigma^2 q(w)$, where

q(w) = w^\mathsf{T}w+ w^\mathsf{T}X_f(X^\mathsf{T}X)^{-1}X_f^\mathsf{T}w.

The first term accounts for future error; the second accounts for estimating the regression mean. Equivalently, the full vector of prediction errors has covariance matrix

\operatorname{Var}(Y_f-X_f\hat\beta) = \sigma^2\left[I_m+X_f(X^\mathsf{T}X)^{-1}X_f^\mathsf{T}\right].

The off-diagonal entries need not be zero: prediction errors at different design points share uncertainty from the same $\hat\beta$. The weighted variance formula uses all entries of this matrix, not just its diagonal.

Theorem: Under the model assumptions above, with $n>p$ and $w\ne 0$, the exact $1-\alpha$ prediction interval for $w^\mathsf{T}Y_f$ is $$ w^\mathsf{T}X_f\hat\beta \ \pm\ t_{n-p,\,1-\alpha/2}\,s\sqrt{q(w)}, $$ where $t_{\nu,\,1-\alpha/2}$ is the $1-\alpha/2$ quantile of a Student $t$ distribution with $\nu$ degrees of freedom.

Proof. The prediction error is a linear combination of jointly normal errors, so it is normal with mean zero and variance $\sigma^2q(w)$. The least-squares residual is the projection of $\varepsilon$ onto the subspace orthogonal to the column space of $X$. Choose an orthonormal basis whose first $p$ vectors span that column space. In this basis, the coordinates of $\varepsilon/\sigma$ are independent standard normal variables. The fitted coefficient error depends on the first $p$ coordinates, while $\mathrm{RSS}/\sigma^2$ is the sum of squares of the remaining $n-p$ coordinates. The future error $\varepsilon_f$ is independent of all these coordinates. Therefore the prediction error is independent of $s^2$, and

\frac{\mathrm{RSS}}{\sigma^2}\sim\chi^2_{n-p}, \qquad \frac{w^\mathsf{T}Y_f-w^\mathsf{T}X_f\hat\beta}{\sigma\sqrt{q(w)}}\sim N(0,1).

By the definition of the Student $t$ distribution as an independent standard normal divided by the square root of an independent chi-square variable divided by its degrees of freedom,

\frac{w^\mathsf{T}Y_f-w^\mathsf{T}X_f\hat\beta}{s\sqrt{q(w)}}\sim t_{n-p}.

The central $1-\alpha$ probability statement for this pivot rearranges to the stated interval. $\square$

Worked Calculations

Worked Example: An Average of Two Future Outcomes

Suppose the observed model has an intercept only, with $n=4$, and observed responses $2,4,4,6$. The fitted mean is $\hat\beta=4$, and

$\mathrm{RSS}=(2-4)^2+(4-4)^2+(4-4)^2+(6-4)^2=8$, so $s^2=8/(4-1)=8/3$. Predict the average of two future responses at the same design point, using $w=(1/2,1/2)^\mathsf{T}$.

Here $X_f$ is a two-row column of ones, and $(X^\mathsf{T}X)^{-1}=1/4$. The predicted average is $w^\mathsf{T}X_f\hat\beta=4$. The future-error contribution is $w^\mathsf{T}w=1/2$, and the estimation contribution is

$w^\mathsf{T}X_f(X^\mathsf{T}X)^{-1}X_f^\mathsf{T}w=(w^\mathsf{T}X_f)^2/4=1/4$.

Thus $q(w)=3/4$, and the estimated standard error is

$s\sqrt{q(w)}=\sqrt{(8/3)(3/4)}=\sqrt{2}$.

For a 95% interval, $t_{3,0.975}\approx3.182$, giving $4\pm3.182\sqrt{2}$, or approximately $(-0.50,8.50)$. The interval predicts the average of two future observations, not the intercept alone.

Worked Example: Covariance Between Two Prediction Errors

In the same intercept-only setting, consider the two individual future responses, again at the same design point. The covariance matrix of their prediction errors is

\sigma^2 \left[ I_2+ \begin{pmatrix}1\\1\end{pmatrix} \frac{1}{4} \begin{pmatrix}1&1\end{pmatrix} \right] = \sigma^2 \begin{pmatrix} 5/4&1/4\\ 1/4&5/4 \end{pmatrix}.

Each prediction error has variance $5\sigma^2/4$, but their covariance is $\sigma^2/4$: the same estimated intercept contributes to both. For the average, the variance is therefore

$\frac14(5\sigma^2/4)+\frac14(5\sigma^2/4)+2\cdot\frac14\cdot\frac14\sigma^2=3\sigma^2/4$.

This agrees with $q(w)\sigma^2$ from the previous calculation. Omitting the covariance would give $5\sigma^2/8$, which is too small.

Worked Example: A Weighted Difference at Two Design Points

Consider a simple linear regression with an intercept and slope, using design points $x=-2,-1,0,1,2$. Suppose the observed responses are $0,-1,5,3,8$. The fitted line is $\hat y=3+2x$: the residuals are $1,-2,2,-2,1$, whose sum and product-sum with $x$ are both zero. Their squared sum is $14$, so $s^2=14/(5-2)=14/3$. Predict the difference between future responses at $x=0$ and $x=2$, in that order.

The future rows are $(1,0)$ and $(1,2)$, and $w=(1,-1)^\mathsf{T}$. The fitted difference is $3-7=-4$. Since $X^\mathsf{T}X$ is diagonal with entries $5$ and $10$, the future-error term is $w^\mathsf{T}w=2$, while $X_f^\mathsf{T}w=(0,-2)^\mathsf{T}$. Therefore

$q(w)=2+(0,-2)\begin{pmatrix}1/5&0\\0&1/10\end{pmatrix}(0,-2)^\mathsf{T}=2+4/10=12/5$.

The estimated standard error is $\sqrt{(14/3)(12/5)}=\sqrt{56/5}\approx3.347$. With $3$ degrees of freedom, the 95% interval is $-4\pm3.182(3.347)$, approximately $(-14.65,6.65)$. Although the weights sum to zero, the slope-estimation contribution remains because the two future design points differ.

Common Pitfall and Interpretation

A tempting but incorrect approach is to calculate the prediction-error variances for the future responses separately, multiply them by the squared weights, and add. That formula is valid only when the prediction errors have zero covariance. In regression, fitted-mean uncertainty is shared across future outcomes, so this covariance generally must be included. The matrix formula makes the correction explicit; the scalar formula $q(w)$ is a concise way to include it.

The weights also determine what uncertainty remains. If the weights cancel a shared intercept contribution, the fitted-mean part can shrink or disappear, but the future-error component $w^\mathsf{T}w$ remains for nonzero weights and independent future errors. A weighted sum is not automatically a mean: its units and scale depend on the chosen weights. Multiplying every weight by a constant multiplies both the target and the interval endpoints by that constant in magnitude.

Key takeaway: For a weighted sum of future outcomes, use its own prediction-error variance, $\sigma^2w^\mathsf{T}[I_m+X_f(X^\mathsf{T}X)^{-1}X_f^\mathsf{T}]w$. The off-diagonal covariance terms are part of the exact interval whenever the future prediction errors are correlated through the fitted regression mean.

Check Your Understanding

Check that you can identify the target, its variance contributions, and the role of covariance.

In the prediction-error variance, what does $w^\mathsf{T}w$ represent, and what does the design-matrix term represent?
Why can prediction errors at two different future design points have nonzero covariance even when the future observation errors are independent?
For an intercept-only model with $n$ observed responses, what is the estimation contribution to the variance of the average of $m$ future responses at the same design point?
What distribution does the standardized prediction error have when $\sigma^2$ is estimated, and what are its degrees of freedom?
Why is adding separately calculated, weighted prediction variances generally incorrect for a weighted sum?

Prediction Intervals for Linear Combinations of Future Outcomes

What You'll Learn