Gaussian Errors and Exact Prediction-Interval Coverage

Why the Assumptions Matter

A regression prediction interval is intended to cover a future individual outcome, not just the mean outcome at its predictor values. Its usual t-based formula accounts for both the uncertainty in the fitted mean and the random error in the future outcome. The formula has an especially useful property under a specified model: with the design held fixed, its coverage is exactly the stated level in finite samples—not merely an approximation for a large sample.

That guarantee depends on more than having a fitted regression line and a residual standard deviation. The errors must satisfy particular conditions, including normality, independence, and a common variance. Here we state those conditions and show how they produce the t distribution used by the interval.

The Fixed-Design Model and Its Conditions

Definition: In the fixed-design linear model, the observed response vector is \(Y=X\beta+\varepsilon\). The design matrix \(X\), containing the predictor values and any specified model terms, is treated as fixed. It has \(n\) rows and \(p\) linearly independent columns. The error vector has independent components, each distributed as \(N(0,\sigma^2)\). Thus, conditional on \(X\), the responses have the stated linear mean and a common error variance \(\sigma^2\).

For a new case, let \(x_0\) be its fixed predictor vector, including the intercept entry if the model has an intercept. Its future outcome is \(Y_0=x_0^\mathsf{T}\beta+\varepsilon_0\). For the usual exact prediction interval, the future error \(\varepsilon_0\) must be independent of the training errors and also distributed as \(N(0,\sigma^2)\). The predictor vector \(x_0\) is fixed for the calculation.

The conditions therefore apply to the training data and to the future case. It is not enough for the training errors alone to be Gaussian with constant variance if the future outcome comes from a different error distribution or variance. In this tutorial, “exact” refers to the model-based probability statement with \(X\) and \(x_0\) held fixed.

Constructing the Prediction Interval

Under the model, the least-squares coefficient estimate is \(\hat{\beta}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}Y\), and the fitted mean at the new predictor vector is \(\hat{Y}_0=x_0^\mathsf{T}\hat{\beta}\). Define the leverage quantity for that prediction as \(h_0=x_0^\mathsf{T}(X^\mathsf{T}X)^{-1}x_0\). The residual standard deviation is \(s=\sqrt{\mathrm{SSE}/(n-p)}\), where \(\mathrm{SSE}\) is the sum of squared training residuals.

Theorem: Under the fixed-design model above, with an independent future error distributed as \(N(0,\sigma^2)\), the usual two-sided \(100(1-\alpha)\%\) prediction interval for \(Y_0\) is

\hat{Y}_0 \mathbin{\pm} t_{1-\alpha/2,\,n-p}\,s\sqrt{1+h_0}.

Here \(t_{1-\alpha/2,\,n-p}\) is the \(1-\alpha/2\) quantile of the t distribution with \(n-p\) degrees of freedom. Under these conditions, the probability that this interval contains \(Y_0\), over the training errors and the future error with \(X\) and \(x_0\) fixed, is \(1-\alpha\).

Why the t Distribution Appears

The prediction error is the difference between the future outcome and its fitted mean. It includes the future error \(\varepsilon_0\) and the estimation error in \(\hat{\beta}\). Their contributions are independent under the stated conditions.

Y_0-\hat{Y}_0 =\varepsilon_0-x_0^\mathsf{T}(\hat{\beta}-\beta).

Because the training errors are independent normal variables, \(\hat{\beta}-\beta\) is normal with covariance matrix \(\sigma^2(X^\mathsf{T}X)^{-1}\). The future error is independent of that estimate and has variance \(\sigma^2\). Consequently, the prediction error is normal with mean zero and variance \(\sigma^2(1+h_0)\). This gives the numerator of the standardized prediction error.

The residual sum of squares comes from the part of the training response orthogonal to the columns of \(X\). Under Gaussian errors, this orthogonal component is independent of the fitted coefficient estimates, and \(\mathrm{SSE}/\sigma^2\) has a chi-square distribution with \(n-p\) degrees of freedom. The future error is independent of both training components as well. These facts give the denominator needed for a t statistic.

T=\frac{Y_0-\hat{Y}_0}{s\sqrt{1+h_0}} = \frac{(Y_0-\hat{Y}_0)/\{\sigma\sqrt{1+h_0}\}} {\sqrt{\{(n-p)s^2/\sigma^2\}/(n-p)}} \sim t_{n-p}.

The numerator in the final expression is standard normal. The quantity \((n-p)s^2/\sigma^2\) is chi-square with \(n-p\) degrees of freedom and is independent of that numerator. By the definition of the t distribution, their ratio has a \(t_{n-p}\) distribution. Therefore,

P\!\left(-t_{1-\alpha/2,\,n-p}\leq T\leq t_{1-\alpha/2,\,n-p}\right)=1-\alpha.

Rearranging the two inequalities for \(Y_0\) gives the stated interval. This derivation also shows why the exact result is finite-sample: it uses the exact normal and chi-square distributions, not a large-sample approximation.

What Each Assumption Contributes

Condition	Role in the exact result	What can go wrong if it fails
Correct linear mean model and fixed \(X,x_0\)	Defines the target mean and the prediction error being calibrated.	A misspecified mean can shift the prediction error away from zero.
Normal errors	Makes the prediction error normal and gives the exact chi-square result for residual variation.	The standardized prediction error need not have a t distribution in a finite sample.
Independence of training errors and of the future error from training errors	Supports the variance addition and independence between the prediction error and residual variance estimate.	Covariances may change the prediction-error variance, and the t ratio need not have the stated distribution.
Constant variance \(\sigma^2\)	Gives training and future errors the common scale used by \(s\) and by the variance formula \(\sigma^2(1+h_0)\).	The residual variance estimate may not represent the future error variance.

Normality is what makes the small-sample distributional statements exact. Independence and constant variance are also substantive: without them, the numerator’s variance or its relationship with \(s\) may differ from what the formula assumes. If the errors are not normal but other conditions are reasonable, the same interval may sometimes be used as an approximation, especially in larger samples; that is not the exact finite-sample result proved here.

Worked Example: A Simple-Regression Prediction

Suppose hypothetical training data use the fixed predictor values \(0,1,2,3,4,5\), with an intercept and a linear term. Assume the model conditions hold, the fitted line predicts 12 units at \(x_0=3\), and the residual standard deviation is \(s=2\) units. Construct a 95% prediction interval.

There are \(n=6\) observations and \(p=2\) fitted coefficients, so \(df=n-p=4\). The predictor mean is \(\bar{x}=2.5\), and \(S_{xx}=\sum (x_i-\bar{x})^2=17.5\). In simple regression, the prediction leverage is \(h_0=1/n+(x_0-\bar{x})^2/S_{xx}\), giving

h_0=\frac{1}{6}+\frac{(3-2.5)^2}{17.5} =0.1667+0.01429 =0.18095.

The 95% critical value is \(t_{0.975,4}\approx2.776\). The prediction standard error is \(s\sqrt{1+h_0}=2\sqrt{1.18095}\approx2.173\) units. Thus the half-width is \(2.776(2.173)\approx6.034\) units, and the interval is \(12\pm6.034\), or approximately \([5.966,18.03]\) units. Its exact 95% coverage follows only if the specified Gaussian, independent, constant-variance model conditions hold for these data and the future outcome.

Worked Example: A Very Small Sample

In a hypothetical intercept-only model, two training outcomes are 9 and 11 units. Assume independent normal errors with a common variance, and suppose the future outcome is independent and follows the same model. Find the 95% prediction interval for a new outcome.

The fitted mean is \(\hat{Y}_0=\bar{y}=10\) units. With \(n=2\) and \(p=1\), the residual degrees of freedom are 1. The sample standard deviation is \(s=\sqrt{\{(9-10)^2+(11-10)^2\}/(2-1)}=\sqrt{2}\) units. For an intercept-only model, \(h_0=1/n=0.5\), so the prediction standard error is

s\sqrt{1+h_0}=\sqrt{2}\sqrt{1.5}=\sqrt{3}\approx1.732\text{ units}.

The 95% critical value is \(t_{0.975,1}\approx12.706\). The half-width is \(12.706\sqrt{2}\sqrt{1.5}=12.706\sqrt{3}\approx22.01\) units. Therefore, the interval is \(10\pm22.01\), or approximately \([-12.01,32.01]\) units. The unusually large width reflects the extreme uncertainty from estimating variance with only one residual degree of freedom; it does not contradict exact coverage under the model.

Worked Example: A Future Outcome With Greater Variance

Suppose a hypothetical intercept-only training model has \(n=5\), and its errors are independent \(N(0,\sigma^2)\). Now suppose the future error is independent but distributed as \(N(0,4\sigma^2)\), so its standard deviation is twice the training error standard deviation. Work out how the prediction-error variance differs from what the usual interval assumes.

For an intercept-only model, \(h_0=1/n=0.2\). The usual formula assumes that the future error variance is \(\sigma^2\), so it uses prediction-error variance \(\sigma^2(1+0.2)=1.2\sigma^2\). Under the hypothetical future variance instead, the actual prediction-error variance is the sum of \(4\sigma^2\) for the future error and \(0.2\sigma^2\) for estimating the mean:

\operatorname{Var}(Y_0-\hat{Y}_0)=4\sigma^2+0.2\sigma^2=4.2\sigma^2.

The actual variance is \(4.2/1.2=3.5\) times the variance assumed by the usual formula; the corresponding standard-deviation ratio is \(\sqrt{3.5}\approx1.871\). The training residuals estimate the training variance, not the larger future variance. Thus the usual t pivot does not apply, and its nominal level is not guaranteed to be the actual coverage.

Common Mistake and Practical Meaning

A common mistake is to say, “The sample is large, so the usual interval has exact coverage even if the errors are not normal.” Large samples can sometimes support an approximation, but they do not make the finite-sample t result exact. Exactness here follows from the specified distributions and independence conditions, including a common variance for training and future errors. A different error distribution or a different future variance changes the distributional argument.

The distinction matters when a prediction interval is used to communicate uncertainty for an individual future outcome. Its width is not justified by the fitted regression alone: the interpretation depends on how well the model’s assumptions describe the training observations and the future case. These assumptions do not establish that a predictor causes an outcome; they establish a probability model for the prediction.

Key takeaway: The usual t prediction interval has exact finite-sample coverage when the fixed-design linear mean is correctly specified, training errors are independent \(N(0,\sigma^2)\), and the future error is independent of them with the same distribution. Normality supplies the exact t calibration; independence and constant variance support the variance and independence calculations behind it.

Check Your Understanding

Use the fixed-design Gaussian model and the prediction interval derivation to answer these questions.

In the usual prediction interval, what two sources of uncertainty are represented by the factor \(\sqrt{1+h_0}\)?
Why does the exact t result require more than normal training errors if the future outcome has a different variance?
For a model with \(n=20\) observations and \(p=4\) fitted coefficients, what are the residual degrees of freedom?
Which distributional step in the derivation relies on Gaussian errors?
If the future error is dependent on the training errors, which parts of the variance and t-pivot argument may fail?