Common Errors in Reading Confidence and Prediction Intervals

What an Interval Does—and Does Not—Say

A reported “95% interval” can be misread in several ways. A confidence interval for a mean response and a prediction interval for a future observation may be centered at the same fitted value, but they answer different questions. Neither statement means that, after the interval has been calculated, there is a 95% frequentist probability that a fixed parameter lies inside it. Nor does 95% coverage at one design point automatically mean 95% coverage for a whole collection of points.

We make these distinctions precise in the fixed-design normal linear model. The key ingredients are the target being covered, the randomness over which coverage is calculated, and the residual degrees of freedom used to estimate the error variance.

Two Targets, Two Sources of Uncertainty

A confidence interval for \(\mu_0\) accounts for uncertainty in estimating the model’s mean response. A prediction interval for \(Y_0\) must account for that same estimation uncertainty and the new error \(\varepsilon_0\). Under the stated model, when \(h_0>0\), the standard pointwise intervals are

respectively. Here \(t_{1-\alpha/2,\nu}\) is the \(1-\alpha/2\) quantile of Student’s \(t\) distribution with \(\nu\) degrees of freedom. The additional \(1\) in the prediction standard error represents the variance of the future error, in units of \(\sigma^2\). It is not an adjustment that can be omitted merely because the interval is centered on a fitted mean.

Proof. Let \(P_X=X(X^\top X)^{-1}X^\top\) and \(M_X=I_n-P_X\). The fitted error \(\hat{\beta}-\beta=(X^\top X)^{-1}X^\top\varepsilon\) depends on \(P_X\varepsilon\), whereas the residual vector is \(M_X\varepsilon\). These are jointly normal and have zero covariance because \(P_XM_X=0\); hence they are independent. Also, \(\|M_X\varepsilon\|^2/\sigma^2\sim\chi^2_\nu\), since \(M_X\) is a projection of rank \(\nu\). Thus \(s\) is independent of \(\hat{\beta}\), with \(\nu s^2/\sigma^2\sim\chi^2_\nu\).

For \(h_0>0\), \(\hat{\mu}_0-\mu_0\) is normal with mean zero and variance \(\sigma^2h_0\). Therefore

where \(Z\sim N(0,1)\), \(U\sim\chi^2_\nu\), and \(Z\) and \(U\) are independent. For prediction, \(Y_0-\hat{\mu}_0=\varepsilon_0-(\hat{\mu}_0-\mu_0)\) is normal with mean zero and variance \(\sigma^2(1+h_0)\). It is independent of \(s\): \(\varepsilon_0\) is independent of the training data, and the fitted error is independent of the residual sum of squares. Consequently,

For either pivot, the probability that its absolute value is no larger than \(t_{1-\alpha/2,\nu}\) is \(1-\alpha\). These events are exactly the events that the corresponding target lies in its displayed interval. If \(h_0=0\), positive definiteness of \((X^\top X)^{-1}\) implies \(x_0=0\); hence \(\mu_0=0\) is known and the mean interval is \([0,0]\). The prediction pivot remains valid, with denominator \(s\sqrt{1+h_0}=s\). \(\square\)

Coverage Is a Repeated-Sampling Statement

For a frequentist confidence interval, the parameter \(\mu_0\) is fixed under the model. The interval’s endpoints vary with the training data. Saying that its coverage is 95% means that, over repeated samples generated under the model with the same fixed design, 95% of the intervals contain \(\mu_0\). It does not assign a 95% probability to \(\mu_0\) after a particular interval has been observed.

For a prediction interval, the target \(Y_0\) is itself random. The exact coverage statement above is a repeated-sampling statement over the training data and the independent future error, conditional on \(X\). It does not, by itself, assert that the conditional probability of \(Y_0\) lying in the particular interval is exactly 95% given the realized training data. That conditional probability would require a different inferential framework or additional assumptions.

Pointwise Coverage Is Not Simultaneous Coverage

The intervals above are pointwise: a particular \(x_0\) is specified, and the coverage statement applies to that point. Even if several intervals each have 95% coverage, that does not establish that all their targets are covered together with probability 95%. For example, if two coverage events are independent and each has probability \(0.95\), the probability both occur is \(0.95^2=0.9025\), not \(0.95\). Without independence, the joint probability still cannot be inferred from the separate marginal coverage statements alone.

Therefore, a collection of separate pointwise intervals should not be described as a 95% simultaneous band or region without a method that controls joint coverage. The distinction concerns the probability statement, not just how the intervals are drawn or displayed.

Common Errors and Corrections

• Wrong: “There is a 95% probability that the fixed mean is in this calculated confidence interval.” Correction: The 95% describes the long-run coverage of the interval procedure under repeated sampling.
• Wrong: “A prediction interval covers the mean, so it needs no extra uncertainty term.” Correction: Its target is a future outcome, and its standard error includes the independent future error through \(1+h_0\).
• Wrong: “Every one of several 95% intervals is 95%, so they jointly cover with probability 95%.” Correction: Pointwise coverage does not determine joint coverage.
• Wrong: “The mean interval always has exact \(1-\alpha\) coverage, including when \(h_0=0\).” Correction: At \(h_0=0\), the mean is known to be zero and the singleton interval has coverage one; the usual pivot is undefined.
• Wrong: “Use \(n-1\) degrees of freedom because there are \(n\) observations.” Correction: In this model, use the residual degrees of freedom \(\nu=n-p\).