What Does “95% Coverage” Refer To?
A prediction interval is intended to contain a future individual outcome at a specified predictor value. Its nominal confidence level describes how often the interval-producing procedure succeeds over repeated use under its assumptions. Understanding that repeated-sampling statement prevents a common confusion: a 95% prediction interval does not, in the usual frequentist interpretation, mean there is a 95% probability that the one interval already calculated contains the one future outcome.
To make the coverage statement precise, consider simple linear regression with predictor values held fixed. Suppose the training responses follow a linear mean model with independent Normal errors having mean zero and common variance. A future response at a prespecified value \(x_0\) follows the same model, with an error independent of the training errors. Under these assumptions, the usual \(t\)-based prediction interval has exact 95% coverage when calibrated at the 95% level. The regression coefficients and error variance are fixed unknown quantities; the training responses and future error vary across repetitions.
In each repetition, the procedure produces a new interval from a new training sample, and a new future outcome is generated for the target predictor value. The event counted as a success is simple: the future outcome falls between the interval’s lower and upper limits. The predictor value \(x_0\) and, in the fixed-design interpretation, the training predictor values are held fixed. It is the responses and errors that vary.
The interval is random before the training responses are observed because its fitted line, residual standard deviation, and endpoints depend on those responses. The future outcome is also random before it is observed. The coverage probability describes the joint repetition of these elements under the stated model. Once the training data have been observed, the interval endpoints are fixed; once the future outcome has been observed, whether it was covered is also fixed.
The Regression Procedure Behind the Statement
For a training sample of size \(n\), let \(\bar{x}\) be the mean of the predictor values, \(S_{xx}=\sum_{i=1}^{n}(x_i-\bar{x})^2\), and \(s\) the residual standard deviation from the fitted simple regression. At a target predictor value \(x_0\), the fitted response is \(\hat{y}_0\). The usual two-sided prediction interval is
Here \(t_{0.975,\,n-2}\) is the 97.5th percentile of a \(t\) distribution with \(n-2\) degrees of freedom. The “1” inside the square root accounts for the error in a new individual outcome; the other terms account for uncertainty in estimating the regression mean at \(x_0\). The formula is used for the exact nominal coverage statement only when the model assumptions supporting its \(t\)-calibration hold: a linear mean, independent Normal errors with common variance, and a future error independent of the training errors and drawn from that same error distribution. In the examples, those assumptions are stipulated as part of the hypothetical setup; a small table of observed values cannot establish them by itself.
For a two-sided interval with nominal level 95%, the repeated-sampling event is
The Gaussian-error prediction-interval result says that this event has probability 0.95 over repetitions under the stated conditions. It does not say every interval will contain its future outcome, nor that exactly 95 out of every 100 intervals will succeed in a finite collection.
Worked Example: Calculating an Interval Before Interpreting Its Coverage
Suppose a hypothetical training data set has predictor values \(-2,-1,0,1,2\) and responses \(7,6,12,10,15\), respectively. A researcher wants to predict a new response when \(x_0=0\). For this illustration, assume the simple linear regression model has independent Normal errors with mean zero and common variance, and that the future case has an independent error from the same distribution. These assumptions justify the exact \(t\)-based calibration; the five observations alone do not verify them.
The sample means and predictor sum of squares are \(\bar{x}=0\) and \(S_{xx}=10\). The fitted line is \(\hat{y}=10+2x\), so \(\hat{y}_0=10\). Its fitted responses are \(6,8,10,12,14\), giving residuals \(1,-2,2,-2,1\). Thus the residual sum of squares is \(1^2+(-2)^2+2^2+(-2)^2+1^2=14\), and
With \(n-2=3\) degrees of freedom, \(t_{0.975,3}\approx3.182\). The estimated standard error for predicting one new response at \(x_0=0\) is
The margin of error is \(3.182(2.366)\approx7.529\), so the interval is
This interval predicts a new individual response at \(x=0\), in the response’s measurement units. Its nominal 95% interpretation is about the procedure that produced these endpoints: if the same fixed predictor design and target \(x_0=0\) were used repeatedly, and new training responses and future outcomes were generated under the stated model each time, approximately 95% of the resulting intervals would contain their corresponding future responses. It is not a statement that this particular interval has a 95% chance of covering its future response.
Why the Long-Run Rate Is Not a Finite-Sample Promise
Repeated-sampling coverage is a probability for the procedure. To see how it relates to a finite run, let \(C_r=1\) when the interval in repetition \(r\) covers its future outcome, and \(C_r=0\) otherwise. If repetitions are independent and each has coverage 0.95, then each \(C_r\) has probability 0.95 of being 1. For \(R\) repetitions, the observed coverage proportion is the average of these indicators:
The expected proportion is 0.95, but the observed proportion varies from run to run. Its standard error decreases as the number of repetitions increases. This is why “95% coverage” means a long-run rate, not a promise that every batch of 100 intervals will contain exactly 95 future outcomes.
Worked Example: Interpreting Coverage in 1,000 Repetitions
Imagine a hypothetical exercise that repeats the same interval procedure 1,000 times under the model assumptions, using the same predictor design and target value in every repetition. Each repetition generates a training sample, calculates a nominal 95% interval, and then generates the corresponding future response.
Suppose 947 of the 1,000 future responses fall inside their intervals. The observed coverage proportion is
This observed rate is close to the nominal 95%, but it is not required to equal it. Under the stated independent-repetition setup, the standard error of the proportion is
or about 0.689 percentage points. The difference between 94.7% and 95.0% is 0.3 percentage points, a small variation relative to that standard error. The 947 successes do not change the nominal level of the procedure; they are one finite-run realization of a procedure whose coverage probability is 0.95 under the model.
What Can Be Said About One Computed Interval?
After the training data are observed, the interval’s limits are fixed. The frequentist coverage statement still describes the performance of the procedure across repetitions, but it is not converted into a probability statement about those particular fixed limits. In frequentist regression, the unknown coefficients and error variance are treated as fixed, not as quantities assigned probability distributions by the interval calculation.
Before a future response is observed, it is reasonable to say that the outcome is uncertain under the model. But the exact 95% coverage result averages over the varying training samples as well as the future errors. It does not generally assert that, conditional on this one realized training sample, the future response has probability exactly 0.95 of falling in the interval. That conditional predictive probability is not what the frequentist coverage guarantee supplies.
After the future outcome is observed, the coverage event has a definite answer: it either fell inside the interval or it did not. There is no remaining uncertainty about that event. A miss does not prove the procedure was incorrectly calibrated; even a correctly calibrated 95% procedure misses in some repetitions. Conversely, one successful prediction does not establish that a procedure has 95% coverage.
Worked Example: One Interval, Two Possible Outcomes
Use the interval \((2.471,\ 17.529)\) calculated above. Consider two hypothetical future responses at \(x=0\), each measured in the same units as the training responses. Suppose the first is \(16.0\) and the second is \(18.2\).
For \(16.0\), the coverage check is \(2.471\leq16.0\leq17.529\), which is true. The outcome is covered. For \(18.2\), the check is \(2.471\leq18.2\leq17.529\), which is false because \(18.2>17.529\). That outcome is not covered.
For either observed outcome, the interval itself is the same fixed interval and the coverage indicator is now known. Neither result can be described as having a 95% probability of coverage: the first event occurred and the second did not. The 95% figure describes how often the interval-producing procedure covers across repeated training samples and future outcomes under the assumptions.
A Short Justification of the Long-Run Interpretation
The exact prediction-interval result under Gaussian errors states that, for a single repetition under the specified regression model, the probability of coverage is 0.95. This gives each coverage indicator \(C_r\) an expected value of 0.95, because \(C_r\) equals 1 on coverage and 0 otherwise:
Averaging \(R\) such indicators gives \(E(\hat{c}_R)=0.95\). When repetitions are independent, the law of large numbers says that as \(R\) grows, the observed average \(\hat{c}_R\) tends to be close to 0.95. This justifies the phrase “95% in the long run.” It does not require the proportion to be exactly 0.95 in any particular finite run.
This explanation also makes clear why the repeated-sampling plan matters. If the regression model is wrong, the errors are not distributed as assumed, or the future cases do not come from the specified process, the stated coverage result need not hold. The nominal level alone cannot guarantee real-world coverage independently of the assumptions.
Common Mistake
Wrong reasoning: “I have calculated this 95% prediction interval, so there is a 95% probability that the future response will be inside these particular limits.”
Correction: The 95% guarantee refers to the interval procedure repeated over training samples and future outcomes under its assumptions. For the realized interval, the frequentist statement is not a 95% probability claim about its fixed endpoints. Once the future response is observed, coverage is simply true or false. Keep the target clear: the nominal level describes the procedure’s long-run performance, not certainty about one particular prediction.
Check Your Understanding
Answer each question using the distinction between a repeated procedure and one realized interval.
- In repeated-sampling coverage for prediction at a fixed \(x_0\), which quantities vary across repetitions, and which predictor value is held fixed?
- A nominal 95% procedure covers 92 of 100 future outcomes in one hypothetical run. Does this alone contradict 95% coverage? Explain.
- What does the frequentist 95% coverage statement say about one interval’s fixed endpoints after its training sample has been observed?
- For a future response observed above an interval’s upper endpoint, is its coverage status still uncertain? Why or why not?
- Name two model conditions needed for the exact Gaussian-error \(t\)-based coverage result described here.