Why “95% Confidence” Needs a Careful Interpretation
A regression confidence interval gives a range of plausible values for a population coefficient, based on a sample. The phrase “95% confidence” does not mean that there is a 95% probability that the particular interval you calculated contains the coefficient. Once the sample has been observed, that interval either contains the fixed coefficient or it does not. The 95% describes how the interval-producing procedure performs across repeated samples under its assumptions.
This distinction matters because an interval is a result from one sample, while a confidence level describes a method. To interpret an interval well, keep track of which quantities are fixed and which vary: the population coefficient is fixed in the model, while the sample, the estimated coefficient, its standard error, and the resulting interval can vary from sample to sample.
The Regression Interval as a Repeated-Sampling Procedure
Consider a regression coefficient \(\beta\), such as the population slope describing the adjusted association between study time and exam score. In one sample, the estimated coefficient is \(b\), and its estimated standard error is \(SE\). A two-sided \(100(1-\alpha)\%\) t-based confidence interval has the form
Here \(t^*\) is the positive critical value chosen for the confidence level and the model’s residual degrees of freedom. For a 95% interval, \(\alpha=0.05\), so the central probability under the relevant t distribution is 0.95. The exact critical value depends on the degrees of freedom; it is not always 1.96.
For the usual exact t-based interpretation, assume the regression model is correctly specified for the coefficient being considered, the observations have independent errors with mean zero and constant variance, and the errors are normally distributed. The predictor values must also permit the coefficient to be estimated; in particular, the model’s design must have full rank. The interval is interpreted conditional on the observed predictor values when those values are treated as fixed. Under these conditions, the t-based statistic has the stated t distribution. If the conditions do not hold, the nominal confidence level need not equal the actual coverage probability.
The statement is about repeating the sampling and calculation in the same way: keep the model and target coefficient the same, take a new sample under the same sampling conditions, fit the regression, estimate the standard error, and construct the interval using the stated confidence level and degrees of freedom. The resulting intervals differ because the data differ. If the assumptions hold, the long-run fraction that cover \(\beta\) is the advertised fraction.
Why the Coverage Statement Is True
The coverage result follows by rearranging the t-statistic event. Let \(df\) be the residual degrees of freedom for the fitted regression, and let \(t^*\) be the critical value that leaves probability \(\alpha/2\) in each tail of the \(t\) distribution with \(df\) degrees of freedom. Under the assumptions above, the statistic \((b-\beta)/SE\) follows that t distribution. Therefore,
Since \(SE\) is positive, multiplying all parts of the inequality by \(SE\) preserves its direction. Adding \(\beta\) then gives
The event inside this probability is exactly the event that the interval from \(b-t^*SE\) to \(b+t^*SE\) covers \(\beta\). The estimate \(b\) and standard error \(SE\) vary between samples, so the interval endpoints vary too. The coefficient \(\beta\), by contrast, is held fixed in this repeated-sampling argument. This is why it is the procedure that has a coverage probability, not the already calculated interval.
Worked Example: An Interval That Covers the Slope
Suppose a hypothetical regression relates weekly study time to exam score, with score measured in points and study time in hours per week. For illustration, let the fixed population slope be \(\beta=2.00\) score points per additional study hour. In one sample, the estimated slope is \(b=2.40\) points per hour, its standard error is \(SE=0.20\) points per hour, and the appropriate 95% critical value is \(t^*=2.101\).
The interval is
The interval runs from about 1.980 to 2.820 score points per additional study hour. The illustrative value \(\beta=2.00\) lies inside it, so this interval covers the slope. In an actual analysis, the population slope is unknown; this example uses its value only to show what “covers” means. The fact that this interval covers the coefficient does not establish that every interval from the procedure will do so.
Worked Example: An Interval That Misses the Slope
Use the same hypothetical regression setting and fixed slope \(\beta=2.00\) score points per additional study hour. A different sample gives \(b=2.50\), \(SE=0.20\), and \(t^*=2.101\). Its 95% interval is
The lower endpoint, 2.0798, is greater than the illustrative true slope of 2.00, so this interval misses the coefficient. That is not, by itself, a contradiction of the 95% confidence level. The procedure is designed to miss in about 5% of repeated samples under its assumptions. A single miss is possible; the confidence level does not promise that a particular interval will succeed.
What “95% in the Long Run” Looks Like
Imagine repeating a hypothetical study 20 times under the same regression conditions, with the same fixed slope \(\beta=2.00\) score points per hour and the same interval procedure. For simplicity, suppose every repetition has \(SE=0.20\) and uses \(t^*=2.101\). For each repetition, define \(z=(b-\beta)/SE\). Its interval covers \(\beta\) exactly when the standardized estimate is between \(-2.101\) and \(2.101\):
Worked Example: Counting Coverage Across Repeated Samples
Suppose the 20 hypothetical repetitions produce the following standardized differences \(z=(b-\beta)/SE\). The interval covers when \(|z|\leq2.101\).
| Repetitions | Values of \(z\) | Coverage result |
|---|---|---|
| 1–5 | 0.15, -1.32, 2.08, -0.44, 1.75 | All 5 cover |
| 6–10 | -2.00, 0.67, -0.91, 1.09, -1.58 | All 5 cover |
| 11–15 | 0.33, 2.24, -0.27, 1.96, -1.11 | 4 cover; \(z=2.24\) misses |
| 16–20 | 0.82, -1.87, 1.42, -0.06, 0.53 | All 5 cover |
There is one miss and 19 covers, so the observed coverage fraction in these 20 repetitions is
For instance, repetition 3 has \(z=2.08\), and \(|2.08|\leq2.101\), so its interval covers. Its estimate is \(b=2.00+2.08(0.20)=2.416\) points per hour, and its interval is \(2.416\pm0.4202=[1.9958,2.8362]\), which contains 2.00. Repetition 12 has \(z=2.24\), so it misses: \(b=2.00+2.24(0.20)=2.448\), and its interval is \(2.448\pm0.4202=[2.0278,2.8682]\), which lies above 2.00. The 19 out of 20 result illustrates a long-run rate; it does not prove that every group of 20 samples will contain exactly 19 covering intervals.
How to Interpret a Confidence Level in Practice
A 95% confidence level describes the performance of a specified interval procedure under its assumptions. It does not describe how likely a particular coefficient value is, nor does it measure the probability that an interval contains the coefficient after the data have been observed. Once the data and endpoints are fixed, the interval either includes the fixed coefficient or it does not. The uncertainty is reflected in the method’s repeated-sampling performance.
The interval also addresses uncertainty about the coefficient in the stated regression model, not every possible limitation of an analysis. For example, a narrow interval does not by itself show that the model is appropriate, that important covariates were included, or that an observed association is causal. Coverage depends on the model, sampling process, and interval procedure being suitable for the question. If the assumptions are substantially violated, a procedure labelled “95%” may cover less—or more—often than 95%.
It is also useful to distinguish nominal coverage from coverage observed in a finite simulation or study. The nominal level is the procedure’s theoretical long-run rate under its assumptions. A finite set of repeated samples will usually show some variation around that rate. Thus, 19 covering intervals out of 20 is a sample result, not a guarantee of exactly 95% coverage in every set of 20.
Common Mistake
A common mistake is to say, “There is a 95% probability that the true slope is inside this interval.” That statement treats the fixed slope as if it were randomly moving after the sample has been collected. In the standard frequentist interpretation used here, the slope is fixed; the interval is what varies across hypothetical repetitions.
A better interpretation is: “This interval was calculated using a procedure that, under its assumptions, covers the fixed population slope in 95% of repeated samples.” For the one observed interval, report its endpoints and units, and describe it as an interval produced by a 95% procedure. Do not turn the procedure’s long-run coverage rate into a probability statement about that particular interval.
Check Your Understanding
Use the repeated-sampling interpretation to answer these questions.
- A 90% interval procedure is used repeatedly under the same valid model. What proportion of intervals should contain the fixed coefficient in the long run?
- Explain why it is not correct, in the usual frequentist interpretation, to say there is a 95% probability that a particular calculated interval contains the fixed coefficient.
- A hypothetical 95% interval is \([1.98, 2.82]\) score points per study hour, and the illustrative fixed slope is 2.00. Does this interval cover? What does the 95% refer to?
- In 100 repeated samples, a procedure produces 91 intervals that cover. Does this single result prove that the procedure has 91% coverage? Explain.
- Name one reason the actual coverage rate might differ from the stated confidence level.