Why the Wording of Confidence Matters
A confidence interval is often reported in a sentence such as “the 95% confidence interval is from 100.38 to 108.22.” It is tempting to read this as saying that, after the data are observed, there is a 95% probability that the unknown parameter lies between those two numbers. In the frequentist framework, that is not what the confidence level means.
The distinction is about what is random. Before a sample is taken, the sample is random, and an interval calculated from that sample is random too. The parameter is treated as fixed, although its value is unknown. The confidence level describes how often the interval-producing procedure captures that fixed parameter over repeated samples—not a probability assigned to the parameter after one particular interval has been observed.
Once the data are observed as \(X=x\), the interval \(C(x)\) is fixed. For a given fixed \(\theta\), whether \(\theta\) is inside that interval is also fixed: it either is or is not. The interval's coverage probability remains a property of the procedure considered over possible samples; it is not a probability that can be attached to the already determined membership event.
A Discrete Example: Coverage Before and After Sampling
The distinction can be seen exactly in a small binomial model. Consider a deliberately simple interval rule. Its purpose is to illustrate the meaning of coverage, not to recommend an interval for practical use.
Worked Example: A Binomial Rule and One Observed Interval
A hypothetical quality-control team tests \(n=4\) independently selected components. Each component either passes or fails, with pass probability \(p\). Let \(X\) be the number that pass, so \(X\sim\operatorname{Binomial}(4,p)\). The team uses \(C(X)=[X/4-0.25,\;X/4+0.25]\), truncated to the parameter space \([0,1]\). There is no interest-rate or mortality assumption in this statistical example. Suppose the fixed parameter is \(p=0.5\), and the observed result is \(X=2\). Find the procedure's coverage at \(p=0.5\), then interpret the observed interval.
Step 1: Identify which samples cover the fixed value. If \(X=0\), the interval is \([0,0.25]\), which does not contain \(0.5\). If \(X=1\), it is \([0,0.5]\), which does contain \(0.5\). If \(X=2\), it is \([0.25,0.75]\), which contains \(0.5\). If \(X=3\), it is \([0.5,1]\), which contains \(0.5\). If \(X=4\), it is \([0.75,1]\), which does not contain \(0.5\).
Step 2: Add the probabilities of the covering outcomes. Under \(X\sim\operatorname{Binomial}(4,0.5)\), the probabilities for \(X=1,2,3\) are \(4/16,6/16,4/16\), respectively. Therefore the coverage probability is \(14/16=0.875\).
Step 3: Calculate the interval from the observed data. With \(X=2\), the observed interval is \([2/4-0.25,\;2/4+0.25]=[0.25,0.75]\). Under the stated teaching value \(p=0.5\), this particular interval contains \(p\).
Interpretation and check. Before testing, the procedure has a \(0.875\) probability of producing an interval that contains \(p=0.5\). After observing \(X=2\), the interval and the fact that it contains \(0.5\) are determined. The \(0.875\) is not a probability that this already observed interval contains \(p\); here, given the stipulated value \(p=0.5\), it does contain it. The coverage calculation checks because precisely \(X=1,2,3\) cover \(0.5\), and their probabilities sum to \(14/16\).
Method and common mistake avoided. Enumerate the sample outcomes that cover the fixed parameter, then sum their sampling probabilities. Do not take the procedure's \(0.875\) coverage and describe it as the probability that the particular interval \([0.25,0.75]\) contains \(p\).
Why Coverage Is a Probability About the Procedure
The notation \(C(X)\) emphasizes that the interval depends on the random data. The coverage event \(\{\theta\in C(X)\}\) can occur for some possible samples and fail for others. Its probability is calculated using the distribution of \(X\) under the fixed parameter value \(\theta\).
For discrete data, this equation adds the probabilities of all data outcomes whose intervals contain \(\theta\). Here \(\mathbf{1}\{\theta\in C(x)\}\) equals 1 when the interval calculated from \(x\) contains \(\theta\), and equals 0 otherwise. For continuous data, the corresponding calculation uses an integral rather than a sum.
This also explains what changes after observation. Before sampling, \(X\) ranges over possible outcomes, so the indicator can be 0 or 1 depending on which outcome occurs. After observing \(X=x\), the indicator \(\mathbf{1}\{\theta\in C(x)\}\) is just a fixed value. For the frequentist model, the parameter \(\theta\) is fixed throughout; the randomness in the coverage probability comes from the sampling process.
This interpretation does not claim that an analyst knows whether an observed interval covers the parameter. The analyst usually does not know the fixed parameter. It says that the frequentist confidence level is not, by itself, a probability distribution for that parameter. A probability statement about a parameter given observed data requires a different framework, such as a Bayesian model with a prior distribution.
A Normal-Mean Interval Before and After Observation
A familiar normal interval makes the repeated-sampling interpretation especially clear. Suppose \(X_1,\ldots,X_n\) are independent observations from \(N(\mu,\sigma^2)\), where the variance \(\sigma^2\) is known. Then the sample mean has distribution \(N(\mu,\sigma^2/n)\). The following interval procedure uses that sampling distribution.
Worked Example: Reading an Observed Normal Interval
A hypothetical laboratory measures a chemical concentration in \(n=36\) independent samples. Assume each measurement is normally distributed with fixed mean \(\mu\) and known standard deviation \(\sigma=12\) concentration units; thus \(X_i\sim N(\mu,12^2)\). There is no interest-rate or mortality assumption. The observed sample mean is \(\bar x=104.30\). Calculate the 95% interval and explain its confidence interpretation. For checking whether this particular interval covers the parameter, suppose the fixed teaching value is \(\mu=102.50\).
Step 1: Calculate the standard error. The known standard deviation is \(12\), and the sample size is \(36\), so \(\sigma/\sqrt n=12/\sqrt{36}=12/6=2\).
Step 2: Calculate the margin of error. The multiplier is \(1.96\), so the margin is \(1.96(2)=3.92\).
Step 3: Form the interval from the observed mean. The endpoints are \(104.30-3.92=100.38\) and \(104.30+3.92=108.22\). The observed interval is therefore \([100.38,108.22]\), in concentration units.
Step 4: Check the stipulated fixed value. Since \(100.38\leq102.50\leq108.22\), this observed interval contains the stipulated \(\mu\).
Interpretation and check. Across repeated samples under the same normal model and fixed \(\mu\), the procedure covers \(\mu\) with probability \(0.95\). The calculation of the observed interval does not turn that statement into a 0.95 probability for \(\mu\) to lie in \([100.38,108.22]\). Under the supplied teaching value, membership is simply true for this interval. The endpoints are centered at \(104.30\), and each is \(3.92\) units from the center.
Method and common mistake avoided. Use the known-\(\sigma\) normal sampling distribution to construct the interval, then distinguish its repeated-sampling guarantee from the observed containment check. Do not use the observed sample mean as evidence that the fixed \(\mu\) itself has become random.
Repeated Intervals Make the Interpretation Visible
It is useful to imagine applying one procedure repeatedly, always under the same model and the same fixed parameter. Each new sample can produce a different interval. Some of those intervals contain the parameter and some do not. The confidence level describes the long-run behavior of this collection of intervals, not a probability that changes from interval to interval after one has been selected.
Worked Example: Tracking Ten Intervals From One Procedure
A hypothetical instrument is used to estimate a fixed mean concentration \(\mu\). Assume independent normal observations with known standard deviation \(\sigma=12\), sample size \(n=36\) on each run, and no interest-rate or mortality assumption. The procedure on each run is \(\bar X\pm1.96(12/\sqrt{36})=\bar X\pm3.92\). For illustration, take the fixed value \(\mu=102\) and suppose ten repeated runs produce the following sample means. Determine which observed intervals cover \(\mu\), and state what the count does and does not establish.
Step 1: Use the same margin on every run. Since \(12/\sqrt{36}=2\), the margin is \(1.96(2)=3.92\). For each run, subtract \(3.92\) from its mean for the lower endpoint and add \(3.92\) for the upper endpoint.
| Observed mean | Interval from the procedure | Contains \(\mu=102\)? |
|---|---|---|
| 97.00 | [93.08, 100.92] | No |
| 98.10 | [94.18, 102.02] | Yes |
| 99.00 | [95.08, 102.92] | Yes |
| 100.00 | [96.08, 103.92] | Yes |
| 101.00 | [97.08, 104.92] | Yes |
| 102.00 | [98.08, 105.92] | Yes |
| 103.00 | [99.08, 106.92] | Yes |
| 104.00 | [100.08, 107.92] | Yes |
| 105.90 | [101.98, 109.82] | Yes |
| 107.00 | [103.08, 110.92] | No |
Step 2: Count coverage in this illustrative record. Eight of the ten displayed intervals contain \(102\), while the intervals from means \(97\) and \(107\) do not. Thus the observed fraction covering the stipulated value in this ten-run record is \(8/10=0.8\).
Interpretation and check. This count records what happened in these ten runs; it does not change the definition of the procedure's confidence level, nor does it assign an 0.8 probability to any one of the displayed intervals. In particular, the interval from the mean \(98.10\) contains \(102\) because \(98.10-3.92=94.18\) and \(98.10+3.92=102.02\). The interval from \(107.00\) misses because its lower endpoint, \(103.08\), is greater than \(102\).
Method and common mistake avoided. Apply the same interval rule to each sample and check containment against the same fixed parameter. Do not treat the fraction from a small illustrative collection as a probability statement about a particular realized interval.
Common Misinterpretations
The distinction can be summarized by separating the random procedure from its realized output. Before sampling, the procedure has a distribution over possible intervals. After sampling, one interval has been calculated. The confidence statement concerns the former.
- Wrong: “The observed interval is \([0.25,0.75]\), so there is a 0.875 probability that \(p\) lies in it.” This takes the binomial procedure's coverage at \(p=0.5\) and incorrectly assigns it to one interval after observing \(X=2\).
- Correction: Before sampling, the stated rule covers \(p=0.5\) with probability \(0.875\). After observing \(X=2\), that interval is fixed. If \(p=0.5\), it contains \(p\); if \(p\) has another fixed value, it either contains that value or does not.
- Wrong: “The parameter is unknown, so it must be random.” Unknown and random are not equivalent. In frequentist inference, the parameter is fixed but not known to the analyst.
- Correction: Describe what is random under the model—in this setting, the sample and the interval calculated from it. Use a Bayesian framework if the intended statement is a posterior probability for the parameter, and specify the prior and model.
- Wrong: “A 95% confidence interval means 95% of the values inside the interval are likely parameter values.” The confidence level does not describe a probability distribution across points in the interval.
- Correction: The confidence level characterizes how the interval procedure performs across repeated samples under its assumptions.
Check Your Understanding
For each problem, distinguish the probability statement made before sampling from what can be said about the interval after the data are observed.
- Let \(X\sim\operatorname{Binomial}(3,p)\), with fixed \(p=1/3\). A rule reports \(C(X)=[X/3-1/3,\;X/3+1/3]\), truncated to \([0,1]\). Which values of \(X\) produce an interval containing \(p\), and what is the coverage probability? If \(X=1\), is the coverage probability the probability that this observed interval contains \(p\)? (Answer: \(X=0,1,2\) cover; coverage is \(P(X\leq2)=26/27\); no, the observed interval either contains the fixed \(p\) or it does not.)
- Independent observations follow \(N(\mu,5^2)\), with known \(\sigma=5\), sample size \(n=25\), and no interest-rate or mortality assumption. A 95% interval uses \(\bar X\pm1.96\sigma/\sqrt n\). If \(\bar x=20.4\), calculate the interval and say what the 95% refers to. (Answer: standard error \(1\), interval \([18.44,22.36]\); 95% refers to repeated-sampling coverage of the procedure under the model.)
- A procedure has \(P_\theta\{\theta\in C(X)\}=0.90\) for a fixed \(\theta\). After one sample, the reported interval is \([7.2,9.8]\). Explain why the 0.90 does not establish a 0.90 probability that \(\theta\) lies in this particular interval. (Answer: the 0.90 is the probability, before sampling, that the procedure produces a covering interval; after observation, this interval either contains the fixed \(\theta\) or does not.)
- For a normal-mean procedure, suppose \(\bar X\sim N(\mu,4)\) and the rule is \(\bar X\pm3.92\). The fixed mean is \(\mu=50\), and an observed sample mean is \(46.5\). Determine whether this interval covers \(\mu\), then distinguish that result from a probability claim. (Answer: the interval is \([42.58,50.42]\), so it covers \(50\); for this observed interval containment is a fixed yes-or-no fact, while the procedure's coverage is evaluated over repeated samples.)
- A quality study uses a fixed-parameter model and produces 20 intervals by repeating the same procedure. Seventeen contain the stipulated parameter value. What is the empirical fraction of these intervals that covered, and why is it not automatically the probability that a selected observed interval contains the parameter? (Answer: \(17/20=0.85\); the fraction summarizes these 20 repetitions, while a selected observed interval's containment is fixed and the frequentist confidence statement is about the procedure.)