What a Confidence Level Means

What Does a Confidence Level Describe?

A confidence interval is produced from a sample, and a different random sample will generally produce a different interval. The population proportion $p$, by contrast, is a fixed feature of the population: it does not change from one sample to another. The confidence level describes the performance of the interval-producing procedure across repeated random samples, not a probability that this fixed $p$ changes or lies somewhere.

For example, a 95% confidence procedure is designed so that, under its assumptions, intervals produced from repeated samples contain the fixed population proportion about 95% of the time. This is a statement about what happens over repetitions of the sampling and calculation process. It does not guarantee that any particular interval contains $p$, nor that exactly 95% of a small batch of intervals will contain it.

Definition: The confidence level of an interval procedure is the probability, over repeated random samples generated under the stated sampling model, that the resulting interval contains the fixed parameter. A procedure with confidence level $1-\alpha$ has coverage probability $1-\alpha$ when its stated coverage is exact. If the method is approximate, its actual coverage is approximately $1-\alpha$.

The distinction is between a random interval and a fixed parameter. Before the sample is collected, the interval’s endpoints are random because they depend on the random sample. After the sample is collected, the endpoints are fixed numbers. In the frequentist interpretation used here, $p$ is fixed throughout; the randomness is in the sample and therefore in the interval.

The One-Proportion Interval as a Procedure

For a random sample of $n$ binary outcomes, let $x$ be the number of successes and let $\hat{p}=x/n$ be the sample proportion. The familiar one-proportion z interval calculates its endpoints using the sample and a standard normal critical value.

Procedure: The one-proportion z interval with nominal confidence level $1-\alpha$ is $$ \hat{p}\ \pm\ z^* \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}, $$ where $z^*$ is the standard normal critical value with central area $1-\alpha$ between $-z^*$ and $z^*$. This is an approximate interval procedure. Its usual conditions include random sampling and independence of observations, together with sufficiently many successes and failures for a normal approximation to be reasonable.

The endpoints depend on $\hat{p}$, so they vary when the sample varies. The confidence level is a property of this rule for making intervals, used with its assumptions; it is not a property of one pair of endpoints by itself. Because the z interval estimates its standard error using $\hat{p}$, its actual repeated-sampling coverage is generally approximate rather than guaranteed to equal the nominal level exactly.

Why the Long-Run Coverage Is Approximately the Nominal Level

A short derivation shows where the coverage interpretation comes from. Suppose $X_1,\ldots,X_n$ are independent binary observations, each with success probability $p$. Then $\hat{p}$ has mean $p$ and standard deviation $\sqrt{p(1-p)/n}$. When the success-failure conditions make the normal approximation reasonable, the standardized sample proportion is approximately standard normal.

Standardize the sample proportion.
Under the independent binary sampling model, the normal approximation gives $Z=(\hat{p}-p)/\sqrt{p(1-p)/n}\approx N(0,1)$.

Use the central normal probability.
By the choice of $z^*$, a standard normal variable falls between $-z^*$ and $z^*$ with probability $1-\alpha$. Therefore, approximately, $P(-z^*\leq Z\leq z^*)=1-\alpha$.

Rearrange the event to describe coverage.
Substituting the expression for $Z$ and multiplying by the positive standard deviation gives $P(\hat{p}-z^*\sqrt{p(1-p)/n}\leq p\leq\hat{p}+z^*\sqrt{p(1-p)/n})\approx 1-\alpha$.

Replace the unknown standard error.
The procedure uses $\sqrt{\hat{p}(1-\hat{p})/n}$ in place of the unknown $\sqrt{p(1-p)/n}$. This substitution produces the one-proportion z interval and explains why its coverage is approximate, not an exact identity in general.

The probability in the derivation is over repeated samples while $p$ remains fixed. The random quantity is $\hat{p}$, and the interval’s endpoints are functions of $\hat{p}$. After a particular sample has been observed, the interval is no longer random in this framework: it either contains the fixed $p$ or it does not.

Worked Examples

Worked Example: One Interval and a Fixed Proportion

Suppose a hypothetical random sample of $n=100$ binary observations contains $x=42$ successes. Assume the sampling and success-failure conditions are reasonable. For illustration, suppose the fixed population proportion is $p=0.45$. Calculate the 95% z interval and determine whether this particular interval covers $p$.

The sample proportion is $$ \hat{p}=\frac{x}{n}=\frac{42}{100}=0.42. $$ For a 95% interval, $z^*=1.96$. The estimated standard error is $$ SE=\sqrt{\frac{0.42(1-0.42)}{100}} =\sqrt{\frac{0.42(0.58)}{100}} =\sqrt{0.002436} \approx 0.04935585. $$

Using the unrounded standard error to calculate the margin gives $$ 1.96(0.04935585)\approx 0.09673747. $$ Thus the interval endpoints, carried through before rounding, are $$ 0.42-0.09673747\approx 0.32326253,\qquad 0.42+0.09673747\approx 0.51673747. $$ Rounded to four decimal places, the interval is $(0.3233,\ 0.5167)$.

In this hypothetical example, the fixed value $p=0.45$ lies between the endpoints, so this particular interval covers $p$. That fact describes the outcome of this one sample and calculation. The 95% confidence level describes the procedure’s coverage over repeated samples, not a 95% probability that this already-calculated interval contains $p$.

Worked Example: Coverage Across Repeated Samples

Imagine applying the same 95% z-interval procedure to 20 fresh random samples, each of size $100$, from one hypothetical population with fixed proportion $p=0.45$. Assume the observations are independent and the success-failure conditions are adequate for the approximation. Suppose 19 of the 20 calculated intervals contain $0.45$.

The observed fraction of intervals that cover the fixed proportion is $$ \frac{19}{20}=0.95. $$ So in this hypothetical batch, the empirical coverage is 95%. The calculation is a direct count: there are 19 covering intervals out of 20 produced.

This batch illustrates the meaning of coverage, but it does not establish that every set of 20 intervals must contain $p$ exactly 19 times. Even when a procedure has 95% coverage under its model, random variation can make the coverage fraction in a finite batch higher or lower than 0.95. The long-run rate is a property of repeated use, not a quota for each small collection.

Worked Example: Comparing 90% and 95% Procedures

Use the same hypothetical sample, $x=42$ successes out of $n=100$, with $\hat{p}=0.42$ and estimated standard error $0.04935585$. Assume the same random-sampling, independence, and success-failure conditions. Compare the intervals calculated with $z^*=1.645$ for 90% confidence and $z^*=1.96$ for 95% confidence.

For the 90% procedure, the margin is $$ 1.645(0.04935585)\approx 0.08119037, $$ so the interval is $$ 0.42\pm 0.08119037 \quad\Longrightarrow\quad (0.3388,\ 0.5012), $$ rounded to four decimal places.

For the 95% procedure, the margin is $$ 1.96(0.04935585)\approx 0.09673747, $$ and the interval is $$ 0.42\pm 0.09673747 \quad\Longrightarrow\quad (0.3233,\ 0.5167), $$ again rounded to four decimal places.

The two procedures produce different intervals from the same data. Under the stated conditions, the 95% procedure is designed to cover the fixed proportion more often over repeated samples than the 90% procedure. The comparison concerns long-run coverage rates. It does not mean that the 95% interval has a 95% probability of containing $p$ while the 90% interval has a 90% probability, once these particular intervals have been calculated.

Exact Coverage, Approximate Coverage, and Assumptions

The phrase “95% confidence” is shorthand for a repeated-sampling property of a method under a specified model. For an exact procedure, the coverage probability equals its stated confidence level. For an approximate procedure such as the usual one-proportion z interval, the intended coverage is close to that level when the approximation and assumptions are suitable. The stated percentage should not be read as a guarantee that the actual coverage is exactly that value in every population and sample size.

The interpretation also depends on the sampling design. If observations are not representative or are dependent in a way the method does not account for, the interval procedure may not achieve its intended coverage. Random sampling and independence are therefore not decorative technicalities: they support the probability model that makes the long-run statement meaningful. The success-failure condition supports the normal approximation used in the z method.

A confidence level does not measure the probability that the sampling assumptions are true, and it does not repair a biased sample. Nor does it describe the proportion of individual observations that fall inside the interval. It refers specifically to the frequency with which the procedure’s intervals include the population parameter when the procedure is repeatedly applied under its assumptions.

Common Mistakes

Wrong: “There is a 95% probability that the fixed population proportion is in this calculated interval.” Correction: In the frequentist interpretation, the fixed $p$ either is or is not in this realized interval. The 95% describes the interval-producing procedure’s long-run coverage.
Wrong: “Exactly 95 out of every 100 intervals must cover $p$.” Correction: A 95% coverage rate is a long-run probability. Finite batches can have different coverage fractions because the samples are random.
Wrong: “A 95% level means 95% of the observations are between the endpoints.” Correction: The interval estimates a population proportion; its confidence level concerns coverage of the fixed parameter, not the share of observations inside the interval.
Wrong: “The procedure has 95% coverage regardless of how the data were collected.” Correction: The coverage claim depends on the sampling model and approximation conditions. A method’s nominal level does not compensate for a design that violates its assumptions.

Key takeaway: A confidence level belongs to an interval procedure. Under its assumptions, it describes the long-run proportion of intervals that contain the fixed population proportion when the sampling-and-calculation process is repeated.

Check Your Understanding

Answer each question using the repeated-sampling meaning of confidence level.

What changes from sample to sample in a confidence-interval procedure, and what remains fixed?
A procedure has 90% coverage. Does that require exactly 9 of every 10 intervals in any finite batch to contain the parameter? Explain.
In the frequentist interpretation, why is “there is a 95% probability that $p$ is in this already-calculated interval” not the meaning of a 95% confidence level?
What does a 95% level describe for the one-proportion z interval when its assumptions are reasonable?
Why does using an approximate normal method mean its actual coverage need not be exactly its nominal confidence level?

What You'll Learn