Choosing a Confidence Level

Why the Confidence Level Is a Choice

A confidence interval reports a range of plausible values for a population proportion, along with a confidence level describing the method used to produce it. Choosing that level involves a practical tradeoff: a method with greater long-run coverage generally produces wider intervals. A very wide interval may be less useful for making a decision, while a narrower interval comes with less long-run coverage.

There is no confidence level that is best for every question. The choice should reflect how much protection against missing the true proportion is desired and how much interval width is useful for the decision. A level should be chosen before examining the results, rather than selected afterward to make an interval look more favorable.

Definition: The confidence level is the long-run proportion of intervals that would contain the fixed population proportion if the same valid sampling and interval procedure were repeated many times. For a one-proportion \(z\) interval, the confidence level is \(1-\alpha\), where \(\alpha\) is the probability placed in the two tails of the standard normal distribution.

For example, a 95% confidence level describes a procedure that captures the population proportion in about 95% of repeated samples when its assumptions are satisfied. It does not mean that there is a 95% probability that the particular interval already calculated contains the fixed proportion. Once the data have been collected, that interval either contains the population proportion or it does not; the 95% describes the method's repeated-sampling performance.

How Confidence Level Changes Width

For a sample with \(x\) successes among \(n\) observations, the sample proportion is \(\hat{p}=x/n\). The usual one-proportion \(z\) interval is

\hat{p}\ \pm\ z^* \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}.

Here \(z^*\) is the positive standard-normal critical value for the chosen confidence level. The standard error is \(SE=\sqrt{\hat{p}(1-\hat{p})/n}\), and the margin of error is \(ME=z^*SE\). Thus the interval's total width is \(2ME\). For a given sample, \(\hat{p}\), \(n\), and \(SE\) do not change when the confidence level changes; the critical value does.

The usual two-sided critical values are approximately 1.645 for 90% confidence, 1.960 for 95%, and 2.576 for 99%. A higher confidence level requires a larger critical value, so it increases both the margin of error and the interval width.

Key result: For a fixed sample and the same one-proportion interval method, increasing the confidence level increases the critical value \(z^*\), and therefore increases the margin of error and interval width. This is the coverage–precision tradeoff: more long-run coverage comes at the cost of a less precise, wider interval.

Worked Comparison of Confidence Levels

Before interpreting intervals at different confidence levels, check whether the procedure is appropriate. The one-proportion \(z\) interval relies on a random sample (or another sampling design that supports inference to the target population), independence of observations (including the 10% condition when sampling without replacement from a finite population), and a sufficiently large number of successes and failures for the normal approximation. The numerical comparisons below assume these conditions hold.

Worked Example: Comparing 90%, 95%, and 99% Intervals

Suppose a hypothetical random sample of 800 eligible voters includes 416 who support a proposed policy. Compare the intervals at 90%, 95%, and 99% confidence. Assume the sample is less than 10% of the eligible-voter population and that each sampled person’s response is independent. There are 416 successes and 384 failures, both well above 10, so the normal approximation condition is met.

The sample proportion and estimated standard error are

\hat{p}=\frac{416}{800}=0.520,\qquad SE=\sqrt{\frac{0.520(1-0.520)}{800}} =\sqrt{\frac{0.2496}{800}} \approx 0.0176635.

At 90% confidence, using \(z^*=1.645\), the margin of error is

ME=1.645(0.0176635)\approx 0.0290565, \qquad 0.520\pm0.0290565=(0.4909,\ 0.5491).

The width is \(2(0.0290565)=0.058113\), or about 5.81 percentage points. At 95% confidence, using \(z^*=1.960\),

ME=1.960(0.0176635)\approx0.0346205, \qquad 0.520\pm0.0346205=(0.4854,\ 0.5546).

This interval is about \(0.069241\), or 6.92 percentage points, wide. At 99% confidence, using \(z^*=2.576\),

ME=2.576(0.0176635)\approx0.0455012, \qquad 0.520\pm0.0455012=(0.4745,\ 0.5655).

Its width is about \(0.091002\), or 9.10 percentage points. All three intervals are centered at the same sample proportion. The 99% interval is widest because it uses the largest critical value, while the 90% interval is narrowest.

In context, the 95% interval estimates that between about 48.54% and 55.46% of eligible voters support the policy, using a procedure designed for 95% long-run coverage under its assumptions. It does not establish that this particular interval has a 95% probability of containing the true support proportion.

Justifying a Level for an Estimation Goal

A useful choice begins with the purpose of the estimate. Ask how damaging it would be for the interval procedure to miss the true proportion and how narrow an interval must be to support the intended interpretation or decision. A higher level may be appropriate when missing the population value would be especially consequential. A lower level may be reasonable when a narrower interval is more useful and the cost of lower long-run coverage is acceptable. These are design judgments, not mathematical rules that dictate one level for every setting.

The width should be discussed in the units readers care about. For a proportion, a change of 0.01 is one percentage point. Saying that a 95% interval is “wider” than a 90% interval is most informative when the difference is expressed as a change in percentage points, not as an unexplained decimal.

Worked Example: Choosing a Level for a Public Report

A hypothetical city office surveys 600 randomly selected residents about support for a proposed service. Of those surveyed, 312 support it. The office wants an interval for a public report. Assume the sample is less than 10% of the resident population, responses are independent, and the 312 supporters and 288 non-supporters satisfy the success-failure condition.

The estimate and estimated standard error are

\hat{p}=\frac{312}{600}=0.520,\qquad SE=\sqrt{\frac{0.520(0.480)}{600}} =\sqrt{0.000416}\approx0.0203961.

At 90% confidence, \(ME=1.645(0.0203961)\approx0.03355\), so the interval is approximately \((0.4864,\ 0.5536)\), with width \(0.0671\), or 6.71 percentage points. At 95% confidence, \(ME=1.960(0.0203961)\approx0.03998\), so the interval is approximately \((0.4800,\ 0.5600)\), with width \(0.0800\), or 8.00 percentage points.

Suppose the office decides that a public estimate should prioritize stronger long-run coverage, even though an extra 1.11 percentage points of interval width makes the result less precise. It can justify reporting the 95% interval: the choice matches that stated priority. If the office instead used 90%, it should explain why the narrower range is useful enough to accept the lower long-run coverage. The data alone do not decide between these choices.

The tradeoff is about the interval procedure, not about changing the observed data. If the same sample is used and the same interval method is applied, changing 90% to 95% does not change the estimate or make the sample more representative. It changes how wide the interval is, and hence the method's long-run coverage.

Worked Example: A High-Consequence Estimate

A hypothetical health department estimates the proportion of adults in a region who have received a seasonal vaccination. In a random sample of 500 adults, 90 report receiving it. Assume the sampling design supports independence, the sample is less than 10% of the adult population, and the success and failure counts, 90 and 410, are both at least 10.

The sample proportion and estimated standard error are

\hat{p}=\frac{90}{500}=0.180,\qquad SE=\sqrt{\frac{0.180(0.820)}{500}} =\sqrt{0.0002952}\approx0.0171814.

If the department prioritizes high long-run coverage for this public-health estimate, it might select 99% confidence. With \(z^*=2.576\),

ME=2.576(0.0171814)\approx0.04426,\qquad 0.180\pm0.04426\approx(0.1357,\ 0.2243).

The interval is about \(2(0.04426)=0.08852\), or 8.85 percentage points, wide. The department should report the resulting loss of precision as well as the higher confidence level. A 99% interval does not eliminate uncertainty, guarantee inclusion of the true proportion, or correct for bias in the sampling process; it describes the coverage of the procedure when its assumptions are satisfied.

A Short Justification of the Tradeoff

The width result follows directly from the interval formula. For a fixed sample, define \(SE=\sqrt{\hat{p}(1-\hat{p})/n}\). This is a fixed nonnegative number once the data are observed. The interval endpoints are \(\hat{p}-z^*SE\) and \(\hat{p}+z^*SE\), so subtracting the lower endpoint from the upper gives

(\hat{p}+z^*SE)-(\hat{p}-z^*SE)=2z^*SE.

The confidence level is the central probability between \(-z^*\) and \(z^*\) under the standard normal curve. Increasing that central probability requires moving each endpoint farther from zero, which means a larger \(z^*\). Since \(SE\) stays fixed for the same sample, \(2z^*SE\) must increase. This justifies why a higher confidence level produces a wider interval for the same data and method.

Common Mistakes

Claiming the level is the probability the parameter is in the observed interval. The population proportion is fixed. The confidence level describes the long-run coverage of the procedure, not a probability assigned to the parameter after the interval is calculated.
Assuming higher confidence also means greater precision. For the same sample and method, greater confidence means a larger margin of error and a wider, less precise interval.
Choosing a level after seeing which interval gives the preferred conclusion. This makes the reported uncertainty dependent on the observed result and can mislead readers. Plan the level in advance and report it transparently.
Thinking a high confidence level repairs a poor sample. A larger critical value does not remove selection bias, nonresponse bias, dependence, or other failures of the assumptions.
Using a universal default without considering the question. Common levels such as 90%, 95%, and 99% are choices, not rules that settle every decision. The level should match the consequences of uncertainty and the precision needed.

Key takeaway: Choose a confidence level by weighing the desired long-run coverage against the width that will result, in context. State the reason for the choice, plan it before examining the interval, and remember that confidence level does not replace sound sampling or valid assumptions.

Check Your Understanding

Use the interval formula and the coverage–precision tradeoff to answer these questions.

For the same sample and interval method, why is a 99% interval wider than a 90% interval?
In repeated sampling, what does a 95% confidence level mean?
A report changes its stated level from 95% to 90% after noticing that the 90% interval excludes a value of interest. What is the problem with this reasoning?
Why does selecting a higher confidence level not correct bias caused by an unrepresentative sample?
When might an analyst reasonably prefer a wider interval with higher long-run coverage? Explain what contextual consideration matters.

What You'll Learn