Errors in Interval-Based Decisions

From Confidence Intervals to Decision Errors

In “Type I Error and Multiple Tests,” we considered errors made by significance-test decisions. A related decision can be made with a confidence interval: check whether the interval contains the null value. This tutorial connects the confidence level of that interval to the chance of making a Type I error with this decision rule.

For example, suppose a two-sided interval is used to decide whether a population proportion equals a particular null value \(p_0\). If the interval excludes \(p_0\), the interval-based rule rejects \(H_0:p=p_0\). If it includes \(p_0\), the rule fails to reject \(H_0\). The confidence level helps describe how often this rule would wrongly exclude the null value over repeated samples when that value really is the population proportion.

Definition: For an interval-based decision, a Type I error occurs when the interval excludes the null value even though that value is the true population parameter. For a two-sided interval with confidence level \(1-\alpha\), the nominal Type I error rate is \(\alpha\): the probability that the interval procedure excludes the true parameter.

A 95% confidence interval has a nominal Type I error rate of \(1-0.95=0.05\), or 5%, for the corresponding interval-based decision. A 90% confidence interval has a nominal rate of \(1-0.90=0.10\), or 10%. This is a long-run description of the procedure across repeated samples, not a claim about the probability that a particular null hypothesis is true.

The connection is clearest when the interval and decision rule are matched: use a two-sided interval for the parameter named in the null hypothesis, and reject when the null value is outside the interval. For the one-proportion z intervals used below, the confidence level and error rate are nominal because the z interval is an approximate method. Do not assume that a confidence interval and a separately calculated test using a different procedure must always give exactly the same numerical result.

How the Confidence Level Sets the Nominal Error Rate

A confidence level describes the long-run success rate of an interval procedure: if the method were used on many random samples in the same way, about that proportion of the resulting intervals would capture the true parameter. Therefore, the complementary proportion of intervals would miss it. When the interval is used to check a null value, missing the true value means excluding it and making a Type I error.

\text{Nominal Type I error rate} = \alpha = 1-\text{confidence level}

For instance, a 99% confidence procedure has a nominal error rate of \(1-0.99=0.01\), or 1%. A 90% procedure has a nominal error rate of \(1-0.90=0.10\), or 10%. The confidence level and error rate move in opposite directions.

For a one-proportion z interval, the interval is calculated as

\hat{p} \pm z^*\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}

Here, \(\hat{p}\) is the sample proportion, \(n\) is the sample size, and \(z^*\) is the critical value for the chosen confidence level. A higher confidence level uses a larger \(z^*\), so the interval is wider. A wider interval is less likely to exclude the true proportion, which corresponds to a lower nominal Type I error rate.

Conditions: Before using a one-proportion z interval, check that the data come from a random sample or a randomized experiment, observations are independent (and check the 10% condition for sampling without replacement), and the Large Counts condition is met: \(n\hat{p} \ge 10\) and \(n(1-\hat{p}) \ge 10\). The z interval gives an approximate confidence level and approximate error control.

Making the Decision From an Interval

For a two-sided interval-based decision about \(H_0:p=p_0\), the rule is straightforward. If \(p_0\) lies outside the interval, reject \(H_0\) at the nominal significance level \(\alpha=1-\text{confidence level}\). If \(p_0\) lies inside the interval, fail to reject \(H_0\) at that level. Failing to reject is not proof that \(p=p_0\); it means the interval does not exclude that value.

Identify the interval and null value.
Confirm that the interval is for the same population parameter named in the null hypothesis.

Find the nominal significance level.
Subtract the confidence level from 1 to get \(\alpha\).

Check inclusion.
If the null value is outside the interval, reject \(H_0\); if it is inside, fail to reject \(H_0\).

Describe the error rate in context.
State the long-run chance of excluding the true parameter, using the nominal rate and noting that a z interval is approximate.

Worked Example: A 90% Interval Excludes the Null Value

A community group takes a random sample of 200 residents from a town of 12,000. Of the sampled residents, 116 say they regularly use a public bike path. Consider the null hypothesis \(H_0:p=0.50\), where \(p\) is the proportion of all town residents who regularly use the path. Use a 90% one-proportion z interval to make an interval-based decision and describe its nominal Type I error rate.

State: We are deciding whether the interval excludes the null value \(p_0=0.50\). The confidence level is 90%, so the nominal significance level is \(\alpha=1-0.90=0.10\).

Plan and conditions: Use a one-proportion z interval. The residents were selected randomly. The 10% condition is met because \(200 \le 0.10(12{,}000)=1{,}200\), supporting independence when sampling without replacement. There are 116 successes and \(200-116=84\) failures, both at least 10, so the Large Counts condition is met.

Do: The sample proportion is \(\hat{p}=116/200=0.58\). For a 90% interval, \(z^*=1.645\). The standard error and margin of error are

\sqrt{\frac{0.58(1-0.58)}{200}} =\sqrt{0.001218} \approx 0.03490

1.645(0.03490)\approx 0.05741

Thus, the interval is

0.58\pm0.05741 =(0.5226,\ 0.6374)

The null value \(0.50\) is below the lower endpoint, so it is outside the interval.

Conclude: The interval-based rule rejects \(H_0:p=0.50\) at the nominal \(\alpha=0.10\) level. The sample provides evidence that the proportion of town residents who regularly use the path differs from 0.50. If 0.50 were actually the population proportion, this 90% interval procedure would have a nominal 10% chance of excluding it and producing a Type I error; the z interval’s error control is approximate.

Worked Example: A 95% Interval Contains the Null Value

A random sample of 100 customers is selected from a retailer’s 5,000 customers. Fifty-four say they would recommend the retailer to a friend. Use a 95% one-proportion z interval to decide whether to reject \(H_0:p=0.50\), where \(p\) is the proportion of all customers who would recommend the retailer.

Plan and conditions: Use a one-proportion z interval and compare its endpoints with 0.50. The sample is random. The 10% condition holds because \(100\le0.10(5{,}000)=500\). There are 54 successes and 46 failures, so both counts meet the Large Counts condition.

Do: Here, \(\hat{p}=54/100=0.54\). For 95% confidence, \(z^*=1.96\). The standard error and margin of error are

\sqrt{\frac{0.54(0.46)}{100}} =\sqrt{0.002484} \approx0.04984

1.96(0.04984)\approx0.09769

The interval is

0.54\pm0.09769 =(0.4423,\ 0.6377)

The null value \(0.50\) is inside this interval.

Conclude: At the nominal \(\alpha=1-0.95=0.05\) level, the interval-based rule fails to reject \(H_0\). The data do not provide convincing evidence that the proportion of all customers who would recommend the retailer differs from 0.50. This does not prove that the population proportion is 0.50. The 95% procedure has a nominal 5% chance of excluding the true proportion and making a Type I error.

Changing the Confidence Level Can Change the Decision

For the same sample, increasing the confidence level makes the interval wider. A null value included in a wide interval may be excluded from a narrower one. This is not a contradiction: the narrower interval uses a lower confidence level and therefore accepts a larger nominal chance of excluding the true parameter.

Worked Example: Comparing 90% and 80% Intervals

A random sample of 100 people is taken from a population of 8,000, and 57 report using a community recreation center. Consider \(H_0:p=0.50\), where \(p\) is the population proportion of users. Compare the interval-based decisions using 90% and 80% confidence intervals.

Plan and conditions: Use the one-proportion z interval at each confidence level. The sample is random, and the 10% condition holds because \(100\le0.10(8{,}000)=800\). There are 57 successes and 43 failures, both at least 10. The same sample proportion and standard error apply to both intervals; only \(z^*\) changes.

Do: The sample proportion is \(\hat{p}=57/100=0.57\). Its standard error is

\sqrt{\frac{0.57(0.43)}{100}} =\sqrt{0.002451} \approx0.04951

For 90% confidence, use \(z^*=1.645\). The margin of error is \(1.645(0.04951)\approx0.08144\), giving

0.57\pm0.08144=(0.4886,\ 0.6514)

The interval contains 0.50, so the 90% interval-based rule fails to reject \(H_0\) at nominal \(\alpha=0.10\).

For 80% confidence, use \(z^*=1.282\). The margin of error is \(1.282(0.04951)\approx0.06347\), giving

0.57\pm0.06347=(0.5065,\ 0.6335)

The interval excludes 0.50, so the 80% interval-based rule rejects \(H_0\) at nominal \(\alpha=0.20\).

Conclude: The 90% interval leads to failing to reject, while the narrower 80% interval leads to rejecting the null value. The 80% procedure has the larger nominal Type I error rate: 20%, compared with 10% for the 90% procedure. The result illustrates the tradeoff: a lower confidence level creates a narrower interval but makes it more likely to exclude the true proportion.

What the Error Rate Does—and Does Not—Mean

The nominal Type I error rate is a property of a decision procedure over repeated samples, under the condition that the null value is true. For example, saying that a 95% interval-based procedure has a nominal 5% Type I error rate means that, in repeated use under its assumptions, it will exclude the true parameter about 5% of the time. It does not mean there is a 5% chance that the null hypothesis is true after seeing a particular interval.

The rate also does not tell us whether a particular rejection is an error. If an interval excludes \(p_0\), the decision may be correct when \(p_0\) is false, or it may be a Type I error when \(p_0\) is true. The interval and sample alone do not reveal which is the case.

As with the significance level discussed in “Significance Level as the Probability of a Type I Error,” a smaller \(\alpha\) reduces the chance of a Type I error for the procedure. Here, choosing a higher confidence level makes the interval wider and reduces the chance of excluding a true null value. That conservative choice can also make it harder for an interval to exclude a null value when the population parameter differs from it.

Common Mistakes and AP Exam Tip

Using the confidence level as the error rate: A 95% interval has a nominal error rate of \(1-0.95=0.05\), not 0.95.
Saying “accept the null” when the interval includes it: Say “fail to reject \(H_0\).” Inclusion means the interval does not rule out the null value; it does not prove that value is true.
Calling every rejection a Type I error: A Type I error occurs only if the null value is actually true and the interval excludes it. The truth is not known just from the sample result.
Treating the rate as a probability about a particular hypothesis: The nominal error rate describes the long-run behavior of the procedure when the null value is true, not the probability that the null is true for this study.
Ignoring the method and conditions: Check the randomization or sampling method, independence and 10% condition where relevant, and Large Counts. For a one-proportion z interval, the stated confidence and error rates are approximate.
Comparing an interval with the wrong null value: Make sure the interval is for the same population parameter named in \(H_0\), and use a two-sided interval when the decision concerns whether the parameter differs from the null value.

For a full-credit response, identify the confidence level, calculate \(\alpha=1-\text{confidence level}\), state whether the null value is inside the interval, and use “reject” or “fail to reject” appropriately. Interpret the nominal Type I error rate as a long-run chance under a true null, in context, rather than as certainty about the result of one sample.

Key takeaway: For a matched two-sided interval-based decision, the nominal Type I error rate is one minus the confidence level. Excluding a true null value is a Type I error; including it means fail to reject, not prove, the null. One-proportion z intervals provide approximate error control.

Check Your Understanding

Answer each question using the interval-based decision rule and the assumptions stated.

A 92% interval is used to decide whether a null value should be excluded. What is the nominal Type I error rate?
A 95% interval for a population proportion is \((0.31,0.47)\). For \(H_0:p=0.40\), what is the decision at the nominal 5% level? Explain briefly.
A 90% interval contains the null value. What can you conclude, and what can you not conclude, about the null hypothesis?
Why does an 80% interval generally have a higher nominal Type I error rate than a 95% interval?
In the recreation-center example, why did the 90% and 80% intervals give different decisions for the same sample?