What a Type I Error Means for a Program
In “Two Possible Errors in a Significance Test,” you learned that a Type I error occurs when a test rejects a true null hypothesis. This tutorial makes that definition specific to a program designed to change a population proportion. The key is to say both what the test concludes and what is actually true about the population.
Suppose a school introduces a new program intended to increase the proportion of students who submit assignments on time. Let \(p_{\text{new}}\) be the population proportion who would submit on time with the program, and let \(p_{\text{existing}}\) be the corresponding population proportion under the existing approach. The study uses sample data to assess whether the program increases that proportion.
The “less than or equal to” part matters. If the program truly has no effect, then the two proportions are equal. But if the program actually lowers the proportion, the null claim of “no increase” is also true. In either case, a test that concludes there is an increase has made a Type I error.
For the usual two-proportion z-test calculation, the null is represented at its boundary, \(p_{\text{new}}=p_{\text{existing}}\). That equality is the value used to calculate the test statistic under the null model. It does not change the contextual meaning of the null: the program does not increase the population proportion. The null also includes the possibility that the new proportion is lower.
A Type I error is a possible outcome, not something we can identify with certainty after a test. We observe the test decision, but the actual population proportions are generally unknown. The error description asks us to imagine the situation in which the null claim is true and the test nevertheless rejects it.
From the Decision to the Contextual Error
A clear contextual statement follows a simple pattern: name the test’s conclusion, then state the population truth that would make that conclusion wrong. For an increase test, the test concludes that the new program raises the proportion. The conclusion is a Type I error if the new program’s actual population proportion is equal to or below the existing proportion.
For an increase question, the alternative says the new program’s population proportion is greater than the existing proportion.
Rejecting the null gives convincing evidence for the increase described by the alternative.
The new proportion might equal the existing proportion or be lower than it.
A Type I error would be concluding that the program increases the population proportion when it actually does not increase it.
This sequence builds on the decision table in “Two Possible Errors in a Significance Test.” The table gives the general rule—reject a true null. The hypotheses tell you what “true null” means in the particular program setting.
Worked Examples: State the Type I Error
Worked Example: On-Time Assignment Submissions
A school evaluates whether a new reminder program increases the proportion of students who submit assignments on time. Let \(p_{\text{new}}\) be the proportion for the population of students using the new program and \(p_{\text{existing}}\) the proportion for the population using the existing approach. The hypotheses are \(H_0:p_{\text{new}}\le p_{\text{existing}}\) and \(H_a:p_{\text{new}}>p_{\text{existing}}\).
Suppose a properly conducted test uses \(\alpha=0.05\) and reports a p-value of 0.041. Since \(0.041<0.05\), the school rejects \(H_0\). The test’s conclusion is that the sample provides convincing evidence that the new program increases the population proportion of on-time submissions.
Now suppose, for the purpose of describing a possible error, that the actual population proportions are \(p_{\text{new}}=0.56\) and \(p_{\text{existing}}=0.60\). The new proportion is lower, so the null claim of no increase is true. Because the test rejected that null and concluded that the new program increases the proportion, this would be a Type I error.
The error is not limited to the case where the proportions are equal. If the actual proportions were both 0.60, the null would also be true and the same rejection would still be a Type I error. The important comparison is whether the new proportion is greater, not whether it is exactly equal.
Worked Example: A Community Clinic’s Appointment Program
A clinic introduces a new appointment program intended to increase the proportion of patients who attend scheduled visits. Let \(p_{\text{new}}\) be the population proportion who would attend with the new program, and \(p_{\text{existing}}\) the proportion under the current system. The clinic tests \(H_0:p_{\text{new}}\le p_{\text{existing}}\) against \(H_a:p_{\text{new}}>p_{\text{existing}}\).
Assume the test is conducted appropriately at \(\alpha=0.05\), and its p-value is 0.018. The clinic rejects \(H_0\) and concludes that there is convincing evidence the new program increases the population proportion of attended visits.
Consider a possible population truth: the actual proportions are \(p_{\text{new}}=0.72\) and \(p_{\text{existing}}=0.72\). The proportions are equal, so the program has not increased the attendance proportion. If the clinic nevertheless rejects \(H_0\), that is a Type I error: it concludes there is an increase when there is none.
The test’s p-value does not tell the clinic whether this error actually occurred. The clinic knows that it rejected the null; it does not know the actual population proportions with certainty. The Type I error description is conditional: if the actual proportions are equal or the new one is lower, then this rejection is a Type I error.
Worked Example: A Program Intended to Change Recycling Participation
A town evaluates a new recycling program and asks whether it changes the population proportion of households that recycle regularly. This question allows for either an increase or a decrease. Let \(p_{\text{new}}\) be the proportion under the new program and \(p_{\text{existing}}\) the proportion under the existing program. The hypotheses are \(H_0:p_{\text{new}}=p_{\text{existing}}\) and \(H_a:p_{\text{new}}\ne p_{\text{existing}}\).
Suppose an appropriate test at \(\alpha=0.05\) produces a p-value of 0.032. The town rejects \(H_0\) and concludes there is convincing evidence that the new program changes the population proportion of households that recycle regularly.
For this two-sided test, a Type I error occurs if the actual proportions are equal. For example, if both actual population proportions are 0.48, then \(H_0\) is true. Rejecting it would mean concluding that the program changes the proportion when, in fact, it does not.
Notice how the error statement depends on the alternative. In the first two examples, the alternative was specifically an increase, so the false conclusion is “the new proportion is greater” and the null includes both equality and a lower new proportion. Here, the alternative is any change, so the null is equality and the Type I error is concluding there is a change when the proportions are equal.
What the Significance Level Means Here
Before conducting a test, researchers choose a significance level, \(\alpha\). It sets the maximum long-run probability of a Type I error for the testing procedure when the null hypothesis is true, assuming the procedure’s conditions are met. For example, with \(\alpha=0.05\), the procedure is designed to have a Type I error rate no greater than 0.05 when the null is true.
For a one-sided test of an increase, the null includes all cases where \(p_{\text{new}}\le p_{\text{existing}}\). The usual two-proportion z-test calculates its reference distribution at the equality boundary. The probability of rejecting is controlled at that boundary; when the actual new proportion is below the existing one, the chance of a false rejection is generally smaller.
Do not interpret \(\alpha=0.05\) as saying there is a 5% chance that the null hypothesis is true, or a 5% chance that a particular rejection is wrong. The significance level describes the long-run behavior of the test procedure in situations where the null is true. It does not reveal the unknown truth in a particular study.
Common Mistakes and AP Exam Tip
- Leaving out the population claim: “Rejecting a true null” is the definition, but it is not a complete contextual explanation. Name the program and population proportion being tested.
- Describing only equality for an increase test: If \(H_a\) claims an increase, \(H_0\) means no increase. A Type I error occurs when the new proportion is actually equal to or less than the existing proportion.
- Mixing up the alternative and the error: The alternative describes the conclusion the test seeks evidence for. The Type I error describes that conclusion being made when the null is actually true.
- Calling a non-rejection proof of no effect: Failing to reject \(H_0\) does not establish that the population proportions are equal or that the program has no effect. It means the test did not provide convincing evidence for the alternative.
- Misreading \(\alpha\): Do not say that \(\alpha\) is the probability the null is true or the probability that this particular conclusion is wrong. Describe it as a long-run Type I error rate when the null is true.
- Using the wrong error for a two-sided question: For “changes” versus “does not change,” the null is equality. A Type I error is concluding that the proportions differ when they are actually equal.
A full-credit contextual answer usually includes both parts: “A Type I error would occur if the test concludes that the new program increases the population proportion of on-time submissions, when the actual proportion under the program is no greater than under the existing approach.” This states the false conclusion and the population truth that makes it false.
Check Your Understanding
For each situation, identify the Type I error in context or explain why the proposed statement is incomplete.
- A program is tested for increasing the proportion of residents who complete a safety course. State the null claim in words and describe the Type I error.
- For an increase test, the actual new-program proportion is 0.43 and the existing proportion is 0.47. The test rejects the null. What kind of outcome is this?
- For an increase test, the actual new-program proportion and existing proportion are both 0.65. The test rejects the null. Describe the error in context.
- A test asks whether a new transit program changes the proportion of riders who arrive on time. What population condition must be true for a rejection to be a Type I error?
- Explain why \(\alpha=0.05\) does not mean there is a 5% probability that a particular rejection is wrong.