What a Type II Error Means for a Program
In “Two Possible Errors in a Significance Test” and “Type I Error Defined in Context,” you learned that a significance test compares a decision based on sample data with what is actually true about a population. This tutorial focuses on the other possible error: failing to reject a null hypothesis that is false.
Return to the school reminder program. The school wants to know whether a new program increases the proportion of students who submit assignments on time. Let \(p_{\text{new}}\) be the population proportion who would submit on time using the new program, and let \(p_{\text{existing}}\) be the population proportion under the existing approach. The hypotheses are \(H_0:p_{\text{new}}\le p_{\text{existing}}\) and \(H_a:p_{\text{new}}>p_{\text{existing}}\).
There are two parts to the definition. First, the test does not reject \(H_0\). Second, the null hypothesis is actually false. For this particular increase test, the null is false when the program really does increase the population proportion. The test has then missed a real increase.
The decision and the population truth are different kinds of information. The decision comes from the sample and the test procedure. The population truth is what would be found if the relevant population proportions were known. Because those proportions are usually unknown, we cannot tell just from a non-rejection whether a Type II error actually occurred.
Failing to Reject Is Not the Same as Proving No Increase
As in “Making a Decision in a Chi-Square Test,” a test result with a p-value greater than the chosen significance level leads to failing to reject \(H_0\). That decision means the sample did not provide convincing evidence for the alternative. It does not establish that the null hypothesis is true.
For the school program, failing to reject \(H_0\) does not show that the new program has no effect, that the proportions are equal, or that the new program makes submissions less likely. The test might fail to detect an increase even when \(p_{\text{new}}\) is greater than \(p_{\text{existing}}\). If that is the actual population situation, the non-rejection is a Type II error.
The distinction is especially important because the null hypothesis for an increase test includes equality and a decrease. A non-rejection is not a decision that either one is true. It is a decision that the evidence is not strong enough to reject the null in favor of the specified increase.
A Step-by-Step Way to Describe the Error
To write a Type II error in context, begin with the test’s conclusion and then imagine the population truth that would make the null false. Do not reverse the decision: a Type II error involves failing to reject, not rejecting.
Here, the alternative says the new program increases the population proportion of on-time assignment submissions.
Failing to reject means the sample did not provide convincing evidence of the increase.
For this test, the null is false if the actual population proportion under the new program is greater than the proportion under the existing approach.
The school fails to find convincing evidence that the program increases on-time submissions, even though the program actually increases the population proportion.
This way of thinking builds on the four possible outcomes in “Two Possible Errors in a Significance Test.” When the null is false, rejecting it is a correct decision; failing to reject it is a Type II error. When the null is true, failing to reject it is a correct decision; rejecting it is a Type I error.
Worked Examples: Identify a Type II Error
Worked Example: The School’s Assignment Reminder Program
A school tests whether its new reminder program increases the population proportion of students who submit assignments on time. Let \(p_{\text{new}}\) and \(p_{\text{existing}}\) represent the population proportions under the new and existing approaches. The hypotheses are \(H_0:p_{\text{new}}=p_{\text{existing}}\) and \(H_a:p_{\text{new}}>p_{\text{existing}}\). The null parameter space for this one-sided test includes equality and decreases. The school uses \(\alpha=0.05\).
Suppose the test produces a p-value of 0.12. Because \(0.12>0.05\), the school fails to reject \(H_0\). It does not have convincing evidence that the new program increases the population proportion of on-time submissions.
Now suppose the actual population proportions are \(p_{\text{new}}=0.68\) and \(p_{\text{existing}}=0.60\). Since \(0.68>0.60\), the program really does increase the proportion, so the null hypothesis is false. The test failed to reject that false null. This outcome is a Type II error.
In context: the school fails to find convincing evidence that the new reminder program increases the population proportion of students who submit assignments on time, even though the program actually increases that proportion from 0.60 to 0.68. The population values here describe a possible truth; the p-value alone does not tell the school whether that truth applies.
Worked Example: A Community Safety Course
A community offers a new reminder service intended to increase the proportion of residents who complete a safety course. Let \(p_{\text{new}}\) be the population proportion who would complete the course with the reminder service and \(p_{\text{current}}\) the proportion under the current approach. The test is \(H_0:p_{\text{new}}=p_{\text{current}}\) versus \(H_a:p_{\text{new}}>p_{\text{current}}\). The null parameter space for this one-sided test includes equality and decreases.
Suppose a test at \(\alpha=0.05\) gives a p-value of 0.18. Since \(0.18>0.05\), the community fails to reject \(H_0\). The test does not provide convincing evidence that the reminder service increases course completion.
Consider the possible population truth \(p_{\text{new}}=0.54\) and \(p_{\text{current}}=0.50\). The new service actually raises the completion proportion, so \(H_0\) is false. Because the test nevertheless failed to reject \(H_0\), this would be a Type II error.
The contextual description must include both pieces: the test did not find convincing evidence of an increase, but the reminder service actually does increase the population proportion of residents who complete the course. Saying only “the service did not work” would incorrectly turn a test decision into a claim about the population truth.
Worked Example: A Two-Sided Test for a Sports Program
A sports club evaluates whether a new warm-up program changes the population proportion of players who report muscle soreness after practice. Let \(p_{\text{new}}\) be the proportion with the new warm-up and \(p_{\text{usual}}\) the proportion with the usual warm-up. Because the question asks whether the proportion changes in either direction, the hypotheses are \(H_0:p_{\text{new}}=p_{\text{usual}}\) and \(H_a:p_{\text{new}}\ne p_{\text{usual}}\).
Suppose a test at \(\alpha=0.05\) gives a p-value of 0.21. Since \(0.21>0.05\), the club fails to reject \(H_0\). It does not have convincing evidence that the population proportions differ.
Suppose, however, that the actual proportions are \(p_{\text{new}}=0.30\) and \(p_{\text{usual}}=0.42\). These values are not equal, so the null hypothesis is false. The club’s failure to reject the false null is a Type II error: it fails to find convincing evidence that the warm-up program changes the population proportion of players reporting soreness, even though the proportions actually differ.
The direction of the difference is not essential to the definition in this two-sided test. The null is false whenever the proportions are unequal. The Type II error is the missed difference, whether the new proportion is higher or lower.
What Affects the Chance of a Type II Error?
For a particular testing procedure, the probability of a Type II error is often represented by \(\beta\). Because a false null can correspond to many different population values, \(\beta\) depends on which specific alternative is actually true. For example, in the school setting, missing a very small increase and missing a large increase need not be equally likely.
The chance of a Type II error is also affected by features of the study and the test, including the sample size and the chosen significance level. In general, a larger sample makes it easier for a test to detect a real difference of a given size. For a fixed sample size and actual population difference, using a smaller \(\alpha\) makes rejection harder and can increase the chance of failing to detect that difference. These are general patterns, not a way to determine whether an error occurred in a particular study.
The related idea of power is the probability that a test rejects \(H_0\) when a particular alternative is true. For that specific alternative, power and the Type II error probability add to 1. Power describes detecting a real effect; a Type II error describes missing one. Neither can be read directly from the fact that a particular test failed to reject.
Do not confuse \(\beta\) with the p-value. The p-value is calculated from the observed sample result under the assumption that the null hypothesis is true. By contrast, \(\beta\) describes the long-run chance of failing to reject for a specified situation in which the null is false. A single p-value does not give the probability that the test made a Type II error.
Common Mistakes and AP Exam Tip
- Calling non-rejection a Type II error automatically: A non-rejection is a Type II error only if the null is actually false. If the null is true, failing to reject is a correct decision.
- Saying the test proves there is no effect: A full-credit answer says the test did not provide convincing evidence for the alternative. It does not claim the null has been proved.
- Using the wrong decision: A Type II error involves failing to reject a false null. Rejecting a true null is a Type I error.
- Leaving out the population truth: “The school misses an increase” is a start, but specify that the program actually increases the population proportion of on-time submissions.
- Using only equality for an increase test: In the school example, the null includes equality and a lower new proportion. A Type II error requires the opposite: the actual new proportion is greater.
- Interpreting \(\beta\) as the chance this result is wrong: \(\beta\) is a long-run probability for a specified alternative, not the probability that a particular non-rejection is a Type II error.
A strong contextual response for the school example is: “A Type II error would occur if the test fails to find convincing evidence that the new reminder program increases the population proportion of students who submit assignments on time, when the program actually increases that proportion.” This states the test decision and the population truth that makes the null false.
Check Your Understanding
For each situation, decide whether a Type II error has occurred and explain the result in context.
- A test of whether a new tutoring program increases the population proportion of students who pass a course fails to reject \(H_0\). In fact, the new program raises the proportion from 0.70 to 0.76. Describe the possible error.
- A test of whether a new transit alert increases the population proportion of riders who arrive on time fails to reject \(H_0\). The actual proportions are equal. Is this a Type II error? Explain.
- For the school reminder program, state the population relationship that must be true for failing to reject \(H_0:p_{\text{new}}\le p_{\text{existing}}\) to be a Type II error.
- Explain why failing to reject the null does not prove that the new program has no effect.
- For a two-sided test of whether a new practice routine changes an injury proportion, what must be true about the population proportions for a non-rejection to be a Type II error?