Two Questions Identify the Error
In “Two Possible Errors in a Significance Test,” “Type I Error Defined in Context,” and “Type II Error Defined in Context,” you learned the definitions of the two errors. This tutorial practices applying them when a scenario gives you a test decision and information about what is actually true.
The main challenge is keeping two things separate. A test’s decision comes from the sample evidence: reject \(H_0\), or fail to reject \(H_0\). The actual truth concerns the population or situation being studied: \(H_0\) is true, or \(H_0\) is false. An error occurs only when the test’s decision does not match that truth.
A useful technique is to make a small decision-and-truth grid before writing your answer. It is the same four-outcome structure introduced earlier in this course, organized to help you check a scenario systematically.
| Test decision | \(H_0\) is actually true | \(H_0\) is actually false |
|---|---|---|
| Reject \(H_0\) | Type I error | Correct decision |
| Fail to reject \(H_0\) | Correct decision | Type II error |
Read a scenario in this order: first, identify what the test decided; second, identify the actual truth; third, match that pair to the grid. The wording “the test found convincing evidence” usually signals rejection of \(H_0\). “The test did not find convincing evidence” usually signals that the test failed to reject \(H_0\). Then check whether the scenario tells you that the null is actually true or false.
Worked Examples: Medical and Quality-Control Decisions
Worked Example: A Medication and Migraine Symptoms
A research team tests whether a new medication reduces the population proportion of patients who experience migraines during a month. Let \(p_{\text{new}}\) be the proportion who would experience migraines with the new medication, and \(p_{\text{usual}}\) the proportion under usual care. The hypotheses are \(H_0:p_{\text{new}}=p_{\text{usual}}\) and \(H_a:p_{\text{new}}<p_{\text{usual}}\).
In a particular study, the test rejects \(H_0\). Later, suppose the actual population proportions are found to be \(p_{\text{new}}=0.31\) and \(p_{\text{usual}}=0.31\). Did the test make an error? If so, which kind?
Identify the decision: The test rejected \(H_0\), so it found convincing evidence that the new medication reduces the population proportion of patients who experience migraines.
Identify the truth: The actual proportions are equal. Because \(0.31\) is not less than \(0.31\), the alternative is false and \(H_0\) is true. The test rejected a true null hypothesis, so this is a Type I error.
In context: The research team concludes that the new medication reduces the population proportion of patients who experience migraines, even though the actual proportion is the same with the new medication as with usual care. The error is not simply that the study’s result was “wrong”; it is specifically a rejection of a true null hypothesis.
Now change only the actual population truth: suppose \(p_{\text{new}}=0.24\) and \(p_{\text{usual}}=0.31\), while the test still rejects \(H_0\). The null is false, and rejecting it is the correct decision. A test result alone cannot tell you which of these possible truths applies.
Worked Example: A Production Line’s Defect Rate
A factory monitors the proportion \(p\) of containers from a production line that have a seal defect. The quality standard is that the defect rate should not exceed \(0.02\). A test is designed to detect whether the rate is above the standard. For the test, the hypotheses are \(H_0:p=0.02\) versus \(H_a:p>0.02\); rates at or below \(0.02\) meet the standard and are treated as the null-side condition for classifying errors.
Suppose the test fails to reject \(H_0\). A later, reliable assessment establishes that the actual defect rate is \(p=0.035\). Classify the result.
Identify the decision: The test failed to reject \(H_0\). The factory did not find convincing evidence that the population defect rate exceeds \(0.02\).
Identify the truth: The actual defect rate is \(0.035\), which is greater than \(0.02\). Therefore, \(H_0\) is false. The test failed to reject a false null hypothesis, so this is a Type II error.
In context: The factory fails to find convincing evidence that more than \(2\%\) of containers have defective seals, even though the actual population defect rate is \(3.5\%\). The test has failed to detect a defect rate above the quality standard.
The phrase “the test failed to reject” does not mean the factory established that the defect rate is at or below \(2\%\). If the actual rate really were \(0.02\) or lower, failing to reject would be a correct decision. It is the additional information that the rate is actually \(0.035\) that makes this particular outcome a Type II error.
Worked Example: A Defect Rate That Meets the Standard
Use the same factory hypotheses: \(H_0:p=0.02\) versus \(H_a:p>0.02\), with rates at or below \(0.02\) treated as the null-side condition for classifying errors. This time, the test rejects \(H_0\), but the actual population defect rate is \(p=0.015\).
Identify the decision: The test rejected \(H_0\), so it found convincing evidence that the defect rate exceeds \(2\%\).
Identify the truth: The actual rate, \(0.015\), is less than \(0.02\), so it meets the standard and is on the null side of the test. The test rejected that null-side condition, which is a Type I error.
In context: The factory concludes that more than \(2\%\) of its containers have defective seals, even though the actual population defect rate is \(1.5\%\) and meets the standard. This could lead the factory to investigate or adjust a production line unnecessarily.
Compare this example with the previous one. The hypotheses and setting are unchanged, but the combination of decision and truth differs. That is why you should not memorize that “a missed defect is always Type II” or that “a rejected null is always Type I.” The truth about \(H_0\) determines whether the decision is an error.
Worked Example: The Legal Analogy
Worked Example: A Verdict and the Actual Truth
A common teaching analogy compares a significance test with a simplified legal decision. In the analogy, treat \(H_0\) as “the defendant is innocent” and the alternative as “the defendant is guilty.” A decision to reject \(H_0\) corresponds roughly to a guilty verdict; failing to reject \(H_0\) corresponds roughly to a not-guilty verdict.
Suppose a jury returns a guilty verdict, but the defendant is in fact innocent. The decision rejects the null of innocence even though that null is true. In the analogy, this is a Type I error: convicting an innocent person.
Now suppose the jury returns a not-guilty verdict, but the defendant is in fact guilty. The decision fails to reject the null of innocence even though that null is false. In the analogy, this is a Type II error: failing to convict a guilty person.
The analogy helps with the decision-and-truth grid, but it should not be taken literally. A courtroom verdict is not a statistical significance test, and legal standards and procedures are not the same as choosing a test’s significance level. Also, “not guilty” is not the same as proving “innocent.” For the purpose of the analogy, it represents a decision not to reject the null, not certainty about the actual truth.
Turn the Scenario Into a Full-Credit Statement
An answer that names only “Type I” or “Type II” may not show that you understand the situation. A clear response includes the decision, the relevant truth, and the context. If the question asks you to describe a possible error, use conditional wording such as “A Type II error would occur if…” unless the scenario explicitly tells you the error has occurred.
Identify the claim represented by \(H_0\) and what the alternative claims. This tells you what it means for the null to be false.
Look for whether the test rejected \(H_0\) or failed to reject it. Translate that decision into the scenario’s language.
Use the information the scenario gives about reality. Decide whether that information makes \(H_0\) true or false.
Use the grid to identify an error, if there is one, then state both parts in context.
For example, “The factory made a Type II error” is less informative than: “The factory fails to find convincing evidence that the population defect rate exceeds \(2\%\), even though the actual defect rate is above \(2\%\).” The second statement makes clear both why the null is false and what the test missed.
Common Mistakes and AP Exam Tip
- Calling every rejection a Type I error: Rejection is a Type I error only when \(H_0\) is actually true. If \(H_0\) is false, rejection is the correct decision.
- Calling every non-rejection a Type II error: A Type II error requires a false null. If \(H_0\) is true, failing to reject is the correct decision.
- Mixing up the decision and the truth: A test decision comes from the data; the truth is about the population or actual situation. Keep them in separate steps before matching them.
- Changing the direction of the claim: In the factory example, the alternative is \(p>0.02\). An actual rate at or below \(0.02\) meets the standard and is on the null side; an actual rate above \(0.02\) is on the alternative side.
- Saying non-rejection proves the null: Write that the test did not provide convincing evidence for the alternative. Do not claim it proved the defect rate is acceptable or that a treatment has no effect.
- Using an error label without context: For full-credit communication, name the decision and describe the actual condition that makes it an error, using the population and outcome in the scenario.
- Treating the legal analogy as literal: Use it to remember the four outcomes, but do not suggest that a legal verdict is itself a statistical test or that “not guilty” proves innocence.
Check Your Understanding
For each situation, identify the decision, determine whether \(H_0\) is actually true or false, and state whether an error occurred.
- A medical study tests whether a treatment lowers the population proportion of patients with a symptom. The test rejects \(H_0\), but the actual symptom proportions are equal. What kind of error, if any, occurred? Describe it in context.
- A factory tests whether its population defect rate exceeds \(1\%\). It fails to reject \(H_0\), even though the actual defect rate is \(1.8\%\). Identify the error and explain why \(H_0\) is false.
- A test of whether a new screening method increases the population proportion of detected cases fails to reject \(H_0\). The actual proportion is higher with the new method. Is this a Type I error, a Type II error, or a correct decision? Explain.
- In the simplified legal analogy, a defendant is actually guilty but receives a not-guilty verdict. Which error does the analogy represent, and what does the verdict correspond to?
- Explain why a test’s failure to reject \(H_0\), by itself, is not enough to conclude that a Type II error occurred.