Two Questions Determine the Outcome
A hypothesis test produces a decision about the null hypothesis, but that decision is not the same thing as knowing whether the null hypothesis is true. The distinction matters because a test can make a correct decision or an error, and the label depends on both the decision and the actual state of the null hypothesis.
For example, a study might test whether a treatment has no average effect. The test can reject that claim or fail to reject it. Separately, in the underlying population, the claim of no effect is either true or false. Combining these two possibilities in each dimension gives four outcomes: correct rejection, correct non-rejection, a Type I error, and a Type II error.
“Fail to reject” is the standard wording for the second decision. It means that the evidence did not meet the test’s rule for rejecting \(H_0\). It does not mean that the test has established the null hypothesis as true. This distinction is built into the four-outcome classification.
The Four Outcomes
Place the test decision in one dimension and the actual truth status of \(H_0\) in the other. Their intersection identifies the outcome.
| Actual state of \(H_0\) | Reject \(H_0\) | Fail to reject \(H_0\) |
|---|---|---|
| \(H_0\) is true | Type I error | Correct non-rejection |
| \(H_0\) is false | Correct rejection | Type II error |
The table can also be understood by asking two questions in order: first, what did the test decide? Second, was \(H_0\) in fact true? For instance, the phrase “the test rejected” is not enough to tell whether the outcome was correct: rejection is correct if \(H_0\) is false, but it is a Type I error if \(H_0\) is true.
Write either “reject \(H_0\)” or “fail to reject \(H_0\).” Do not infer the decision from the scientific conclusion alone; identify what the test actually decided.
For classifying a possible outcome, ask whether \(H_0\) is assumed true or false in that scenario. In real applications, this underlying state is generally not known with certainty.
Use the table: rejection with a false null is correct; rejection with a true null is Type I error; non-rejection with a true null is correct; non-rejection with a false null is Type II error.
Worked Example: A Correct Rejection
Suppose a hypothetical test examines whether a new process changes the population mean completion time from 50 minutes. The hypotheses are \(H_0:\mu=50\) minutes and \(H_a:\mu\ne50\) minutes. The test uses \(\alpha=0.05\), gives \(p=0.018\), and the scenario stipulates that the true population mean is 53 minutes. Classify the outcome.
Step 1: Find the decision. Since \(p=0.018<0.05=\alpha\), the test rejects \(H_0\).
Step 2: Identify the actual state. The stipulated true mean is 53 minutes, not 50 minutes. Therefore, \(H_0:\mu=50\) is false.
Step 3: Combine them. The test rejected a false null hypothesis. This is a correct rejection, not an error. In context, the test’s decision to reject the claim that the population mean completion time is 50 minutes agrees with the stipulated population value of 53 minutes.
The fact that the null is stipulated false is part of this hypothetical classification exercise. In an actual study, the observed p-value would not reveal with certainty whether the population mean is exactly 50 minutes.
Worked Example: A Correct Non-Rejection
Consider a hypothetical test of \(H_0:p=0.40\) against \(H_a:p>0.40\), where \(p\) is the population proportion of items meeting a specified quality standard. Let \(\alpha=0.05\), and suppose the test gives \(p\text{-value}=0.31\). For this scenario, assume the actual population proportion is \(0.40\). Classify the outcome.
Step 1: Find the decision. The comparison is \(0.31>0.05\), so the test fails to reject \(H_0\).
Step 2: Identify the actual state. The assumed population proportion is \(0.40\), so \(H_0:p=0.40\) is true.
Step 3: Combine them. The test failed to reject a true null hypothesis. This is a correct non-rejection.
This label does not mean that the test proved the population proportion is \(0.40\). It means that, in this hypothetical scenario, the decision not to reject happens to agree with the stipulated truth. The same test decision could have a different classification if the underlying proportion were not \(0.40\).
Errors Are Defined by Both Truth and Decision
The names of the two errors are easiest to learn by starting with what the test did, then checking whether the null was true. A Type I error occurs when the test rejects \(H_0\) even though \(H_0\) is true. A Type II error occurs when the test fails to reject \(H_0\) even though \(H_0\) is false. Thus, neither error is a decision by itself; each is a particular combination of decision and actual state.
Worked Example: Rejecting a True Null
A hypothetical test evaluates \(H_0:\mu=12\) grams against a two-sided alternative at \(\alpha=0.05\). The test reports \(p=0.032\), while the scenario stipulates that the true population mean is exactly 12 grams. Classify the outcome.
Decision: \(0.032<0.05\), so the test rejects \(H_0\).
Actual state: The stipulated mean is 12 grams, so \(H_0\) is true.
Classification: The test rejected a true \(H_0\), so this is a Type I error. In context, the test concludes that the population mean differs from 12 grams when the scenario says it is actually 12 grams.
The numerical comparison determines the test decision; the stipulated population value determines whether that decision is an error. The p-value alone does not tell us that the null is true or false.
Worked Example: Failing to Reject a False Null
A hypothetical test evaluates \(H_0:\mu=100\) minutes against \(H_a:\mu<100\) minutes at \(\alpha=0.05\). It gives \(p=0.12\), and the scenario stipulates that the actual population mean is 96 minutes. Classify the outcome.
Decision: \(0.12>0.05\), so the test fails to reject \(H_0\).
Actual state: The true mean is 96 minutes, not 100 minutes, so \(H_0\) is false.
Classification: The test failed to reject a false \(H_0\), so this is a Type II error. In context, the test does not find sufficient evidence for a mean below 100 minutes even though the stipulated population mean is 96 minutes.
Why the Four-Outcome Table Is Complete
The classification follows from two yes-or-no distinctions. The test either rejects \(H_0\) or fails to reject it. The null hypothesis is either true or false. There are therefore \(2\times2=4\) combinations. Each combination has exactly one label, as shown in the table.
This also shows why the error labels cannot be swapped. A Type I error belongs only to the cell where \(H_0\) is true and the test rejects it. A Type II error belongs only to the cell where \(H_0\) is false and the test fails to reject it. The two remaining cells are correct decisions. No outcome can be classified from the decision alone, because the same decision appears in two different cells.
For practical interpretation, this framework separates what is observed from what is unknown. Researchers observe data and apply a test, leading to a decision. The actual population state is usually not directly known, which is why it is not generally possible to label a real test result as correct or erroneous with certainty. The four labels describe the possible outcomes and provide a precise vocabulary for discussing test performance.
Common Mistake: Treating Non-Rejection as Proof
A frequent mistake is to say, “The test failed to reject \(H_0\), so \(H_0\) is true.” That reasoning confuses the decision with the actual state. A test can fail to reject a false null, which is exactly a Type II error. A large p-value is not proof that the null hypothesis is true; it indicates that the observed data did not meet the test’s rejection rule.
The correction is to report the decision without overstating what it establishes. For example, say, “The test failed to reject \(H_0\); the data did not provide sufficient evidence for the alternative at the chosen significance level.” Do not say that the null has been accepted or proven. To classify the hypothetical outcome further, you would also need to know whether \(H_0\) is stipulated to be true or false.
Check Your Understanding
For each scenario, identify the test decision, the actual state of \(H_0\), and the resulting outcome.
- A test rejects \(H_0\), and the scenario stipulates that \(H_0\) is false. What is the outcome?
- A test fails to reject \(H_0\), and the scenario stipulates that \(H_0\) is true. What is the outcome?
- A test rejects \(H_0\), and the scenario stipulates that \(H_0\) is true. What kind of error has occurred?
- A test fails to reject \(H_0\), and the scenario stipulates that \(H_0\) is false. What kind of error has occurred?
- Why is “fail to reject \(H_0\)” not equivalent to “\(H_0\) is true”?