The Decision Comes From Comparing the P-Value With Alpha
In “The Significance Level Alpha Explained,” you learned that \(\alpha\) is chosen before the results are examined. Earlier, in “What a P-Value Says About Sample Means,” you learned what the p-value measures under the null hypothesis. This tutorial brings those ideas together: compare the p-value with the planned \(\alpha\) to decide whether to reject the null hypothesis or fail to reject it.
The comparison is a rule, not a new calculation. A p-value at or below alpha meets the preselected threshold for evidence against the null. A p-value above alpha does not meet that threshold. Keep the alternative hypothesis and the context in view when you explain what the decision says.
When you reject \(H_0\), the test provides statistically significant evidence against the null hypothesis at the chosen significance level. In context, say that the data provide convincing evidence for the claim in \(H_a\). When you fail to reject \(H_0\), say that the data do not provide convincing evidence for the claim in \(H_a\) at that significance level.
Failing to reject is not the same as accepting or proving \(H_0\). It means that the result did not cross the chosen evidence threshold. The data may be compatible with the null, or the test may not have provided enough evidence to distinguish the null from the alternative. Do not replace “fail to reject” with “accept,” “prove,” or “show there is no difference.”
What a P-Value of 0.041 Decides
Suppose a valid test about population means produces a p-value of 0.041, rounded. The decision depends on the significance level selected for the test. Compare the same reported p-value with each threshold:
| Preselected significance level | Comparison | Decision |
|---|---|---|
| \(\alpha=0.01\) | \(0.041>0.01\) | Fail to reject \(H_0\) |
| \(\alpha=0.05\) | \(0.041\le0.05\) | Reject \(H_0\) |
Thus, the same test result leads to different decisions under these two different, preselected thresholds. At the 0.01 significance level, the result does not meet the stricter threshold. At the 0.05 level, it does meet the threshold. The p-value has not changed; only the standard for making the decision has changed.
These are alternative decision plans, not an invitation to choose whichever conclusion seems preferable after seeing the result. The appropriate alpha must have been selected before the data were examined. If the planned alpha was 0.01, do not switch to 0.05 because the p-value is 0.041. If the planned alpha was 0.05, do not switch to 0.01 to avoid a rejection.
Worked Example: A Mean Difference With a P-Value of 0.041
A fictional group of community gardeners compares the mean weekly mass of tomatoes harvested from two growing methods. The research question is whether the true population mean harvest differs between the methods. A two-sample t test, reported as valid by the study team, gives a two-sided p-value of 0.041, rounded. Compare the result with two possible significance levels.
Let \(\mu_1\) and \(\mu_2\) be the true mean weekly tomato harvest mass, in kilograms, for all plots using method 1 and method 2, respectively. The hypotheses are \(H_0:\mu_1-\mu_2=0\) and \(H_a:\mu_1-\mu_2\ne0\).
Use a two-sample t test. The plots were selected through independent random samples from the two defined groups, supporting inference to those populations. The samples are less than 10% of their respective populations, supporting the 10% condition. Each group’s plot shows no strong skewness or outliers, so the data are reasonably compatible with t procedures. The reported p-value is for the two-sided alternative, which matches the question about a difference in either direction.
Compare the reported p-value, 0.041, with each proposed alpha. At \(\alpha=0.01\), \(0.041>0.01\), so fail to reject \(H_0\). At \(\alpha=0.05\), \(0.041\le0.05\), so reject \(H_0\).
If the study had preselected \(\alpha=0.01\), these data would not provide convincing evidence that the true mean weekly tomato harvest mass differs between the two growing-method populations. If it had preselected \(\alpha=0.05\), these data would provide convincing evidence of a difference in the true mean weekly harvest masses.
The two conclusions are conditional on which significance level the study planned to use. The team must report the planned alpha rather than select one after seeing 0.041. Neither decision gives the probability that \(H_0\) is true, and the rejection at 0.05 does not describe how large or important the mean difference is.
Worked Decisions for Both Outcomes
A decision sentence should connect the comparison to the research question. It is not enough to write “reject” or “do not reject” without naming the null hypothesis or explaining what the data say about the population parameter. The examples below focus on that final interpretation.
Worked Example: Rejecting at a Strict Significance Level
A fictional environmental team tests whether the true mean nitrate concentration in water from a defined set of wells exceeds a reference value. The team planned a one-sided one-sample t test at \(\alpha=0.01\). The test’s conditions were checked: the wells were randomly sampled, the sample was less than 10% of the target population, and the sample distribution showed no strong skewness or outliers. The calculator reports a p-value of 0.008, rounded.
Let \(\mu\) be the true mean nitrate concentration, in the units used for the reference value, for water from all wells in the target population. The hypotheses are \(H_0:\mu=\mu_0\) and \(H_a:\mu>\mu_0\), where \(\mu_0\) is the reference value.
Compare the p-value with the planned threshold: \(0.008\le0.01\). Therefore, reject \(H_0\). At the 0.01 significance level, the sample provides convincing evidence that the true mean nitrate concentration in the target population of wells exceeds the reference value.
The conclusion follows the direction of \(H_a\). It does not claim that every well exceeds the reference value, that the mean exceeds it by a particular amount, or that there is a 0.8% probability that the null hypothesis is true.
Worked Example: Failing to Reject at a Conventional Level
A fictional school district investigates whether the true mean time students spend traveling to school differs from 25 minutes. It planned a two-sided one-sample t test at \(\alpha=0.05\). The sample was randomly selected, was less than 10% of the district’s student population, and showed no strong skewness or outliers. A valid test reports a p-value of 0.12.
Let \(\mu\) be the true mean travel time, in minutes, for students in the district. The hypotheses are \(H_0:\mu=25\) minutes and \(H_a:\mu\ne25\) minutes. Compare the result with the planned threshold: \(0.12>0.05\). Therefore, fail to reject \(H_0\).
At the 0.05 significance level, these data do not provide convincing evidence that the true mean student travel time differs from 25 minutes. This conclusion does not establish that the mean is exactly 25 minutes. It reports that the evidence from this test did not meet the preselected threshold for rejecting that value.
When the P-Value Equals Alpha or Is Rounded
The decision rule includes equality: a p-value equal to alpha is at the threshold, so reject \(H_0\). For instance, if a test reports \(p=0.05\) and the planned significance level is \(\alpha=0.05\), then \(p\le\alpha\), and the decision is to reject. State the comparison explicitly when the values are equal.
Be careful when a p-value is rounded. A displayed value may conceal a slightly smaller or larger unrounded value, which matters when it is extremely close to alpha. If your calculator or software provides more digits, compare the unrounded p-value with the planned alpha. If only the rounded value is available and the comparison is right at the boundary, report the precision limitation rather than pretending the displayed digits settle it exactly.
Worked Example: A Reported P-Value at the Boundary
A fictional sports program tests whether the true mean time to complete a fitness course differs from a target time. The team preselected \(\alpha=0.05\), and the test reports \(p=0.050\) to three decimal places.
If 0.050 is the exact p-value, then \(p=\alpha\), so reject \(H_0\). The conclusion is that the data provide statistically significant, convincing evidence at the 0.05 level that the true mean course time differs from the target. If 0.050 is rounded from an unreported value, ask for additional digits before deciding: the exact value could be just below or just above 0.05. Do not silently treat a rounded boundary value as more precise than it is.
Common Mistakes and AP Exam Tips
- Reversing the comparison: A small p-value, at or below alpha, leads to rejection. A p-value above alpha leads to failing to reject. Write the inequality before stating the decision if you are unsure.
- Saying “accept the null”: Failing to reject does not prove the null hypothesis. A full-credit conclusion says the data do not provide convincing evidence for the alternative claim at the chosen significance level.
- Leaving out the context: “Reject \(H_0\)” alone is incomplete as an interpretation. Identify the population parameter and connect the decision to the research question, including the direction named by the alternative.
- Changing alpha after seeing the p-value: This changes the standard to fit the result. State the alpha chosen in advance and use that value for the decision.
- Treating rejection as proof or as practical importance: Rejecting \(H_0\) is evidence against it, not certainty. Statistical significance by itself does not show that a difference is large or important in practice.
- Ignoring a rounded boundary result: If the displayed p-value equals alpha after rounding, the unrounded value may determine the decision. Use available precision and explain any limitation.
For a strong AP response, show the comparison, name the decision, and finish with a sentence in context. For example: “Because the p-value of 0.041 is less than the preselected \(\alpha=0.05\), reject \(H_0\). The data provide convincing evidence that the true population mean harvest masses differ between the two growing methods.” For a p-value above alpha, keep the same structure but use “fail to reject” and “do not provide convincing evidence.”
Check Your Understanding
For each situation, use the stated alpha and write a decision in context where enough information is provided.
- A two-sided test about two population means has \(p=0.041\). What is the decision at \(\alpha=0.01\), and what is it at \(\alpha=0.05\)?
- A valid test about a population mean has \(p=0.007\) and planned \(\alpha=0.01\). State the decision and describe the evidence using the alternative hypothesis.
- A two-sided test has \(p=0.18\) and planned \(\alpha=0.05\). Write an appropriate conclusion and explain why it does not prove the null hypothesis.
- If the exact p-value is 0.03 and the planned alpha is 0.03, what is the decision under the stated rule?
- Why is it not appropriate to change a planned \(\alpha=0.01\) to \(\alpha=0.05\) after observing \(p=0.041\)?