Reading a Conclusion Before Accepting It
A hypothesis-test result is often compressed into a sentence: “The result was significant,” “There was no effect,” or “The null hypothesis is probably true.” Each sentence may sound plausible, but it can say more than the test supports. Diagnosing a conclusion means checking whether it describes the p-value correctly, distinguishes evidence from a final verdict, and considers the size of the effect in context.
A p-value has a precise conditional meaning. Under the null hypothesis and the assumptions of the test, it is the probability of obtaining a test statistic at least as extreme as the one observed, in the direction specified by the alternative hypothesis. It is not a general measure of how likely a claim is, nor does it by itself tell us whether an observed effect matters in practice.
In the examples below, each reported test is assumed to be appropriate for its question, with its assumptions checked. The task is not to recalculate the test, but to identify what its result does and does not justify.
Three Questions for Diagnosing a Claim
When reviewing a statement about a test, separate three questions that are often blended together:
- What does the p-value mean? It describes how unusual the observed result, or a more extreme one, would be under \(H_0\) and the test assumptions.
- What decision was made? A decision to reject or fail to reject \(H_0\) depends on the stated significance level \(\alpha\). Failure to reject is not acceptance of \(H_0\).
- How large and consequential is the effect? The estimated effect and its units must be compared with what would matter in the setting. Statistical significance does not make a small effect important.
These questions help keep the conclusion aligned with the evidence. A low p-value can be evidence against \(H_0\) without assigning a probability to the truth of \(H_0\). A high p-value can mean the data do not provide strong evidence against \(H_0\), without showing that the effect is exactly zero. And a low p-value can accompany an effect too small to matter practically.
Why a P-Value Is Not the Probability That the Null Is True
A p-value conditions on the null hypothesis: it asks how often results at least this extreme would occur if \(H_0\) and the test assumptions held. The statement “the probability that \(H_0\) is true, given the result” reverses that conditioning. These are different probabilities.
The distinction can be seen in a simplified setting where the possibilities are \(H_0\) and \(H_a\), and \(E\) denotes a specified event representing the observed evidence. Bayes’ rule gives:
Even in this simplified expression, the probability of \(H_0\) after seeing evidence depends on more than \(P(E \mid H_0)\): it also depends on the probability of the evidence under \(H_a\) and on the probabilities assigned to the hypotheses beforehand. A p-value is a tail probability under \(H_0\), not this posterior probability. Moreover, the event defining a p-value includes outcomes at least as extreme as the observed test statistic; it is not generally the probability of the exact data observed.
Worked Example: Correcting “There Is a 3% Chance the Null Is True”
Hypothetical scenario. A survey analysis tests \(H_0\): the population mean response is 50 units, against a two-sided alternative that it differs from 50. The test is appropriate and its assumptions are checked. The reported p-value is 0.030. A writer concludes, “There is only a 3% chance that the population mean is 50.”
What the number says. The reported p-value is \(0.030=3.0\%\). Under the null hypothesis that the population mean is 50 units, and assuming the test model holds, the probability of obtaining a test statistic at least as far from its null value as the observed statistic is 3.0%.
What the number does not say. It does not give \(P(H_0 \mid \text{data})\), the probability that the population mean is 50 after observing the data. The writer has reversed the conditioning: the p-value concerns the probability of results under \(H_0\), not the probability of \(H_0\) given the results.
Correction. “The test produced a p-value of 0.030. If the population mean were 50 units and the test assumptions held, results at least this extreme would occur with probability 0.030. The result is evidence against \(H_0\), but the p-value is not the probability that \(H_0\) is true.”
Why Nonsignificance Does Not Establish No Effect
A result is statistically significant at level \(\alpha\) when its p-value is at most \(\alpha\); otherwise, the test fails to reject \(H_0\). That rule classifies the result relative to a threshold. It does not convert failure to reject into evidence that the null hypothesis is exactly true.
A test can fail to reject because the effect is small, because the data are variable, because the sample provides limited information, or because several of these factors act together. A nonsignificant result therefore does not, by itself, distinguish “no effect” from “an effect that the data do not estimate precisely enough to detect.” The warranted conclusion is about the strength of evidence from this test, not proof of absence.
Worked Example: Correcting “The Treatment Had No Effect”
Hypothetical scenario. A study compares a treatment group with a control group on a symptom score measured in points. Define the estimated difference as the treatment-group mean minus the control-group mean. The estimated difference is \(2.1\) points, with standard error \(1.6\) points. A suitable two-sided large-sample test of \(H_0\): difference \(=0\) reports \(p=0.189\). At \(\alpha=0.05\), the result is not statistically significant. The report says, “The treatment had no effect.”
Check the test result. The standardized statistic is the estimated difference divided by its standard error:
For a two-sided standard-normal test, this statistic gives \(p \approx 0.189\), as reported. Since \(0.189>0.05\), the test fails to reject \(H_0\) at the 5% significance level.
Diagnose the claim. The decision does not establish that the true treatment effect is zero. The estimated difference is 2.1 points, but the test result does not provide strong enough evidence against a zero difference at the chosen threshold. The estimate and its uncertainty do not justify changing “not statistically significant” into “no effect.”
Correction. “The study did not find statistically significant evidence of a difference in mean symptom score at the 5% level (\(p=0.189\)). The estimated treatment-minus-control difference was 2.1 points; this result does not establish that the true effect is zero.”
Statistical Significance Is Not Practical Importance
Statistical significance describes a result relative to a null model and a chosen significance level. Practical importance asks whether the size of the effect would make a meaningful difference in the setting. These are separate judgments. A small effect can be estimated precisely enough to yield a low p-value, while a larger estimated effect can remain uncertain and yield a higher p-value.
A practical interpretation needs a meaningful scale: for example, the original measurement units and a contextually justified benchmark for what would matter. The p-value alone supplies neither that benchmark nor a measure of how beneficial, harmful, or useful an effect would be. A significant test result is not a substitute for comparing the effect estimate with the consequences that matter in the application.
Worked Example: Correcting “Statistically Significant Means Clinically Important”
Hypothetical scenario. A large, well-conducted study compares mean systolic blood pressure in a treatment group and a control group. Define the effect as treatment minus control. The estimated difference is \(-0.20\) mmHg, with standard error \(0.069\) mmHg. A suitable two-sided large-sample test of \(H_0\): difference \(=0\) reports \(p=0.0037\). Investigators regard a reduction of 5 mmHg as a practically important benchmark in this setting.
Check the reported statistic. The standardized test statistic is:
For a two-sided standard-normal test, \(p \approx 2P(Z \geq 2.90) \approx 2(0.00187)=0.00374\), which rounds to \(0.0037\). Since \(0.0037<0.05\), the result is statistically significant at the 5% level.
Compare with practical importance. The estimated reduction is \(0.20\) mmHg, whereas the stated benchmark is \(5\) mmHg. The estimate is small relative to that benchmark: \(0.20/5=0.04\), or 4% of the benchmark size. The low p-value supports evidence against a zero difference under the test model, but it does not show that the estimated reduction is clinically important.
Correction. “The study found statistically significant evidence of a mean blood-pressure difference (\(p=0.0037\)). The estimated treatment-group mean was 0.20 mmHg lower than the control-group mean, an effect small relative to the 5 mmHg practical-importance benchmark.”
A Practical Diagnosis Routine
A careful conclusion can be built by checking each claim against what the test result actually supplies. Keep the roles of the p-value, the decision rule, and the effect estimate distinct.
State what \(H_0\) claims and what direction or difference \(H_a\) addresses. A p-value is tied to these hypotheses and the test’s assumptions.
Describe how unusual results at least as extreme would be under \(H_0\). Do not turn that conditional probability into a probability that \(H_0\) is true.
Compare the p-value with \(\alpha\), if a threshold is specified. Say “reject” or “fail to reject” \(H_0\), not “prove” or “accept” it.
Give its direction and units, then assess its size against a relevant practical benchmark. Do not infer importance from a small p-value alone.
This routine also helps reveal when a sentence combines a valid observation with an invalid leap. “The result was statistically significant” may correctly report a test decision; adding “so the treatment is important” needs a separate argument about the effect size and context.
Common Mistakes
- Wrong: “A p-value of 0.04 means a 4% chance that \(H_0\) is true.” Correction: It is a tail probability calculated under \(H_0\) and the test assumptions, not a probability assigned to \(H_0\).
- Wrong: “Because \(p>0.05\), there is no effect.” Correction: The test failed to reject at that threshold; it did not prove the effect is zero.
- Wrong: “Because \(p<0.05\), the effect is important.” Correction: Statistical significance and practical importance are different; report the effect size in context.
- Wrong: “The p-value is the probability that chance caused the result.” Correction: It is calculated under a specified null model, not a probability that a particular cause generated the data.
Check Your Understanding
For each statement, identify what is wrong and give a more defensible interpretation.
- A test reports \(p=0.02\), and a researcher says, “There is a 2% probability that the null hypothesis is true.” What conditional probability does the p-value describe instead?
- A study fails to reject \(H_0\) at \(\alpha=0.05\). Why is “the effect is zero” stronger than the test result supports?
- A very large study reports \(p=0.001\) for an estimated effect of \(0.1\) units. What additional contextual comparison is needed before calling the effect important?
- In a two-sided test, the p-value is \(0.07\) and \(\alpha=0.05\). State the decision without saying that \(H_0\) has been proved or accepted.
- Explain why the same p-value need not imply the same practical consequence in two studies measuring different outcomes.