From a Test Result to a Defensible Conclusion
A significance test does not produce a verdict that a research claim is true or false. It asks a narrower question: if the null hypothesis were true, how surprising would the observed data, or data more extreme, be? A useful conclusion therefore does more than announce that a result is “significant.” It identifies the research question, states what the data provide evidence for, and respects what the study design and test cannot establish.
The wording matters because the same test result can be overinterpreted in several ways. A small p-value is not proof of a hypothesis, and a large p-value is not proof that two groups are equal. Even strong evidence of a difference between groups does not, by itself, establish that one group’s treatment caused the difference. The appropriate conclusion depends both on the test result and on how the data were collected.
What the Decision Does—and Does Not—Say
A significance test begins with a null hypothesis \(H_0\), which represents a specified model such as equal population proportions or no population mean difference. The alternative hypothesis \(H_a\) describes the departure relevant to the research question. The p-value measures how often a test statistic at least as extreme as the observed statistic would occur if \(H_0\) were true and the test model’s assumptions held.
Before interpreting a result, compare the p-value with the significance level chosen for the test. If the p-value is at most \(\alpha\), reject \(H_0\). If it is greater than \(\alpha\), fail to reject \(H_0\). The phrase “fail to reject” is deliberate: the test did not provide sufficiently strong evidence against the null at that threshold. It did not show that the null is true.
A p-value also is not the probability that \(H_0\) is true. Its conditioning goes in a different direction: it is a probability about data, calculated under the assumption that \(H_0\) holds. In symbols, a p-value concerns a probability such as \(P(\text{test statistic at least this extreme}\mid H_0)\). That is not \(P(H_0\mid\text{observed data})\). Reversing those statements changes what is being conditioned on.
Likewise, “statistically significant” does not automatically mean “important.” A small difference may be statistically detectable with enough information but have little practical consequence. A conclusion should report the observed difference in context, with units, rather than relying on the word “significant” to describe its size or value.
Worked Conclusions for Common Test Results
Worked Example: Evidence of a Difference in a Randomized Trial
In a hypothetical randomized trial, 100 participants are assigned to a new reminder system and 100 to standard practice. Completion within a week is recorded: 50 participants in the reminder group complete the task, compared with 25 in the standard-practice group. Test whether the population completion proportions differ, using a two-sided test at \(\alpha=0.05\).
Let \(p_1\) and \(p_2\) be the completion proportions for the two assigned groups. The hypotheses are \(H_0:p_1=p_2\) and \(H_a:p_1\ne p_2\). The observed proportions are \(\hat{p}_1=50/100=0.50\) and \(\hat{p}_2=25/100=0.25\), a difference of \(0.25\), or 25 percentage points.
For the pooled two-proportion test, the null model uses the combined estimate \(\hat{p}=(50+25)/(100+100)=0.375\). The conditions are plausible for this hypothetical example: participants were randomly assigned, each participant contributes one outcome, and the two assigned groups are independent. Under the null, the expected numbers of completions and non-completions in each group are 37.5 and 62.5, respectively, so the expected counts are adequate for the normal approximation.
The pooled standard error and test statistic are:
The two-sided p-value is approximately \(0.00026\), which is less than \(0.05\). Reject \(H_0\). The data provide strong evidence that completion proportions differ between participants assigned to the reminder system and those assigned to standard practice. Because assignment was randomized, the difference supports a causal conclusion about the reminder system for participants in this trial: the system increased completion in this group. The result alone does not show that the same effect will occur in every population or setting.
Worked Example: A Nonsignificant Difference in Sample Means
In a hypothetical study, two independent random samples of 25 adults each report their average sleep duration. The first group has \(\bar{x}_1=6.8\) hours and \(s_1=1.5\) hours; the second has \(\bar{x}_2=6.2\) hours and \(s_2=1.6\) hours. Test for a difference in population means using a two-sided Welch t test at \(\alpha=0.05\).
Let \(\mu_1\) and \(\mu_2\) be the population mean sleep durations represented by the samples. The hypotheses are \(H_0:\mu_1-\mu_2=0\) and \(H_a:\mu_1-\mu_2\ne0\). Assume the samples were selected independently and randomly, and that the sleep-duration distributions have no severe skewness or extreme outliers. With 25 observations per group and those distribution conditions, a Welch t procedure is reasonable; it does not require equal population variances.
The observed mean difference is \(6.8-6.2=0.6\) hours. The standard error and test statistic are:
The Welch degrees of freedom are approximately \(47.8\), giving a two-sided p-value of about \(0.178\). Since \(0.178>0.05\), fail to reject \(H_0\). These samples do not provide sufficient evidence at the 5% level that the population mean sleep durations differ. The observed sample means are 0.6 hours apart, but the test does not prove that the population means are equal, nor does it establish that any difference is exactly zero.
Worked Example: A Significant Association in an Observational Study
A hypothetical survey independently samples 80 night-shift workers and 80 day-shift workers at several workplaces. Of those sampled, 44 night-shift workers and 24 day-shift workers report frequent fatigue. Test whether the population proportions reporting frequent fatigue differ, using \(\alpha=0.05\).
Let \(p_1\) and \(p_2\) denote the proportions reporting frequent fatigue among night-shift and day-shift workers, respectively. Use \(H_0:p_1=p_2\) and \(H_a:p_1\ne p_2\). The observed proportions are \(44/80=0.55\) and \(24/80=0.30\), a difference of 0.25, or 25 percentage points. The samples are independent and each worker contributes one response. Under \(H_0\), the pooled estimate is \((44+24)/(80+80)=0.425\); the expected counts are 34 and 46 in each group, so the count condition is satisfied.
The pooled standard error and test statistic are:
The two-sided p-value is approximately \(0.0014\), below \(0.05\). Reject \(H_0\): the survey provides evidence of an association between shift type and frequent fatigue among the population represented by the survey. However, shift type was observed, not randomly assigned. Other differences between workers or workplaces could help explain the association. The result does not establish that working a night shift causes fatigue.
A Short Justification for Cautious Language
The test decision follows from comparing a probability under the null model with a preselected threshold. It is a rule for describing evidence, not a rule that converts uncertainty into certainty. The reasoning can be written in three steps:
Assume \(H_0\) and the test assumptions. This determines the reference distribution of the test statistic.
The p-value is the probability, under \(H_0\), of a test statistic at least as extreme as the observed one, in the direction specified by \(H_a\).
If the p-value is at most \(\alpha\), the result is evidence against \(H_0\). Otherwise, evidence against \(H_0\) is insufficient at that threshold. Neither outcome proves a hypothesis.
This logic explains why the conclusion must preserve the phrase “under the null model” when interpreting a p-value. A probability computed assuming \(H_0\) is true cannot, without additional assumptions, be relabeled as a probability that \(H_0\) is true. It also explains why a nonsignificant test cannot prove equality: the decision merely says the observed evidence did not cross the chosen threshold.
Match the Claim to the Study Design
A test can address whether a difference or association is consistent with random variation under a null model. It cannot repair limitations in how observations were collected. Random assignment can support causal interpretation because it creates comparable treatment groups on average, subject to the study’s execution and scope. Random sampling supports generalization to the population sampled from, subject to sampling and nonresponse limitations. These are distinct design features: random assignment concerns cause, while random sampling concerns representation.
In an observational study, groups may differ in characteristics other than the one being compared. A significant result can support the claim that two variables are associated in the population represented by the data, but it does not isolate a causal effect. In a randomized experiment, a difference between groups can support a causal interpretation of assignment to the treatment, but generalizing beyond the participants still requires justification.
A well-formed conclusion usually names the groups or population, describes the evidence in the direction of the research question, and includes the decision at the chosen significance level. For example: “At the 5% level, the data provide evidence that the two population proportions differ.” Add a cautious design qualification when needed: “Because this was an observational survey, the association does not establish that one variable causes the other.” Avoid stating only “the result is significant”; that leaves the reader unsure what was tested and what the finding means.
Common Mistakes
- “The p-value is 0.18, so there is an 18% chance the null is true.” This reverses the conditional probability. The p-value is calculated assuming the null model, and is not the probability that the null is true.
- “We failed to reject, so the groups are equal.” A nonsignificant result may reflect limited information or a small difference. State that evidence of a difference was insufficient; do not claim equality unless the study and method address that question directly.
- “The result is significant, so the effect is important.” Statistical significance describes evidence relative to a model and threshold, not the practical size or value of an effect. State the observed difference in context and units.
- “The groups differ, so the treatment caused the difference.” A test result alone does not establish causation. Check whether the study used random assignment; observational evidence supports an association, not by itself a causal claim.
- “The result applies to everyone.” A conclusion should match the population the data can represent. Random assignment does not automatically make a sample representative of a broader population.
Check Your Understanding
For each item, focus on what the test result supports and what it does not establish.
- A test has \(p=0.032\) and \(\alpha=0.05\). State the decision and describe what the p-value does not mean.
- A two-sided test gives \(p=0.41\). Why is “the two population means are equal” an unjustified conclusion?
- A survey finds a statistically significant association between exercise frequency and a health outcome. What can be concluded, and why is a causal claim not warranted from this result alone?
- Explain the difference between random assignment and random sampling in relation to causal conclusions and generalization.
- A very large sample produces a statistically significant difference of 0.02 units. Why should the conclusion not equate statistical significance with practical importance?