From a Test Result to a Meaningful Conclusion
A test result is useful only when its numbers are connected to the question that prompted the analysis. A p-value by itself does not say what was measured, which claim was tested, or what the sample showed. A clear interpretation brings together three things: the null claim, the observed result, and the strength of evidence against that claim.
This tutorial works through two common settings. In a campus survey, the quantity of interest is a population proportion. In a treatment study, the quantity of interest is a difference between population means. The calculations that produce each test result are taken as given; the focus is on interpreting the results accurately and in context.
The wording “at least as extreme” depends on the alternative hypothesis. For a lower-tailed test, more extreme results are smaller values of the statistic; for an upper-tailed test, they are larger values. For a two-sided test, results far from the null value in either direction count as extreme. Before interpreting a p-value, identify the alternative so you know which outcomes it includes.
A Reliable Interpretation Sequence
A context-specific conclusion can be built in a few steps. First identify the population quantity and the null claim. Then report what the sample actually showed, including its direction and units. Next interpret the p-value under the null model. Finally, state what the evidence supports—and what it does not establish.
State the parameter and the value or relationship assumed under \(H_0\). Give the alternative in the question’s direction.
Give the sample proportion or difference in sample means, with its direction and units. This describes the data, not the p-value.
Explain how often results this extreme or more extreme would occur under \(H_0\), assuming the test’s model and conditions hold.
Describe the strength of evidence about the population claim. Do not turn evidence into proof, certainty, or a claim that the null is true.
The interpretation has a useful logical form. If \(T\) is the statistic used in a test and \(t_{\text{obs}}\) is its observed value, a one-sided upper-tail p-value is
The condition after the vertical bar matters: the probability describes results generated under the null claim and model. It does not describe the probability of the null claim itself. The direction of the inequality also matters: it follows from the alternative and observed statistic, not from whichever direction makes the p-value smaller.
Worked Example: A Campus Survey About Shuttle Use
A campus announcement claims that 60% of undergraduates use the campus shuttle at least once a week. A hypothetical simple random sample of 250 undergraduates includes 133 who report weekly shuttle use. A valid lower-tailed test reports \(p\text{-value}=0.017\). Interpret the result for the claim that the population proportion is below 60%.
Population quantity and hypotheses. Let \(p\) be the proportion of all undergraduates at this campus who use the shuttle at least once a week. The claim being investigated is whether the proportion is below 0.60:
Observed sample result. The sample proportion is
Thus, 53.2% of the sampled students reported weekly shuttle use. The sample result is 0.068, or 6.8 percentage points, below the null value:
What the p-value means. Under the null model that the population proportion is 0.60, a lower-tailed p-value of 0.017 means there is about a 1.7% chance of obtaining a sample result at least this far below 0.60, or still farther below it, under the test’s assumptions. This is not a 1.7% chance that the null claim is true.
Conclusion. The result provides evidence against the claim that 60% of undergraduates use the shuttle weekly, in the direction that the proportion is lower. It does not show that exactly 53.2% of all undergraduates use the shuttle; 53.2% is the sample proportion. The inference depends on the sampling and response process being suitable. For example, substantial nonresponse related to shuttle use could make the respondents unrepresentative even if the initial sample was random.
Interpreting a Difference Between Treatment Means
In a treatment study, the sample difference in means gives both direction and magnitude for the observed group results. The p-value answers a different question: how unusual a test statistic at least as extreme as the one observed would be under the null model. Keeping these roles separate prevents a p-value from being mistaken for a measure of the treatment’s size.
For a study comparing a treatment group with a control group, let \(\mu_T\) and \(\mu_C\) denote their population means. If the research question is whether the treatment improves an outcome, a suitable one-sided alternative is \(\mu_T-\mu_C>0\). The null claim is that the population means are equal. Before interpreting a test output, the study design and analysis assumptions should be considered. For the example below, assume treatment was randomly assigned, participants’ outcomes are independent, and the group data are suitable for the stated Welch two-sample t procedure (in particular, there are no severe outliers or problematic distribution shapes for these sample sizes).
Worked Example: A Treatment’s Effect on Quiz Scores
In a hypothetical randomized study, 20 students receive a study-skills intervention and 20 receive usual study guidance. Their quiz scores are measured in points. The intervention group’s mean is 78.6 points and the control group’s mean is 74.4 points. A prespecified upper-tailed Welch t-test reports \(t=2.18\), \(df=37.9\), and \(p\text{-value}=0.0178\). Interpret the result for the claim that the intervention raises the population mean score.
Claim. Let \(\mu_T\) be the population mean score for students receiving the intervention and \(\mu_C\) the population mean score for students receiving usual guidance. The hypotheses are
Observed difference. The estimated treatment-minus-control difference is
The observed mean score is 4.2 points higher in the intervention group. This is the magnitude and direction of the sample difference; it is not a sample size.
Evidence and conclusion. Under the null claim of equal population means and the test’s assumptions, a t statistic of 2.18 or greater in the direction of improvement has probability 0.0178. The result is evidence that the intervention group’s population mean score is higher than the control group’s, consistent with the study’s improvement claim. Because treatment was randomly assigned, a causal interpretation can be appropriate for this study population if the design was carried out as described and other relevant conditions hold. The result does not prove the intervention improves every student’s score or guarantee the same effect in other settings.
When the Evidence Is Not Strong
A relatively large p-value does not establish the null claim. It says that, if the null and test model hold, results at least as extreme as those observed are not unusual enough to provide strong evidence against the null. The sample can still show a difference in one direction; the p-value indicates that the evidence for a population difference is limited.
Worked Example: A Treatment Result That Falls Short of the Evidence Standard
In a second hypothetical randomized study, the intervention group has 18 students with a mean score of 76.1 points, while the control group has 18 students with a mean score of 74.0 points. The planned question is whether the intervention raises mean scores. Assume the design and Welch t-test assumptions are suitable. The reported test output is \(t=1.58\), \(df=32\), and one-sided \(p\text{-value}=0.062\). The study uses \(\alpha=0.05\).
The estimated difference is
The sample mean is 2.1 points higher in the intervention group. Under the null claim of equal population means, the probability of obtaining a t statistic of 1.58 or larger in the improvement direction is 0.062, or 6.2%, under the stated model.
Since \(0.062>0.05\), the result does not meet the study’s prespecified threshold for rejecting \(H_0\). There is insufficient evidence from this study to conclude that the intervention raises the population mean score. This is not evidence that the two population means are exactly equal, and it does not erase the observed 2.1-point sample difference.
Why the Wording Matters
A sound interpretation keeps three statements distinct:
- What the sample showed: for example, a shuttle-use proportion of 0.532 or a treatment-minus-control mean difference of 4.2 points.
- What the p-value says: how often a result at least as extreme would occur under the null claim and test model.
- What the conclusion supports: whether the result provides evidence for or against the population claim, in the direction specified by the alternative.
The p-value does not report the effect’s magnitude. In the treatment example, the 4.2-point difference describes the sample effect’s magnitude and direction; the p-value describes how unusual the test result would be under the null model. Both numbers can be useful, but they answer different questions.
Common mistake: “The p-value is 0.062, so there is a 6.2% chance the treatment has no effect.” This reverses the conditional statement. The 6.2% is a probability of results at least this extreme assuming the null claim and model. A better conclusion is: “The result does not provide sufficient evidence at the 0.05 level that the intervention raises mean quiz scores.” That wording reports the evidence without claiming the null is true.
A strong conclusion names the population, the outcome, and the direction of the evidence. It also avoids claiming more than the design supports: a survey describes a population only insofar as its sample and response process support that inference, while a treatment study’s causal interpretation depends on its design.
Check Your Understanding
For each question, focus on the distinction between the observed result, the p-value, and the population conclusion.
- A campus survey finds \(\hat{p}=0.41\) for weekly bicycle commuting. In a test of \(H_0:p=0.45\) versus \(H_a:p<0.45\), the p-value is 0.03. What does the p-value mean in context?
- A treatment group’s sample mean is 3.5 points above the control group’s mean. Which number describes the observed magnitude and direction: the mean difference or the p-value? Explain.
- A treatment study tests whether an intervention raises mean scores and reports \(p\text{-value}=0.12\) at \(\alpha=0.05\). Write a conclusion that does not claim the population means are equal.
- Why is “there is a 3% probability that the null hypothesis is true” not a correct interpretation of a p-value of 0.03?
- What information must be identified before deciding which outcomes count as “at least as extreme” for a p-value?