Why a Nonsignificant Result Does Not Prove No Effect

What a Nonsignificant Result Does—and Does Not—Say

A hypothesis test can end without rejecting the null hypothesis. That result is often described as “nonsignificant,” but the word does not mean that the data have shown an effect to be zero. It means the data did not cross the test’s rejection threshold. Whether that is weak evidence about the effect or evidence that effects are small depends on the study’s precision and on which effect sizes it could reasonably detect.

The distinction matters whenever a study asks whether a treatment, policy, or exposure changes a quantitative outcome. A small study may miss an important effect because its estimate is imprecise. A much larger study may produce a precise estimate close to zero, making effects large enough to matter less plausible. The same test outcome—failure to reject—can therefore have different interpretations.

A Model for Interpreting the Test

To make the reasoning concrete, consider a two-sided one-sample z test for a population mean change, \(\theta\), measured in outcome points. Suppose the \(n\) observed changes are independent, come from a representative sampling process, and have a normal distribution with known population standard deviation \(\sigma\). Under this model, the sample mean has distribution \(\bar{x}\sim N(\theta,\sigma^2/n)\). If the sample is large enough for a normal approximation, that can replace the normal-population assumption.

The test compares \(H_0:\theta=0\) with \(H_a:\theta\ne 0\). Its standard error is \(SE=\sigma/\sqrt{n}\), and the test statistic is \(z=\bar{x}/SE\). At \(\alpha=0.05\), the two-sided test rejects when \(|z|>1.96\). A \(p\)-value above 0.05 means only that the observed statistic is not sufficiently far from zero under this rule. The test does not directly calculate the probability that \(\theta=0\).

Power depends on a particular alternative value, not just on the sample size or on the word “nonsignificant.” For a two-sided z test, the critical boundary for the sample mean is \(c=1.96SE\). If the true mean is \(\delta\), power is the probability that the sample mean falls above \(c\) or below \(-c\):

Here \(\Phi\) is the cumulative distribution function of the standard normal distribution, and \(P_\delta\) means that probabilities are calculated assuming the true mean is \(\delta\). This formula shows why a nonzero effect can go undetected: the distribution of sample means under that effect can still overlap the nonrejection region.

Why Failure to Reject Cannot Establish Zero

For the test above, failure to reject occurs whenever \(-c\leq\bar{x}\leq c\). If the true mean is \(\delta\), the probability of that event is

For a finite standard error, a normal distribution centered at a nonzero \(\delta\) still assigns positive probability to this interval. Thus, even when a real effect exists, a study can fail to reject the null. Low power makes that outcome more common for the effect sizes of interest; it does not make the null more likely to be true.

A related useful summary is a confidence interval. Under this z model, a 95% confidence interval for \(\theta\) is \(\bar{x}\pm1.96SE\). In repeated use, this procedure produces intervals that contain the fixed true mean 95% of the time, under the model assumptions. For a two-sided z test at the 5% level, the interval contains zero exactly when the test does not reject \(H_0:\theta=0\). The interval also shows which nonzero values remain compatible with the data at that level.

High Power for a Target Is Not Power for Every Effect

A study can have high planned power to detect one prespecified effect and still produce a nonsignificant result. In that case, the result can weigh against effects as large as the target, while leaving smaller effects plausible. Power is not a guarantee that every effect will be detected, nor does high power for one effect imply high power for all effects.

When Can a Result Support “Practically Small”?

“Not statistically significant” and “too small to matter” are different claims. The second requires a definition of what size of effect would be practically important. Let \(m>0\) be a prespecified margin in outcome units, with effects between \(-m\) and \(m\) considered negligible for the application. A confidence interval entirely within that range indicates that the estimated effect is sufficiently precise to make effects outside the chosen margin less compatible with the data, at the interval’s stated confidence level.

This is a precision-based interpretation, not proof that the effect is zero. The margin should be justified in context and chosen without looking at the observed result. If the interval extends beyond the margin, the study may not distinguish negligible effects from important ones—even if the test against zero is nonsignificant. If the entire interval lies within the margin, that is stronger evidence that any effect is small enough to be practically negligible under the chosen definition.

A Practical Interpretation Checklist

When a test is nonsignificant, do not stop at the \(p\)-value. First confirm the test’s model and assumptions, then ask what effects the design was intended to detect and how precise the estimate turned out to be. Planned power is calculated for specified alternatives under planning assumptions; it is not a universal property that answers every question about the observed effect.

Common Mistake

A common mistake is to reason, “The test was nonsignificant, so there was no effect.” This confuses a decision rule with a conclusion about the true parameter. A test can fail to reject because the estimate is close to zero, because the data are variable, because the sample is small, or because the effect is smaller than the study could detect. The correction is to report the failure to reject, describe the estimate and its uncertainty, and relate the interval and planned power to effects that matter in context.

The reverse overstatement is also risky: high planned power for a chosen effect does not prove no effect when the result is nonsignificant. It only says that, under the specified model and planning assumptions, the test would usually reject if that particular effect were true. Smaller effects may still be plausible, and a practical conclusion requires precision relative to a meaningful threshold.