Failure to Reject Is Not Acceptance

What a Nonsignificant Result Says

A test can produce a result that does not cross its prespecified significance threshold. In that case, the decision rule says to fail to reject the null hypothesis. This is a limited conclusion about the evidence in the sample under the chosen test. It is not a declaration that the null hypothesis is true.

The distinction matters whenever a study compares a population value with a benchmark. A sample might be consistent with the benchmark, but also too imprecise to distinguish it from other plausible values. For example, a sample mean close to a target may reflect a population mean at the target, or a different population mean combined with sampling variation. A nonsignificant result alone does not settle which explanation is correct.

A p-value is calculated under the assumption that the null model is correct. It measures how often that model would produce a test statistic at least as extreme as the observed one, using the test’s definition of extremeness. Thus, a p-value above \(\alpha\) says the observation was not sufficiently extreme by that rule. It is not the probability that the null hypothesis is true, nor is it a measure of the probability that the observed difference arose “by chance.”

A Test Model for the Examples

The examples below use a two-sided test for a population mean when the population standard deviation \(\sigma\) is known and the population is normally distributed. Assume the observations are a random, independent sample. Under the null hypothesis \(H_0:\mu=\mu_0\), the sample mean has standard error \(\sigma/\sqrt{n}\), and the test statistic follows the standard normal distribution:

The alternative hypothesis is \(H_a:\mu\ne\mu_0\). For this two-sided test, results far enough in either direction from zero count as extreme. The p-value is \(P(|Z|\ge |z_{\text{obs}}|)\), calculated using the standard normal distribution under \(H_0\). At \(\alpha=0.05\), the rejection region is approximately \(Z<-1.96\) or \(Z>1.96\); equivalently, the two-sided p-value must be at most 0.05 for rejection.

This model is appropriate for the hypothetical examples because each specifies a random, independent sample from a normally distributed population, with its population standard deviation known. Those conditions justify the stated normal test calculation. They are assumptions of the examples, not facts inferred from a nonsignificant result.

Worked Examples: Failing to Reject a Benchmark

Why Non-Rejection Cannot Prove Equality

A test is designed to reject \(H_0\) when the data would be unusually extreme if the null model were true. But samples vary. Even if the population value differs from the benchmark, a particular sample can land near the null value and produce a nonsignificant result. This possibility is not a contradiction: the test’s rule is about how extreme the observed data are, not whether the population value has been proven equal or unequal.

The first example gives a direct illustration. Its null model has \(\mu_0=500\) grams, \(SE=4\) grams, and a two-sided 5% rejection region of approximately \(|Z|>1.96\). Now suppose, for illustration, that the true population mean were 504 grams instead. Under the same normal sampling model, the statistic \(Z=(\bar{x}-500)/4\) would have mean \((504-500)/4=1\) and standard deviation 1.

A failure to reject would occur when \(-1.96\le Z\le1.96\). Under the illustrative alternative mean of 504 grams, this probability is

Here \(\Phi\) is the cumulative distribution function of the standard normal distribution. Under those assumptions, a sample would fail to reject the 500-gram null in about 83% of repeated samples even though the hypothetical true mean is 504 grams. This calculation shows why non-rejection is not equivalent to establishing the null: a population value different from the benchmark can still often produce data inside the non-rejection region.

The calculation does not say that the mean in the packaging example is 504 grams. It describes what could happen under a particular alternative value and sampling model. Its purpose is to show that non-rejection can occur when the null is false. The observed p-value by itself cannot tell us the probability that this alternative—or any other particular value—is true.

Common Mistake: Treating “Not Significant” as “No Difference”

A common error is to reason: “The p-value is above 0.05, so the population value equals the benchmark.” The conclusion does not follow. The test has not rejected the equality stated in \(H_0\), but it has not shown that equality is correct. In particular, “no statistically significant evidence of a difference” is not interchangeable with “evidence that there is no difference.”

A better report states the decision and its limits: name the population quantity and benchmark, give the direction of the sample estimate if relevant, report the p-value or decision, and say that the sample did not provide sufficient evidence to reject the null at the chosen level. Do not turn a failure to reject into a claim of exact equality or certainty.

If the question is specifically whether a population value is close enough to a benchmark for a defined purpose, ordinary failure to reject a difference is not, by itself, an answer. Such a claim needs an explicit definition of what counts as close enough and an appropriate method for assessing that claim. A nonsignificant difference test does not supply that definition.