Why Failure to Reject Is Not Acceptance

What a Non-Rejection Does—and Does Not—Say

A hypothesis test gives a decision under a specified rule. When the test does not reject the null hypothesis, the conclusion is limited: the observed data did not fall in the rejection region at the chosen significance level. That result is not proof that the null hypothesis is true. A test may fail to reject because the null is a reasonable description of the parameter, or because the sample is too imprecise to distinguish the null from alternatives.

This distinction matters in actuarial work. Suppose a hypothetical review compares average processing time under a revised claims procedure with a target. A test that does not detect a difference does not, by itself, establish that processing times meet the target or are sufficiently close to it. The result depends on the data, the test’s sensitivity, and the claim being evaluated.

Definition: In a hypothesis test, rejecting $H_0$ means the data meet the test’s stated criterion for evidence against $H_0$. Failing to reject $H_0$ means they do not meet that criterion. Unless a procedure specifically defines a separate decision such as equivalence, “fail to reject” is not the same decision as “accept.”

For a test with significance level $\alpha$, the rejection region is chosen to control the probability of rejecting $H_0$ when $H_0$ is true. The p-value expresses how extreme the observed test statistic is, relative to the null model, in the direction specified by the test. If the p-value exceeds $\alpha$, the decision is to fail to reject $H_0$. Neither the p-value nor this decision is the probability that $H_0$ is true.

Why a Test Can Miss a Real Difference

Consider a test with a fixed rejection rule. Even when a particular alternative value is true, sampling variation can produce observations that do not enter the rejection region. The probability of that outcome at the alternative is the Type II error probability, often denoted by $\beta$. The power at that alternative is $1-\beta$, the probability of rejecting $H_0$ when that alternative is true.

\text{Power at }\theta_1=P_{\theta_1}(\text{reject }H_0), \qquad \beta(\theta_1)=P_{\theta_1}(\text{fail to reject }H_0)

These probabilities depend on the alternative value, the sampling distribution, the sample size, the variability, and the rejection rule. A test’s significance level alone does not say how likely it is to detect a particular departure from the null.

Justification: A test is a decision rule that partitions possible data outcomes into a rejection region and its complement. Under any specified alternative $\theta_1$, those regions still have probabilities that sum to one. Therefore, $$ P_{\theta_1}(\text{reject }H_0)+P_{\theta_1}(\text{fail to reject }H_0)=1. $$ The first probability is power and the second is $\beta(\theta_1)$. If the power is below one, there is a positive chance of failing to reject even when that alternative is true. Thus non-rejection cannot establish that the null is true.

Worked Example: Non-Rejection Despite a Meaningful Alternative

A hypothetical process has independent observations from a normal population with known standard deviation $\sigma=30$. A sample of $n=100$ observations is used to test $H_0:\mu=100$ against $H_1:\mu>100$ at $\alpha=0.05$. Suppose the observed sample mean is $\bar{x}=104$. Determine the decision and then calculate the test’s power when the actual mean is $\mu=106$.

Method and assumptions. Use a one-sided z-test for a normal population with known $\sigma$, independent observations, and the stated sample size. The standard error is $$ SE(\bar{X})=\frac{\sigma}{\sqrt{n}}=\frac{30}{\sqrt{100}}=\frac{30}{10}=3. $$

Observed result. Under $H_0$, the standardized statistic is $$ z=\frac{\bar{x}-100}{3}=\frac{104-100}{3}=\frac{4}{3}=1.3333. $$ The upper-tail p-value is $P(Z\ge 1.3333)\approx 0.09121$. Since $0.09121>0.05$, fail to reject $H_0$. This says that the result does not meet the chosen threshold for rejecting the null; it does not establish that $\mu=100$.

Power at $\mu=106$. The upper-tail critical value is $z_{0.95}=1.64485$, so the rejection rule is $$ \bar{X}>100+1.64485(3)=104.93455. $$ When $\mu=106$, $\bar{X}$ has mean $106$ and standard error $3$. Therefore, $$ \begin{aligned} \text{Power} &=P_{\mu=106}(\bar{X}>104.93455)\\ &=P\left(Z>\frac{104.93455-106}{3}\right)\\ &=P(Z>-0.35515)=\Phi(0.35515)\approx 0.6387. \end{aligned} $$ The probability of failing to reject at this alternative is $1-0.6387=0.3613$. Thus even if the actual mean is $106$, this test has about a 36.13% chance of not rejecting.

Check and common mistake. The rejection cutoff is above the null mean, as required for an upper-tail test. The power is the probability of exceeding that cutoff under $\mu=106$, not under the null. Reversing the tail or using the null mean to calculate power would answer a different question.

A Large P-Value Is Not a Probability That the Null Is True

A p-value is calculated assuming the null model. It measures the probability, under that model, of obtaining a test statistic at least as extreme as the observed statistic in the test’s specified direction. It is not $P(H_0\mid\text{data})$. A p-value above $\alpha$ also does not show that the data are especially likely under $H_0$; it only indicates that the chosen rejection criterion was not met.

The distinction is visible in a confidence interval. A wide interval may include the null value while also including parameter values that would matter in practice. The interval then shows uncertainty across a broad range, rather than affirmatively supporting the null value. The earlier material on two-sided tests and confidence interval duality gives the formal relationship between a test and its corresponding interval.

Worked Example: A Non-Significant Result With a Wide Interval

In a hypothetical audit, independent measurements are modeled as normal with known standard deviation $\sigma=15$. A random sample of $n=25$ has mean $\bar{x}=52$. Test $H_0:\mu=50$ against $H_1:\mu\ne 50$ at the 5% level, and calculate the corresponding 95% confidence interval.

Method and assumptions. Use the two-sided z-test and the known-standard-deviation normal interval. The sample standard error is $$ SE(\bar{X})=\frac{15}{\sqrt{25}}=\frac{15}{5}=3. $$

The observed statistic is $$ z=\frac{52-50}{3}=\frac{2}{3}=0.6667. $$ The two-sided p-value is $2P(Z\ge 0.6667)\approx 0.5050$. Since $0.5050>0.05$, fail to reject $H_0$.

For a 95% interval, use $z_{0.975}=1.96$: $$ \begin{aligned} \bar{x}\pm 1.96SE &=52\pm 1.96(3)\\ &=52\pm 5.88\\ &=[46.12,\;57.88]. \end{aligned} $$ The interval includes $50$, consistent with the non-rejection from the two-sided test. It also includes values as low as $46.12$ and as high as $57.88$, so the data have not pinned the mean close to $50$.

Check and common mistake. The interval is wide because the standard error is 3. Saying “the mean is 50 because the test was not significant” ignores the other values still compatible with the interval. The result supports uncertainty about $\mu$, not a conclusion that it equals the null value.

When Can Data Support a Claim of “Close Enough”?

Some questions concern not exact equality but practical similarity. For example, a hypothetical administrator might regard a change in average processing time of less than 5 minutes in either direction as operationally negligible. The relevant claim is then that the parameter difference lies within a prespecified tolerance band, not that it equals exactly zero.

A confidence interval can provide affirmative evidence for such a bounded claim if it is sufficiently precise and lies entirely inside the acceptable range. This is different from merely observing that a test of an exact null did not reject. The margin must be chosen for substantive reasons before examining the result; selecting it afterward to make the interval fit would not provide a trustworthy assessment.

Worked Example: Evidence for Practical Equivalence

A hypothetical comparison estimates the change $\mu$ in mean service time under a revised procedure, with positive values indicating longer times. Assume independent normal observations and a known standard error of $1$ minute. The estimated change is $\bar{x}=0.6$ minutes. Before collecting data, the organization defined changes between $-5$ and $5$ minutes as practically negligible. Assess whether the data support that bounded claim using a 90% confidence interval.

Method and assumptions. With a normal sampling distribution and known standard error, the 90% interval uses $z_{0.95}=1.64485$. Its endpoints are $$ \begin{aligned} 0.6\pm 1.64485(1) &=0.6\pm 1.64485\\ &=[-1.04485,\;2.24485]\\ &\approx[-1.045,\;2.245]\text{ minutes}. \end{aligned} $$ The interval is entirely inside the prespecified range: $-1.045>-5$ and $2.245<5$. At the corresponding 5% one-sided level for each bound, the interval supports the claim that the change is within the practical-equivalence margin, under the stated model.

This conclusion is not a proof that the true change is exactly zero. It is evidence for the more useful, explicitly bounded statement that plausible values are within the chosen tolerance. The distinction is that the interval is narrow enough to exclude changes considered operationally important.

Check and common mistake. The interval must lie wholly inside both bounds. If it crossed either $-5$ or $5$, the data would not establish equivalence by this criterion, even if a test of $\mu=0$ failed to reject. A non-significant test of exact equality and evidence of practical equivalence answer different questions.

Common Interpretive Traps

“We accept the null.” Unless the testing framework defines an acceptance decision for a separate claim, report “fail to reject $H_0$.” A non-rejection is not proof of the null.
“The p-value is the chance the null is true.” The p-value is calculated under the null model; it is not a posterior probability for the null.
“No significant difference means no meaningful difference.” A test may have low power for the effects that matter. Examine the standard error, sample size, and uncertainty in the estimate.
“A confidence interval containing the null proves equality.” It only indicates that the null value was not excluded by that interval. Other practically important values may also remain plausible.
“A non-significant result proves equivalence.” Equivalence requires a prespecified margin and sufficiently precise evidence that the parameter lies within it.

Key takeaway: A failure to reject is a statement about a test decision, not a declaration that the null is true. To assess whether an important effect remains plausible, examine power and uncertainty. To support a claim of practical similarity, define an acceptable range in advance and show that the evidence is precise enough to fit within it.

Check Your Understanding

For each problem, distinguish the test decision from what the result can establish. Assume the stated models and sampling conditions hold.

A normal population has known $\sigma=16$. For $n=64$, test $H_0:\mu=50$ against $H_1:\mu>50$ at $\alpha=0.05$ when $\bar{x}=53.2$. Find the standard error and p-value, state the decision, and explain whether the result establishes $\mu=50$. (Answer: $SE=2$, $z=1.6000$, upper-tail $p\approx0.0548$; fail to reject, but this does not establish $\mu=50$.)
A normal population has known $\sigma=2.5$. With $n=25$, test $H_0:\mu=12$ against $H_1:\mu\ne12$ when $\bar{x}=12.4$. Give the p-value and 95% interval, then interpret the non-rejection. (Answer: $SE=0.5$, $z=0.8000$, two-sided $p\approx0.4237$; 95% interval $[11.42,13.38]$, which includes 12 but also leaves substantial uncertainty.)
An upper-tail test has $H_0:\mu=20$, known standard error $2$, and $\alpha=0.05$. Its rejection cutoff is $20+1.64485(2)=23.2897$. Find power at $\mu=24$ and the probability of failing to reject there. (Answer: power $=P(Z>-0.35515)\approx0.6387$; probability of failing to reject $\approx0.3613$.)
A hypothetical procedure comparison has estimated mean change $1.0$ minute and known standard error $1.2$ minutes. The acceptable range was set in advance as $(-4,4)$ minutes. Does the 90% interval support practical equivalence? (Answer: $1.0\pm1.64485(1.2)=[-0.97382,2.97382]$ minutes, entirely inside $(-4,4)$; it supports the bounded claim under the stated model, not exact equality to zero.)
A report says a two-sided test produced $p=0.27$ at $\alpha=0.05$, and concludes “there is a 27% chance that the null is true.” Identify the error and state the valid conclusion. (Answer: $0.27$ is not the probability that the null is true; it is a null-model tail probability for a result at least as extreme as observed. The decision is to fail to reject at 0.05, which does not establish the null.)

What You'll Learn