Why a Nonsignificant Difference Does Not Establish Equivalence

Two Questions That Require Different Tests

In a bioavailability comparison, a conventional test of no difference asks whether the data provide evidence that a treatment contrast differs from zero. Equivalence asks a different question: do the data rule out differences large enough to fall outside prespecified equivalence limits? A result can fail to show a difference without ruling out differences that matter for equivalence.

The distinction is not merely a choice of wording. The tests have different null hypotheses, and they use the same estimate and standard error to answer different questions. A large p-value from the conventional test is therefore not, by itself, evidence that the treatments are equivalent.

For a bioavailability endpoint such as AUC, \(\delta\) might be the population mean difference between test and reference treatments, measured in the endpoint’s units. The limits must be scientifically justified and fixed independently of the observed results. The examples below use hypothetical data and symmetric limits for clarity; the logic also applies with asymmetric limits.

What Each Decision Means

In a conventional test at significance level \(\alpha=0.05\), a p-value above 0.05 means that the data do not provide sufficient evidence to reject \(H_0:\delta=0\). It does not establish that \(\delta=0\), nor does it show that \(\delta\) is close enough to zero for equivalence. The estimate may be imprecise, leaving a wide range of plausible values.

TOST, the two one-sided tests procedure, declares equivalence only when both one-sided null hypotheses are rejected at level \(\alpha\). One test rules out values at or below the lower limit; the other rules out values at or above the upper limit. Because both conditions must hold, equivalence requires evidence that the contrast is inside both limits.

For a model-based estimate \(\hat{\delta}\) with standard error \(SE\) and \(df\) degrees of freedom, the TOST statistics are

The lower-limit test uses an upper-tail p-value, and the upper-limit test uses a lower-tail p-value. TOST concludes equivalence only if each p-value is below \(\alpha\). This is an intersection-union decision: both component tests must pass, so one successful test cannot compensate for the other failing.

Why the TOST Interval Is Not the Conventional Interval

Assume the estimate and standard error support the usual \(t\)-based inference: under the specified model, the standardized estimation error follows a \(t\) distribution with the stated degrees of freedom. The standard error and degrees of freedom must be appropriate for the estimand and study design. At level \(\alpha\), each one-sided TOST uses the critical value \(t_{1-\alpha,df}\). The associated two-sided interval is a \(100(1-2\alpha)\%\) confidence interval.

To see the connection, the lower-limit test rejects when \(t_L>t_{1-\alpha,df}\). Rearranging this inequality gives \(\hat{\delta}-t_{1-\alpha,df}SE>L\). The upper-limit test rejects when \(t_U<-t_{1-\alpha,df}\), which rearranges to \(\hat{\delta}+t_{1-\alpha,df}SE<U\). Taken together, these conditions say the interval lies inside both limits.

At \(\alpha=0.05\), this is the 90% confidence interval, not the 95% interval used for a conventional two-sided test at the same significance level. A conventional test’s failure to reject is associated with a 95% interval that includes zero. That condition places no requirement on whether the interval fits inside the equivalence limits.

Precision, Not Just the Point Estimate

The following examples show why neither conventional significance nor nonsignificance determines the equivalence decision on its own. In each case, the limits and the precision of the estimate matter. The reported values are hypothetical.

Assumptions, Checks, and Limits of Interpretation

The calculations depend on a valid estimate of the intended treatment contrast and a valid standard error. The analysis model must reflect the design, including any pairing, repeated measurements, or other structure relevant to the data. The assumed sampling distribution must be reasonable for the inferential procedure used, and the degrees of freedom or large-sample approximation must be appropriate. Equivalence limits must be justified for the endpoint and chosen before examining results; changing them after seeing the interval changes the question being tested.

A useful review is to compare the estimate, both limits, the standard error, and the relevant interval on the same scale and in the same units. Then check that the conventional test uses the zero-difference null and that TOST uses the two margin-based nulls. If an interval-based decision and a p-value-based decision appear inconsistent, first verify that the interval confidence level matches the one-sided test level and that the same estimate, standard error, degrees of freedom, and limits were used.

These procedures also have limits. A TOST conclusion concerns the specific estimand, endpoint, population, and limits analyzed; it does not establish equality or equivalence for every aspect of treatment. A failure to conclude equivalence is not proof that the treatments are nonequivalent: the evidence may simply be too imprecise to rule out values beyond a limit. Conversely, equivalence is meaningful only if the limits are scientifically justified. Statistical calculations cannot supply that justification.

Common Mistake

The practical value of keeping these conclusions separate is that it prevents “no evidence of a difference” from being reported as “evidence of equivalence.” Report the estimated contrast and its uncertainty alongside the relevant decision, with units and the equivalence limits. A p-value from a conventional test alone cannot provide that account.