Why Equivalence Margins Must Be Set Independently of Results

Why the Margin Is Part of the Claim

An equivalence analysis does not ask merely whether an estimated difference is small, or whether a conventional test fails to detect a difference. It asks whether the true effect is sufficiently close to zero according to a defined tolerance. The equivalence margins specify that tolerance. For an effect measured as a difference, write the prespecified limits as \(-\Delta\) and \(+\Delta\), where \(\Delta>0\). The choice of \(\Delta\) is therefore part of the scientific question, not a setting to adjust until the data produce a desired result.

In bioavailability and bioequivalence, the effect may be analyzed on a difference scale or on a transformed scale associated with a ratio. The margin must be justified on the scale used for the analysis and interpreted in the relevant clinical and regulatory context. This tutorial uses differences to make the boundary logic transparent; the same principle applies to other scales.

Definition: For a true effect \(\theta\) and a fixed equivalence margin \(\Delta>0\), the equivalence claim is that \(-\Delta<\theta<\Delta\). TOST evaluates this claim by testing both \(H_{0L}:\theta\leq-\Delta\) against \(H_{aL}:\theta>-\Delta\), and \(H_{0U}:\theta\geq\Delta\) against \(H_{aU}:\theta<\Delta\). Equivalence is concluded only when both null hypotheses are rejected at the specified significance level \(\alpha\).

The margins define which true effects count as unacceptable under the analysis. If the margin is changed after seeing the estimate or confidence interval, the analysis is no longer answering the same prespecified question. In particular, choosing a wider margin because the observed interval narrowly misses a narrower one can turn a result that did not establish equivalence into one that does, without collecting any new information.

Why a Fixed Margin Gives the TOST Interpretation

Suppose an estimate \(\hat{\theta}\) has standard error \(SE\), and its sampling distribution is assessed using a \(t\) reference distribution with \(df\) degrees of freedom. For a fixed margin, the two TOST statistics are

t_L=\frac{\hat{\theta}+\Delta}{SE}, \qquad t_U=\frac{\hat{\theta}-\Delta}{SE}.

The lower-bound test rejects when \(t_L>t_{1-\alpha,df}\); the upper-bound test rejects when \(t_U<-t_{1-\alpha,df}\). Here \(t_{1-\alpha,df}\) is the \(1-\alpha\) quantile of the \(t\) distribution with \(df\) degrees of freedom. Both conditions must hold. Equivalently, the \(100(1-2\alpha)\%\) two-sided confidence interval must lie strictly inside the equivalence limits, using the same standard error and reference distribution.

The equivalence of these criteria follows directly from the two inequalities. The lower-bound test requires

\frac{\hat{\theta}+\Delta}{SE}>t_{1-\alpha,df} \quad\Longleftrightarrow\quad \hat{\theta}-t_{1-\alpha,df}SE>-\Delta.

The upper-bound test requires

\frac{\hat{\theta}-\Delta}{SE}<-t_{1-\alpha,df} \quad\Longleftrightarrow\quad \hat{\theta}+t_{1-\alpha,df}SE<\Delta.

Together, these statements say that both endpoints of the corresponding confidence interval fall inside the fixed margins. This interpretation and its nominal error control concern the test as specified: the margin, significance level, and analysis procedure are fixed independently of the results being tested. If a margin is selected because it makes the observed interval fit, the two inequalities still describe a calculation, but the usual prespecified-test interpretation no longer follows.

A Boundary-Focused Example

Worked Example: A Narrowly Missed Limit Becomes a Pass

Suppose a hypothetical study estimates a mean treatment difference of \(8.0\) mg, with \(SE=1.5\) mg and \(df=24\). The study plan specified equivalence margins of \(-10\) mg and \(+10\) mg, based on a prior clinical justification. Use \(\alpha=0.05\). For \(df=24\), \(t_{0.95,24}\approx1.711\).

Calculate the interval. The corresponding 90% confidence interval is

8.0\ \text{mg}\ \pm 1.711(1.5\ \text{mg}) =8.0\ \text{mg}\ \pm 2.567\ \text{mg} =[5.433,\ 10.567]\ \text{mg}.

The lower endpoint is above \(-10\) mg, but the upper endpoint, \(10.567\) mg, exceeds \(+10\) mg. The interval is not wholly inside the prespecified limits, so equivalence is not established at the 5% level.

Check the upper-bound test directly. Its statistic is

t_U=\frac{8.0-10}{1.5}=-1.333.

For this upper-bound test, rejection requires a statistic below \(-1.711\). Since \(-1.333\) is not below that critical value, the upper-bound null is not rejected; its one-sided \(p\)-value is approximately \(0.097\). The lower-bound statistic is \((8.0+10)/1.5=12.00\), which provides ample evidence against the lower-bound null. Because both tests must pass, the TOST conclusion is not equivalence.

Now imagine that, after looking at the interval, someone changes the margin from \(10\) mg to \(10.6\) mg. The same interval, \([5.433,10.567]\) mg, now fits inside \((-10.6,10.6)\) mg. For the upper-bound test with this revised margin,

t_U=\frac{8.0-10.6}{1.5}=-1.733.

This is below \(-1.711\); its one-sided \(p\)-value is approximately \(0.048\). The lower-bound test also rejects, so the revised calculation would declare equivalence. But the apparent change in conclusion came only from widening the limit until it cleared the observed endpoint. It does not establish that a difference as large as \(10.6\) mg is clinically acceptable. Unless the revised margin has a valid justification independent of these results, this is not a defensible way to make the equivalence claim.

Choosing the Margin Before the Result

A margin should follow from the meaning of an acceptable difference, not from the width or location of the interval produced by the study. Depending on the application, its rationale may involve clinical consequences, established regulatory criteria, prior evidence about treatment effects, or other scientifically justified considerations. Those considerations must be translated to the analysis scale. The justification should make clear why effects beyond either limit would be unacceptable and why effects within the limits are considered close enough for the intended decision.

The margin is usually documented in the protocol or analysis plan before the relevant results are available to the people making the equivalence decision. This protects the distinction between two questions: “Are the observed data precise enough to establish equivalence within a scientifically acceptable range?” and “Can a range be found that includes the observed interval?” Only the first is the planned TOST question.

Worked Example: A Prespecified Clinically Justified Limit

Consider a separate hypothetical study of a biomarker difference measured in mg/L. Before examining the confirmatory study results, the investigators justify an equivalence margin of \(\pm4.0\) mg/L on clinical grounds and specify \(\alpha=0.05\). The estimate is \(-1.1\) mg/L, \(SE=1.2\) mg/L, and \(df=29\). Use \(t_{0.95,29}\approx1.699\).

The 90% confidence interval is

-1.1\ \text{mg/L}\ \pm 1.699(1.2\ \text{mg/L}) =-1.1\ \text{mg/L}\ \pm 2.039\ \text{mg/L} =[-3.139,\ 0.939]\ \text{mg/L}.

Both endpoints lie strictly between \(-4.0\) and \(+4.0\) mg/L. Equivalently, \(t_L=(-1.1+4.0)/1.2=2.417\), which exceeds \(1.699\), and \(t_U=(-1.1-4.0)/1.2=-4.250\), which is below \(-1.699\). Both one-sided tests reject their null hypotheses. The data establish equivalence at the specified level for the justified margin. The conclusion is meaningful because the range was chosen for scientific reasons before these results were used to assess equivalence.

The estimate in this example is not what made \(\pm4.0\) mg/L acceptable. A different estimate or a wider interval could have produced a different TOST decision, while the clinically justified margin remained the same. This separation is essential: the data determine whether the evidence is sufficiently precise for the claim, while the margin defines the claim.

What Goes Wrong When the Margin Is Data-Dependent

For a fixed margin and a valid analysis, TOST’s error properties are evaluated under the stated hypotheses and significance level. If the analyst examines the estimate or interval, tries several candidate margins, and reports the one that yields equivalence, the reported decision is selected using the same random data being tested. The probability of obtaining a favorable result by this selection process is not generally controlled by the nominal \(\alpha\) of a single TOST with a prespecified margin. The exact effect depends on how the margins were selected; it is not correct to assume that every post-result change produces the same amount of error inflation. The key point is that the usual single-test interpretation no longer applies without a method that accounts for the selection.

A margin may be informed by evidence available before the confirmatory results, including external evidence or a separate pilot, if the rationale is scientifically sound and the decision process is specified appropriately. “Independent of the results” does not mean that a margin must be chosen without any evidence. It means that the margin must not be tuned to the observed confirmatory estimate, interval, or test outcome to obtain a desired decision.

A common mistake is to reason, “The 90% interval missed the limit by only a small amount, so increasing the margin by that amount is harmless.” The correction is that a small numerical change can alter the null hypotheses being tested and the scientific definition of acceptable equivalence. A boundary is not a rounding allowance. If a clinically justified margin was fixed in advance and the interval crosses it, report that equivalence was not established under that criterion. Do not relabel the result by silently replacing the criterion.

Diagnostics, Alternatives, and Limitations

Several practical checks can reveal that a margin may have been chosen or altered in response to results:

The protocol or analysis plan does not specify the margin, or the stated rationale appears only after the treatment effect is known.
The proposed margin closely matches an observed confidence-interval endpoint without a separate clinical or regulatory explanation.
Several margins were evaluated, but only the one producing equivalence is reported.
The margin is justified by saying that it permits equivalence, rather than by describing which true differences would be acceptable.
The analysis uses a margin on a different scale from the one justified, without a clear and correct scale conversion.

These are warning signs, not automatic proof of misconduct or invalidity. A transparent report should state the limits, their rationale, when they were specified, the estimate and interval with units, and the TOST decision. If a post-result analysis is presented, it should be identified as exploratory rather than represented as the original confirmatory test.

The main alternative is not to “choose a better margin” after the fact. If a result fails to establish equivalence under the planned limits, report that result and its uncertainty. A conventional test of a zero difference answers a different question: failing to reject zero difference does not demonstrate that the effect lies within acceptable limits. Likewise, observing that the point estimate is inside the margins is insufficient when its interval extends outside them. Neither alternative repairs a margin selected to fit the result.

This principle does not by itself establish that a chosen margin is clinically appropriate, nor does prespecification guarantee a sound study. The margin still needs a defensible scientific basis, and the study must use an appropriate estimand, analysis, and uncertainty calculation. Prespecification protects the interpretation of the decision rule; it cannot substitute for sound judgment about what counts as acceptable equivalence.

Key takeaway: TOST evaluates equivalence relative to a defined tolerance. Justify and fix that tolerance independently of the confirmatory results; otherwise, a nominally favorable decision may reflect the margin-selection process rather than evidence for the originally intended equivalence claim.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A study has an estimate of \(3.0\) mg and a 90% confidence interval of \([0.5,5.5]\) mg. Its prespecified margins are \(\pm5\) mg. Is equivalence established? Explain.
Why can choosing a wider margin after seeing the interval undermine the nominal interpretation of TOST?
A protocol specifies \(\pm2.0\) units, but the interval misses the upper limit by \(0.1\) unit. Is increasing the margin to \(\pm2.1\) justified solely by that fact?
A margin is justified using external evidence before a confirmatory study begins. The study’s interval falls outside that margin. Should the margin be changed to obtain a positive conclusion?

Solutions. (1) No. The upper endpoint, \(5.5\) mg, exceeds the prespecified upper limit of \(5\) mg, so the interval is not wholly inside the equivalence region. (2) The margin defines the null and alternative hypotheses. Selecting it using the same results can make the decision favorable by construction, so the nominal error control for a single fixed-margin test does not generally apply to that selection process. (3) No. The numerical closeness of the endpoint does not show that \(2.1\) units is a clinically acceptable limit. A change needs an independent scientific justification, not merely a desire to include the interval. (4) No. The result should be reported relative to the justified margin. A different analysis may be explored transparently, but changing the confirmatory criterion to obtain a positive conclusion changes the question and does not rescue the original claim.