Making the TOST Decision From Both Tests

Why the Decision Requires Both Tests

An equivalence conclusion is a joint claim: the treatment effect is greater than a lower margin and less than an upper margin. Evidence against only one boundary does not establish the other part of that claim. For example, an estimate can be convincingly above the lower margin while remaining too uncertain to rule out effects at or beyond the upper margin.

Let \(\theta\) denote the population treatment effect on the analysis scale, and let \(L\) and \(U\) be the prespecified lower and upper equivalence margins, with \(L<U\). The estimate is \(\hat{\theta}\), its standard error is \(SE\), and the reference distribution has \(df\) degrees of freedom. The two one-sided tests (TOST) assess the lower and upper boundaries separately at significance level \(\alpha\).

The overall null is the union of the component nulls: at least one of \(\theta\le L\) or \(\theta\ge U\) is true. To conclude equivalence, the data must provide evidence against both possibilities. Rejecting just one component leaves the other part of the overall null unrefuted. This is why the decision is not based on an average of the two p-values, nor on whether one test is especially convincing.

Apply Each Rejection Criterion

For the standard one-sided \(t\) procedure, the statistics and directional p-values are

Here \(T_{df}\) has a Student \(t\) distribution with \(df\) degrees of freedom. Large positive values of \(t_L\) support \(\theta>L\), while large negative values of \(t_U\) support \(\theta<U\). The p-values therefore use opposite tails. At level \(\alpha\), reject the lower component null when \(p_L\le\alpha\), and reject the upper component null when \(p_U\le\alpha\). Equivalently, if \(t_{1-\alpha,df}\) is the \(1-\alpha\) quantile of the \(t\) distribution, the criteria are \(t_L\ge t_{1-\alpha,df}\) and \(t_U\le -t_{1-\alpha,df}\).

These criteria presume that the treatment-effect estimate and standard error come from an appropriate model, the stated degrees of freedom are valid for the chosen procedure, and the test-statistic reference distribution is justified under each boundary null. In a crossover analysis, for instance, the model and its standard error must reflect the design and the analysis of the treatment effect. The margins and \(\alpha\) must also be specified before interpreting the two results; selecting them after seeing the estimates would change the meaning of the tests.

When Exactly One Test Rejects

The mixed outcomes are important because they are not equivalence conclusions. One test may reject clearly while the other does not. In that situation, the appropriate report identifies which boundary was ruled out and which was not; it does not turn the one rejection into an overall success.

Decision Logic and a Short Justification

The two-test decision can be justified directly from the hypotheses. Equivalence requires \(\theta>L\) and \(\theta<U\) simultaneously. The null says at least one of those inequalities fails. Thus, evidence against the overall null requires evidence against each of its two component possibilities.

If either component test fails to reject, its corresponding boundary remains compatible with the test outcome at level \(\alpha\). It follows that the conjunction of the two rejection criteria is required. If both component tests are conducted at level \(\alpha\), the probability of falsely declaring equivalence when the overall null is true is controlled at no more than \(\alpha\): whenever the overall null is true, at least one component null is true, and a false equivalence declaration requires falsely rejecting that true component null. No adjustment that treats the two tests as independent is needed for this intersection-union decision.

Diagnostics, Alternatives, and Common Pitfalls

Before finalizing a decision, verify that each p-value uses the correct tail. The lower test is a right-tail test; the upper test is a left-tail test. A sign error or using the same tail for both tests can reverse a decision. Check the calculations against the critical-value form as well: the lower statistic must be sufficiently positive, and the upper statistic sufficiently negative. If the p-value and critical-value decisions disagree, inspect the signs, degrees of freedom, standard error, and rounding.

Results near \(\alpha\) deserve particular care. A p-value close to the threshold can change its displayed rounded value, so retain adequate precision for the decision and report enough digits to make the comparison transparent. A test statistic near its critical value is also a cue to verify that the standard error and model specification are appropriate. Do not change the margins, significance level, or analysis model merely because one result is inconvenient; any justified model change needs to be evaluated and reported on its own terms.

The principal alternative summary is a confidence interval constructed at the level corresponding to TOST. The earlier material on the confidence-interval rule establishes the equivalent interval criterion. Here the practical distinction is one of presentation: the two-test approach displays each boundary-specific result, while the interval approach displays the range of effects compatible with the data. Neither presentation permits a conclusion when one boundary criterion fails. A conventional test of no difference is not a substitute for TOST: failing to detect a difference does not, by itself, establish that effects outside the equivalence margins have been ruled out.

A common mistake is to reason, “The estimated effect lies between the margins, and one test rejects, so the treatments are equivalent.” The estimate’s location alone does not account for uncertainty, and one rejection addresses only one boundary. The correction is to evaluate both directional tests and require both p-values to be at most \(\alpha\). If exactly one rejects, report that mixed result and conclude that equivalence was not established.