Diagnosing an Incorrect Confidence Level in TOST

Why the Confidence Level Matters in TOST

In a two one-sided tests procedure (TOST), equivalence is established only when evidence is sufficient against both claims that the parameter lies outside the equivalence region. The interval used to summarize that decision must match the significance level of those two tests. A confidence interval with a familiar but mismatched level can therefore report a different conclusion from the stated TOST.

The most common case is a TOST with each one-sided test conducted at \(\alpha=0.05\). Its matching two-sided confidence interval is a 90% interval, not a 95% interval. A 95% interval is wider and can fail to fit within the equivalence limits even when both 5% one-sided tests reject. Conversely, an 80% interval is narrower and may fit within the limits when a 5% TOST does not.

Definition: Let \(\theta\) be the parameter being assessed for equivalence, with prespecified lower and upper equivalence limits \(L\) and \(U\), where \(L<U\). A TOST at one-sided significance level \(\alpha\) tests \(H_{01}:\theta\leq L\) against \(H_{a1}:\theta>L\), and \(H_{02}:\theta\geq U\) against \(H_{a2}:\theta<U\). Equivalence is concluded only if both null hypotheses are rejected.

The Confidence Level That Matches the Tests

Suppose an estimate \(\hat{\theta}\) has standard error \(SE\), and the analysis uses a normal reference distribution or a t reference distribution with \(df\) degrees of freedom. Let \(c\) be the critical value for a two-sided confidence interval. The interval is \(\hat{\theta}\pm cSE\). For a TOST at level \(\alpha\), each one-sided test uses the upper-tail critical value at probability \(1-\alpha\): \(z_{1-\alpha}\) for a normal reference distribution, or \(t_{1-\alpha,df}\) for a t reference distribution. The corresponding two-sided interval uses that same critical value and thus has confidence level \(1-2\alpha\).

\text{Matching two-sided confidence level}=1-2\alpha

For example, when \(\alpha=0.05\), the critical value is the 95th percentile of the relevant reference distribution: \(z_{0.95}=1.645\) or \(t_{0.95,df}\). The interval is therefore a 90% two-sided interval. The familiar 95% two-sided interval instead uses the 97.5th percentile, which does not match two one-sided tests at 0.05.

Key result: Under the same model, standard error, and reference distribution, a TOST at one-sided level \(\alpha\) concludes equivalence exactly when the corresponding \(100(1-2\alpha)\%\) two-sided confidence interval lies strictly within \((L,U)\). The comparison is strict because the component alternatives require \(\theta>L\) and \(\theta<U\).

This relationship assumes that the estimate and its uncertainty are calculated using a valid model, and that the confidence interval and both tests use the same estimate, standard error, reference distribution, and degrees of freedom. The limits \(L\) and \(U\) must be specified independently of the observed estimate. If the estimate is on a log scale, the comparison is made to limits on that same scale; any later presentation on a ratio scale must represent the same interval after back-transformation.

Why the Interval and TOST Decisions Agree

The equivalence criterion can be derived directly from the two tests. For a normal-reference analysis, the lower-limit test statistic is \((\hat{\theta}-L)/SE\), and the upper-limit test statistic is \((\hat{\theta}-U)/SE\). Both tests reject at level \(\alpha\) if the lower statistic exceeds \(z_{1-\alpha}\) and the upper statistic is less than \(-z_{1-\alpha}\). The same argument applies with the appropriate t critical value when a t reference distribution is used.

\frac{\hat{\theta}-L}{SE}>c \quad\text{and}\quad \frac{\hat{\theta}-U}{SE}<-c

Because \(SE>0\), rearranging the first inequality gives \(\hat{\theta}-cSE>L\). Rearranging the second gives \(\hat{\theta}+cSE<U\). Taken together, these inequalities say that both endpoints of the interval \([\hat{\theta}-cSE,\hat{\theta}+cSE]\) are inside the equivalence limits. Since \(c=z_{1-\alpha}\) or \(t_{1-\alpha,df}\), this is precisely the two-sided interval with confidence level \(1-2\alpha\). Using a different critical value breaks this correspondence.

Worked Calculations

Worked Example: A 95% Interval Hides a Successful 5% TOST

In a hypothetical comparison, the estimated difference between two treatments is \(\hat{\theta}=0.20\) exposure units, with \(SE=0.12\). The prespecified equivalence limits are \(L=-0.10\) and \(U=0.40\). Assume a valid normal reference distribution and conduct each one-sided test at \(\alpha=0.05\).

The matching critical value is \(z_{0.95}=1.645\), so the 90% interval is:

0.20\pm 1.645(0.12) =0.20\pm 0.1974 =[0.0026,\ 0.3974]\ \text{exposure units}

Both endpoints are strictly between \(-0.10\) and \(0.40\), so the interval criterion supports equivalence. Check the component tests: the lower statistic is \((0.20-(-0.10))/0.12=2.50\), which exceeds 1.645. The upper statistic is \((0.20-0.40)/0.12=-1.667\), which is less than \(-1.645\). Both tests reject at 0.05, consistent with the interval decision.

If a 95% interval is mistakenly used, its critical value is \(z_{0.975}=1.960\). The result is \(0.20\pm1.960(0.12)=0.20\pm0.2352=[-0.0352,\ 0.4352]\) exposure units. This wider interval crosses the upper limit. It would incorrectly suggest that the confidence-interval criterion for the 5% TOST is not met. The correction is to use the 90% interval that matches the stated one-sided level.

Worked Example: An 80% Interval Creates False Reassurance

Suppose a hypothetical estimate is \(\hat{\theta}=0.44\) exposure units with \(SE=0.10\), and the limits are \(L=-0.10\) and \(U=0.60\). Assume the normal approximation is justified and the intended TOST uses \(\alpha=0.05\).

The matching 90% interval is:

0.44\pm1.645(0.10) =0.44\pm0.1645 =[0.2755,\ 0.6045]\ \text{exposure units}

The upper endpoint exceeds \(U=0.60\), so equivalence is not established. In the upper-limit test, the statistic is \((0.44-0.60)/0.10=-1.60\), which is not less than \(-1.645\); its one-sided p-value is approximately 0.0548, above 0.05. Thus this test does not reject its null hypothesis.

An 80% interval uses \(z_{0.90}=1.282\), giving \(0.44\pm1.282(0.10)=[0.3118,\ 0.5682]\) exposure units. This narrower interval lies within the limits, but it corresponds to a TOST with \(\alpha=0.10\), not 0.05. It cannot support the intended 5% decision. The correction is to restore the 90% interval and report that equivalence has not been established at the specified level.

Worked Example: Using the t Critical Value at the Correct Level

Consider a hypothetical estimate \(\hat{\theta}=0.17\) log-ratio units, \(SE=0.04\), and \(df=9\), with equivalence limits \(L=-0.25\) and \(U=0.25\) log-ratio units. Assume the model supports the t reference distribution and that the standard error and degrees of freedom are valid. For a 5% TOST, \(t_{0.95,9}=1.833\).

The matching 90% interval is:

0.17\pm1.833(0.04) =0.17\pm0.07332 =[0.09668,\ 0.24332]\ \text{log-ratio units}

This interval is strictly within \((-0.25,0.25)\). The lower test statistic is \((0.17-(-0.25))/0.04=10.5\), exceeding 1.833. The upper statistic is \((0.17-0.25)/0.04=-2.00\), less than \(-1.833\); its one-sided t p-value with 9 degrees of freedom is approximately 0.0383. Both tests reject at 0.05, so equivalence is supported.

A mistaken 95% interval uses \(t_{0.975,9}=2.262\): \(0.17\pm2.262(0.04)=[0.07952,\ 0.26048]\) log-ratio units. It crosses the upper limit. It is an interval at the wrong confidence level for the stated TOST; it does not overturn the correct 90% interval and test result.

Diagnosing a Mismatch

When an interval-based conclusion disagrees with a TOST result, do not resolve the disagreement by choosing whichever result is more convenient. First check whether the interval confidence level matches the one-sided test level. Then check that both calculations use the same estimate, standard error, reference distribution, degrees of freedom, and equivalence limits. For equal one-sided levels, the basic mapping is:

One-sided TOST level \(\alpha\)	Matching two-sided interval	Critical-value percentile
0.10	80%	\(1-\alpha=0.90\)
0.05	90%	\(1-\alpha=0.95\)
0.025	95%	\(1-\alpha=0.975\)

The percentile column identifies the critical value for either one component one-sided test. For a t-based analysis, the corresponding t quantile must use the analysis degrees of freedom. Substituting a normal critical value for a t critical value, or using an incorrect \(df\), is a separate mismatch even if the nominal confidence percentage is correct.

A useful diagnostic is to calculate the two component test statistics directly and compare them with their one-sided critical values. If both tests reject but the interval appears not to fit inside the margins, check first for the wrong interval level or a different reference distribution. If the interval fits but one of the tests does not reject, the interval and test calculations are not using the same quantities or conventions. Under the stated assumptions, with matched inputs and critical values, the two decisions must agree.

Common Mistakes and Practical Limits

Using a 95% interval automatically because the test level is 0.05. That convention is familiar for some two-sided tests of a difference, but TOST uses two one-sided tests. At 0.05 per test, the matching two-sided interval is 90%.
Using an 80% interval because it fits the margins. A narrower interval may make equivalence look more convincing, but it corresponds to a more liberal one-sided level. It does not justify a conclusion at a stricter, prespecified level.
Changing the confidence level after seeing the estimate. The level must follow the planned one-sided test level; selecting it after observing which interval fits the margins compromises the stated error-rate interpretation.
Mixing critical values or model outputs. A correct confidence percentage paired with the wrong distribution, degrees of freedom, or standard error does not reproduce the TOST decision.

The interval equivalence is an algebraic correspondence under the chosen inferential model. It does not repair a poor model, an invalid standard error, or an unsuitable choice of equivalence limits. It also does not mean that different confidence levels are interchangeable: each level encodes a different test threshold. The direct TOST calculations are a useful cross-check, while the matched interval is often a compact way to display the estimate, uncertainty, and equivalence assessment together.

Key takeaway: Match each one-sided TOST at level \(\alpha\) to a two-sided \(100(1-2\alpha)\%\) confidence interval, using the same estimate, standard error, and reference distribution. At \(\alpha=0.05\), use a 90% interval—not a 95% interval.

Check Your Understanding

For each question, identify the correct level or decision. Solutions follow the questions.

A TOST uses one-sided \(\alpha=0.025\). What two-sided confidence level matches it?
A normal-reference TOST at \(\alpha=0.05\) has \(\hat{\theta}=0.10\), \(SE=0.08\), and limits \(-0.10\) and \(0.30\). Does the matched interval support equivalence?
Why can a 95% interval disagree with a TOST conducted at one-sided \(\alpha=0.05\)?
What should be checked if a matched confidence interval and the two component tests appear to give different decisions?

Solutions. (1) The matching level is \(1-2(0.025)=0.95\), or 95%. (2) The matched interval uses \(z_{0.95}=1.645\): \(0.10\pm1.645(0.08)=0.10\pm0.1316=[-0.0316,0.2316]\). Both endpoints are strictly between \(-0.10\) and \(0.30\), so it supports equivalence. (3) A 95% two-sided interval uses the more extreme \(z_{0.975}\) critical value and is wider; it is not the interval dual to two tests each at 0.05. (4) Verify that the interval and tests use the same estimate, standard error, equivalence limits, reference distribution, and degrees of freedom, and that the interval confidence level is \(1-2\alpha\).