Worked Example: Equivalence for a Difference Estimate

Evaluating a Difference Against Symmetric Margins

A difference estimate close to zero can be compatible with equivalence, but closeness alone is not enough. Its uncertainty matters: an estimate may lie near zero while its confidence interval still includes differences outside the acceptable range. In this tutorial, the estimate is evaluated against symmetric equivalence margins on a difference scale, using both the confidence interval and the two one-sided tests (TOST).

Let \(\delta\) denote the true difference of interest, defined in a fixed order such as test treatment minus reference treatment. Let \(\hat{\delta}\) be its estimate, \(SE\) its standard error, and \(df\) the degrees of freedom for the estimate. The prespecified symmetric margins are \(-\Delta\) and \(+\Delta\), where \(\Delta>0\). All quantities must use the same units.

Definition: Equivalence within symmetric margins means that the true difference lies strictly between the lower and upper limits: \(-\Delta<\delta<\Delta\). TOST assesses this by testing each boundary separately. At significance level \(\alpha\), equivalence is supported only when both one-sided null hypotheses are rejected.

The hypotheses are \(H_{0L}:\delta\leq-\Delta\) versus \(H_{aL}:\delta>-\Delta\), and \(H_{0U}:\delta\geq\Delta\) versus \(H_{aU}:\delta<\Delta\). The equivalence conclusion requires evidence against both null hypotheses. A result from only one test is not enough.

Model and Calculation Conditions

For a model-based estimate with a t reference distribution, the standardized estimate at a boundary has a t distribution with \(df\) degrees of freedom when that boundary is true under the model. This exact reference is appropriate, for example, when the estimate is a linear contrast in a normal-error model with its standard error estimated using the model’s residual variance and the stated degrees of freedom. With large-sample estimates, a normal reference may instead be justified by the approximation used for the estimate. The reference distribution, standard error, and degrees of freedom must come from the analysis that produced \(\hat{\delta}\); they should not be selected to make equivalence easier to show.

The calculation also presumes that the difference parameter and its direction are correctly specified, the standard error represents uncertainty in that estimate, and the margins are justified and set independently of the observed result. A valid test statistic cannot repair an inappropriate estimand or a margin chosen after seeing the estimate.

Test statistics and p-values: For the lower boundary, \(t_L=(\hat{\delta}+\Delta)/SE\) and \(p_L=P(T_{df}\geq t_L)\). For the upper boundary, \(t_U=(\hat{\delta}-\Delta)/SE\) and \(p_U=P(T_{df}\leq t_U)\), where \(T_{df}\) has a t distribution with \(df\) degrees of freedom. Reject \(H_{0L}\) when \(p_L<\alpha\), and reject \(H_{0U}\) when \(p_U<\alpha\).

The directions of the tails matter. Evidence for the lower test is in the positive direction, away from \(-\Delta\); evidence for the upper test is in the negative direction, away from \(+\Delta\). The tests are not asking whether the estimate differs from zero.

The Matching Confidence Interval

When each one-sided test uses significance level \(\alpha\), the corresponding two-sided interval is the \(100(1-2\alpha)\%\) confidence interval. With a t reference distribution, its endpoints are \(\hat{\delta}\pm t_{1-\alpha,df}SE\), where \(t_{1-\alpha,df}\) is the \(1-\alpha\) quantile of the t distribution. Thus, for \(\alpha=0.05\), the matching interval is a 90% confidence interval, not a 95% confidence interval.

\left[\hat{\delta}-t_{1-\alpha,df}SE,\ \hat{\delta}+t_{1-\alpha,df}SE\right] \subset(-\Delta,\Delta) \quad\Longleftrightarrow\quad p_L<\alpha\ \text{and}\ p_U<\alpha

The inclusion must hold for the entire interval. An interval endpoint at or beyond a margin does not satisfy the strict inclusion. The equivalence of the interval and test decisions follows directly from the boundary calculations.

Check the lower limit.
The interval’s lower endpoint exceeds \(-\Delta\) exactly when \(\hat{\delta}-t_{1-\alpha,df}SE>-\Delta\). Rearranging and dividing by positive \(SE\) gives \(t_L>t_{1-\alpha,df}\), the lower one-sided rejection condition.

Check the upper limit.
The upper endpoint is below \(+\Delta\) exactly when \(\hat{\delta}+t_{1-\alpha,df}SE<\Delta\). Rearranging gives \(t_U<-t_{1-\alpha,df}\), the upper one-sided rejection condition.

Require both conditions.
The interval lies wholly inside the margins exactly when both endpoint conditions hold. Therefore, the matching interval decision and the two-test decision agree.

Worked Calculations

Worked Example: Both Tests Support Equivalence

Hypothetical values. Suppose a model estimates a difference in exposure of \(\hat{\delta}=0.000\) concentration units, with \(SE=0.0400\), \(df=20\), and prespecified margins \(-0.100\) to \(+0.100\) concentration units. Use \(\alpha=0.05\). Assume the model and standard-error conditions stated above are appropriate, and that the margins were set before these results were examined.

Lower test. The statistic is \(t_L=(0.000+0.100)/0.0400=2.500\). Its upper-tail p-value is \(P(T_{20}\geq2.500)\approx0.01062\), which is below 0.05, so reject \(H_{0L}\).

Upper test. The statistic is \(t_U=(0.000-0.100)/0.0400=-2.500\). Its lower-tail p-value is \(P(T_{20}\leq-2.500)\approx0.01062\), also below 0.05, so reject \(H_{0U}\).

Matching interval. The 90% t critical value for \(df=20\) is approximately 1.7247. The interval is \(0.000\pm1.7247(0.0400)=0.000\pm0.06899\), or approximately \([-0.06899,0.06899]\) concentration units. Both endpoints are strictly inside \([-0.100,0.100]\). Both tests and the matching interval support equivalence within the specified margins.

Worked Example: Neither Test Clears Its Boundary

Hypothetical values. A different analysis estimates a difference in a laboratory measurement of \(\hat{\delta}=0\) units, with \(SE=0.06000\), \(df=12\), and symmetric margins \(\pm0.10002\) units. Use \(\alpha=0.05\). Treat the estimate and standard error as meeting the t-model conditions.

Lower test. Here \(t_L=(0+0.10002)/0.06000=1.667\). The upper-tail probability is \(P(T_{12}\geq1.667)\approx0.06069\), above 0.05; do not reject \(H_{0L}\).

Upper test. The statistic is \(t_U=(0-0.10002)/0.06000=-1.667\). The lower-tail probability is \(P(T_{12}\leq-1.667)\approx0.06069\), also above 0.05; do not reject \(H_{0U}\).

Matching interval. The 90% t critical value for \(df=12\) is approximately 1.7823, so the interval is \(0\pm1.7823(0.06000)=0\pm0.1069\), or approximately \([-0.1069,0.1069]\) units. It extends beyond both margins. Equivalence is not established. This does not prove that the true difference is outside the margins; the result is inconclusive about equivalence at the chosen level.

Worked Example: A Nonzero Estimate Can Still Be Equivalent

Hypothetical values. Suppose the estimated difference in a pharmacokinetic endpoint is \(\hat{\delta}=0.0200\) endpoint units, with \(SE=0.0400\), \(df=20\), and prespecified margins of \(\pm0.120\) endpoint units. Use \(\alpha=0.05\). Assume the t-based standard error is valid and the margin was justified independently of this estimate.

Lower test. \(t_L=(0.0200+0.120)/0.0400=3.500\). Since this exceeds \(t_{0.95,20}\approx1.7247\), its upper-tail p-value is less than 0.05, and \(H_{0L}\) is rejected.

Upper test. \(t_U=(0.0200-0.120)/0.0400=-2.500\). Its lower-tail probability is \(P(T_{20}\leq-2.500)\approx0.01062\), so \(H_{0U}\) is rejected as well.

Matching interval. The 90% interval is \(0.0200\pm1.7247(0.0400)=0.0200\pm0.06899\), giving approximately \([-0.04899,0.08899]\) endpoint units. It lies inside \([-0.120,0.120]\). The estimate is not exactly zero, but the tests and interval support equivalence within these margins.

Reading the Result and Diagnosing Errors

A confidence interval is often the quickest way to see the conclusion: compare both endpoints with both margins. The two p-values provide the same decision in a form that identifies which boundary, if any, remains unsupported. In this setting they are equivalent decision tools when they use the same estimate, standard error, reference distribution, degrees of freedom, and \(\alpha\). The interval also shows the range of difference values compatible with the data under its confidence procedure; it does not assign a probability to the fixed true difference.

A t critical value is appropriate when uncertainty is estimated and the stated t model applies. A normal critical value may be used when the analysis justifies a normal reference, such as a suitable large-sample approximation or known variance. Substituting normal values for a t procedure with finite degrees of freedom generally narrows the interval and changes the p-values, so the reference distribution must match the analysis rather than be chosen for convenience.

Several checks help identify a calculation that is going wrong:

Check the sign and tail. The lower statistic uses \(\hat{\delta}+\Delta\) and an upper tail; the upper statistic uses \(\hat{\delta}-\Delta\) and a lower tail.
Check the units. The estimate, margins, and standard error must all be on the same difference scale and in the same units.
Check the confidence level. At \(\alpha=0.05\), use a 90% interval to match TOST, not a 95% interval.
Check both boundaries. Passing one test does not establish equivalence; both p-values must be below \(\alpha\), and both interval endpoints must be inside the margins.
Check the inputs before rounding. A rounded test statistic, endpoint, or p-value can obscure a decision close to a threshold. Keep adequate precision during calculation and report enough digits to make the comparison clear.

Common mistake: “The estimated difference is small and a test of zero would not be significant, so the treatments are equivalent.” This reasoning confuses failure to detect a difference from zero with evidence that the true difference is sufficiently close to zero. Equivalence requires both one-sided tests to reject their boundary nulls, or, equivalently, the matching confidence interval to fit wholly inside the prespecified margins.

Key takeaway: For a symmetric difference range \((-\Delta,\Delta)\), calculate both boundary tests using their correct tails and compare the matching \(100(1-2\alpha)\%\) interval with both margins. Claim equivalence only when both boundary tests reject.

Check Your Understanding

Use the stated \(\alpha\), degrees of freedom, and margins in each question. Solutions follow the questions.

With \(\hat{\delta}=0.010\), \(SE=0.030\), \(df=20\), and margins \(\pm0.090\), calculate the two test statistics. Does each exceed its relevant \(df=20\), \(\alpha=0.05\) critical boundary in the correct direction?
Why does a TOST at \(\alpha=0.05\) correspond to a 90% two-sided interval rather than a 95% interval?
If the lower-bound test has \(p_L=0.02\) and the upper-bound test has \(p_U=0.08\), what is the equivalence conclusion at \(\alpha=0.05\)?
An interval is \([-0.07,0.11]\) units and the margins are \([-0.10,0.10]\) units. What does the interval say about equivalence?

Solutions. (1) \(t_L=(0.010+0.090)/0.030=3.333\), and \(t_U=(0.010-0.090)/0.030=-2.667\). Both exceed the corresponding one-sided boundary in their required directions because \(t_{0.95,20}\approx1.7247\); both tests reject, supporting equivalence. (2) Each one-sided test allocates \(\alpha=0.05\) to its own tail. The two tails together leave central coverage \(1-2\alpha=0.90\), giving the matching 90% interval. (3) Equivalence is not established: the upper test has \(p_U=0.08>0.05\), so both required nulls have not been rejected. (4) The upper endpoint, 0.11, lies outside the upper margin 0.10. The interval is not wholly within the equivalence range, so it does not support equivalence.