Calculating the Two TOST P-Values

Why TOST Has Two P-Values

An equivalence conclusion requires evidence against two different possibilities: that the true effect is too small and that it is too large. The two one-sided tests (TOST) procedure therefore produces two p-values, one for each equivalence boundary. Neither p-value alone answers whether both conditions for equivalence are met.

Let \(\theta\) be the treatment effect on the analysis scale, with lower and upper equivalence bounds \(L\) and \(U\), where \(L<U\). Let \(\hat{\theta}\) be its estimate and \(SE\) its standard error. Both bounds and the estimate must be expressed on the same scale and in the same units. The lower-bound test asks whether the effect is greater than \(L\); the upper-bound test asks whether it is less than \(U\).

Definition: The lower-bound p-value, \(p_L\), is the upper-tail probability for testing \(H_0:\theta\leq L\) against \(H_a:\theta>L\). The upper-bound p-value, \(p_U\), is the lower-tail probability for testing \(H_0:\theta\geq U\) against \(H_a:\theta<U\). Equivalence is concluded at significance level \(\alpha\) only when both p-values are no greater than \(\alpha\).

Suppose the appropriate reference distribution has cumulative distribution function \(F\). For a t reference distribution, write \(F_{t,df}\), where \(df\) is the degrees of freedom. For a standard normal reference distribution, write \(\Phi\). The two observed statistics are the estimate's standardized distances from the respective bounds:

T_L=\frac{\hat{\theta}-L}{SE}, \qquad T_U=\frac{\hat{\theta}-U}{SE}.

The p-values use different tails:

p_L=1-F(T_L), \qquad p_U=F(T_U).

Thus, for a t reference distribution, \(p_L=1-F_{t,df}(T_L)\) and \(p_U=F_{t,df}(T_U)\). For a standard normal reference distribution, replace \(F_{t,df}\) with \(\Phi\). A lower-bound result that is far above \(L\) gives a large positive \(T_L\) and a small upper-tail p-value. A result far below \(U\) gives a negative \(T_U\) and a small lower-tail p-value.

Why Each Probability Is Evaluated at Its Boundary

Each null hypothesis is composite: the lower-test null includes every value \(\theta\leq L\), and the upper-test null includes every value \(\theta\geq U\). A one-sided p-value for a composite null is calculated using the null value that makes the observed result least surprising. Under the usual location model for the estimate, that value is the corresponding boundary.

For example, suppose \(\hat{\theta}\) has standard error \(SE\), and under the reference model its standardized value is centered at the true \(\theta\). For the lower test, the probability of obtaining a statistic at least as large as the observed one is greatest among \(\theta\leq L\) when \(\theta=L\). Values below \(L\) shift the statistic downward, reducing that upper-tail probability. Therefore the lower-test p-value is the upper-tail probability calculated at \(L\). The same reasoning, in the opposite tail, gives the upper-test p-value at \(U\).

\sup_{\theta\leq L}P_{\theta}(T_L\text{ at least as large as observed}) = P_{L}(T_L\text{ at least as large as observed}),

and the upper-test null's largest probability of a result at least as far toward equivalence is attained at \(\theta=U\). These boundary probabilities are the p-values used in the procedure.

Choosing and Using the Reference Distribution

The formulas require a valid reference distribution for the standardized estimate under each boundary null. In a conventional model-based analysis, that may be a t distribution with the model's appropriate \(df\), or a standard normal distribution when the analysis justifies a normal reference. The estimated standard error must correspond to \(\hat{\theta}\), and the uncertainty model used to obtain it must be appropriate for the design and analysis.

For a t test, use the degrees of freedom attached to the estimate and its standard error. Do not substitute the number of subjects for \(df\) automatically: model-based degrees of freedom may depend on the fitted model. For a normal test, the standardized values are z statistics and the tails are evaluated using \(\Phi\). Use the distribution specified by the valid analysis; choosing whichever distribution gives smaller p-values is not justified.

The estimate, bounds, and standard error must all use compatible units. If the estimate and bounds are on a log scale, the standard error must also be on that scale. The numerical ratios in the formulas are dimensionless, as required for comparison with a t or standard normal distribution.

Worked Example: One Test Passes but the Other Does Not

Hypothetical data. A fitted model gives a treatment-effect estimate \(\hat{\theta}=0.24\) log-scale units, \(SE=0.10\) log-scale units, and \(df=12\). The equivalence bounds are \(L=-0.20\) and \(U=0.24\) log-scale units. Calculate both p-values at \(\alpha=0.05\).

Lower-bound test. The standardized distance above the lower bound is

T_L=\frac{0.24-(-0.20)}{0.10} =\frac{0.44}{0.10} =4.40.

This is an upper-tail test, so

p_L=1-F_{t,12}(4.40)\approx 0.0004327.

Upper-bound test. Here the estimate is exactly at the upper bound:

T_U=\frac{0.24-0.24}{0.10}=0, \qquad p_U=F_{t,12}(0)=0.5000.

The estimate provides strong evidence that the effect exceeds the lower bound, but no evidence that it is below the upper bound. Since \(p_U>0.05\), the two tests do not jointly support equivalence. The conclusion is not that the true effect equals the upper bound; the test simply does not reject the upper-bound null.

Using the Two P-Values Together

The alternative hypothesis for equivalence is the conjunction \(L<\theta<U\). Both one-sided null hypotheses must therefore be rejected. At significance level \(\alpha\), the decision rule is:

\text{Conclude equivalence if and only if } p_L\leq\alpha \quad\text{and}\quad p_U\leq\alpha.

Equivalently, the overall TOST p-value can be reported as \(\max(p_L,p_U)\), and equivalence is concluded when this maximum is no greater than \(\alpha\). The maximum is the relevant p-value for the intersection-union decision because failure of either component test prevents the joint alternative from being established.

Worked Example: Both Normal-Reference Tests Pass

Hypothetical data. An analysis gives \(\hat{\theta}=0.02\), \(SE=0.04\), and equivalence bounds \(L=-0.10\) and \(U=0.12\), all in the same analysis-scale units. Assume the specified reference distribution is standard normal. Calculate the p-values and apply \(\alpha=0.05\).

For the lower bound,

T_L=\frac{0.02-(-0.10)}{0.04} =\frac{0.12}{0.04} =3.00, \qquad p_L=1-\Phi(3.00)\approx 0.001350.

For the upper bound,

T_U=\frac{0.02-0.12}{0.04} =\frac{-0.10}{0.04} =-2.50, \qquad p_U=\Phi(-2.50)\approx 0.006210.

Both p-values are below 0.05, so both one-sided null hypotheses are rejected and the TOST decision supports equivalence at the 5% level. The larger p-value is approximately 0.006210; it is still below the prespecified threshold.

A t-Reference Example and Tail Checks

With a t reference distribution, the procedure is identical except that the appropriate t CDF and degrees of freedom are used. The sign of each statistic is an important check: \(T_L\) should increase as the estimate moves upward from \(L\), while \(T_U\) should become more negative as the estimate moves downward from \(U\). These directions should agree with the tails used to calculate the p-values.

Worked Example: Calculating Both t-Reference P-Values

Hypothetical data. Suppose \(\hat{\theta}=0.00\), \(SE=0.06667\), and \(df=18\), with bounds \(L=-0.15\) and \(U=0.15\). All values are on the same analysis scale. Calculate both p-values at \(\alpha=0.05\).

For the lower test,

T_L=\frac{0.00-(-0.15)}{0.06667} \approx 2.250, \qquad p_L=1-F_{t,18}(2.250) \approx 0.01860.

For the upper test,

T_U=\frac{0.00-0.15}{0.06667} \approx -2.250, \qquad p_U=F_{t,18}(-2.250) \approx 0.01860.

The equal p-values arise from the symmetric bounds, estimate, and reference distribution in this example. Both are below 0.05, so the joint decision supports equivalence. In a nonsymmetric case the two p-values need not be equal, and each must be calculated separately.

Interpretation, Diagnostics, and Common Errors

A p-value is a probability calculated under a specified null hypothesis and reference model. For example, \(p_U=0.0186\) means that, if the upper-bound null were true at its least-favourable boundary and the model assumptions held, the probability of a result at least as far in the direction of equivalence as the observed result would be 0.0186. It is not the probability that the null is true, nor the probability that the true effect lies within the equivalence bounds.

Before reporting the joint decision, check that the estimate, standard error, and both bounds refer to the same parameter and scale; that the correct reference distribution and degrees of freedom were used; and that the correct tail was evaluated for each test. Earlier material on residual diagnostics and crossover-model estimates is relevant to whether the estimate, standard error, and reference model are defensible. A correct CDF calculation cannot repair an inappropriate model or standard error.

A frequent error is to use the same tail for both tests. The lower test looks for evidence that the estimate is above \(L\), so it uses an upper tail. The upper test looks for evidence that the estimate is below \(U\), so it uses a lower tail. Another error is to use a two-sided p-value for either component: TOST consists of two one-sided tests, not two two-sided tests. Finally, a small p-value for one test is not enough. The decision is determined by the larger of the two p-values.

The p-value calculation is a direct way to express the two tests. An equivalent confidence-interval rule can express the same TOST decision using an interval and the bounds; it is useful as a cross-check but does not change the need for both one-sided conditions. The p-values are especially useful when the analysis report provides an estimate, standard error, and reference distribution but not the two component test results.

Key takeaway: Calculate \(p_L\) from the upper tail at the lower bound and \(p_U\) from the lower tail at the upper bound. Equivalence is supported only when both are no greater than the prespecified \(\alpha\); equivalently, their maximum must be no greater than \(\alpha\).

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Write the formulas for \(p_L\) and \(p_U\) using a reference CDF \(F\).
A lower-bound statistic is \(T_L=1.80\). Which tail gives \(p_L\), and why?
With a standard normal reference, an upper-bound statistic is \(T_U=-1.50\). What is \(p_U\), approximately?
At \(\alpha=0.05\), suppose \(p_L=0.012\) and \(p_U=0.071\). What is the joint decision?
Why is the overall TOST p-value the larger of the two component p-values?

Solutions. (1) \(p_L=1-F(T_L)\) and \(p_U=F(T_U)\). (2) Use the upper tail, \(1-F(1.80)\), because the lower test's alternative is \(\theta>L\). (3) \(p_U=\Phi(-1.50)\approx 0.0668\). (4) Equivalence is not concluded: although the lower test rejects, \(p_U>0.05\), so the upper-bound null is not rejected. (5) Both component null hypotheses must be rejected; the larger p-value is the one that can prevent the joint decision from meeting the significance threshold.