Why TOST Reports Need a Consistency Check
A TOST report may give two one-sided p-values, a confidence interval, equivalence margins, and a test level. These are not separate pieces of evidence: when they use the same estimate, standard error, degrees of freedom, scale, and margins, they should imply the same equivalence decision. If they do not, at least one input, calculation, or reporting convention needs review.
The purpose of this tutorial is diagnostic. Rather than repeating how to conduct an analysis from raw or summary data, it sets out the relationships that let a reader audit reported results. The checks apply to an estimate of a treatment contrast, such as a difference on its original scale or a difference between log-transformed exposure measures.
The Mathematical Link Between Tests and Intervals
For a standard t-based analysis, suppose the estimate \(\hat{\theta}\) has standard error \(SE\) and degrees of freedom \(df\). The usual model assumes that the standardized estimation error follows a t distribution under the boundary value being tested, with the stated degrees of freedom. The standard error and degrees of freedom must be appropriate for the estimator and study design. Write \(F_{df}\) for the cumulative distribution function of that t distribution.
The lower-margin test rejects when the estimate is sufficiently above \(L\); the upper-margin test rejects when it is sufficiently below \(U\). The corresponding one-sided p-values are
These are tail probabilities under the relevant boundary nulls, not probabilities that either null hypothesis is true. Under the stated t model, the boundary values \(L\) and \(U\) determine the p-values; the composite nulls include values still less favorable to equivalence. The TOST decision at level \(\alpha\) is equivalence only if \(p_L<\alpha\) and \(p_U<\alpha\).
The matching two-sided confidence interval has confidence level \(1-2\alpha\). Let \(t_{1-\alpha,df}\) be the \(1-\alpha\) quantile of the t distribution. Its endpoints are
A short derivation explains why the two reports must agree. Since \(F_{df}\) is increasing, \(p_L<\alpha\) is equivalent to \(t_L>t_{1-\alpha,df}\). Substituting the definition of \(t_L\) and rearranging gives \(\hat{\theta}-t_{1-\alpha,df}SE>L\). Similarly, \(p_U<\alpha\) is equivalent to \(t_U<-t_{1-\alpha,df}\), which rearranges to \(\hat{\theta}+t_{1-\alpha,df}SE<U\). Thus both tests reject exactly when both interval endpoints are strictly inside the margins.
The confidence level is an important part of this equivalence. A 5% TOST uses a 90% two-sided interval, not a 95% interval. A 95% interval is wider and can cross a margin even when the two 5% one-sided tests both reject. That is a comparison of different procedures, not necessarily an inconsistency.
Worked Example: Matching Two P-Values to a 90% Interval
Suppose a hypothetical analysis estimates a difference of \(\hat{\theta}=0.20\) units, with \(SE=0.10\), \(df=19\), and equivalence margins \(L=0.00\) and \(U=0.40\). The prespecified test level is \(\alpha=0.05\). Check the p-values and matching interval.
For the lower margin, \(t_L=(0.20-0.00)/0.10=2.00\). For the upper margin, \(t_U=(0.20-0.40)/0.10=-2.00\). The appropriate tails are \(P(T_{19}\geq2.00)\approx0.0300\) and \(P(T_{19}\leq-2.00)\approx0.0300\). Both p-values are below 0.05, so TOST concludes equivalence.
The matching interval is 90%, with \(t_{0.95,19}\approx1.729\). Its endpoints are \(0.20-(1.729)(0.10)=0.0271\) and \(0.20+(1.729)(0.10)=0.3729\) units. Since \(0.00<0.0271\) and \(0.3729<0.40\), the interval is strictly within the margins. The p-values and interval give the same decision.
A Practical Audit Sequence
When reported results appear inconsistent, compare the ingredients before judging the conclusion. The following sequence helps isolate the cause without recomputing an entire analysis from the beginning.
Check that the tests and interval concern the same treatment contrast, endpoint, population, and scale. A difference on the original scale and a difference on the log scale are not interchangeable quantities.
Verify that the lower test uses \(L\) and the upper test uses \(U\), and that the interval is judged against those same limits. For ratio analyses, also check that limits and results are expressed on compatible scales.
The lower-margin p-value is an upper-tail probability and the upper-margin p-value is a lower-tail probability. Compare each with the stated \(\alpha\), and use a two-sided interval at confidence level \(1-2\alpha\) for the corresponding interval check.
Check that both p-values and interval endpoints use the same \(SE\), \(df\), and model-based distribution. Different standard errors or degrees of freedom can change the decision.
Use unrounded values to resolve borderline comparisons where possible. A displayed p-value or endpoint can round to a threshold even though its unrounded value lies just above or below it.
Worked Example: A 95% Interval That Seems to Contradict TOST
Use the hypothetical estimate, standard error, degrees of freedom, and margins from the previous example, but suppose a report gives a 95% interval and says it crosses the margins. Is that enough to contradict the TOST result?
The TOST p-values remain approximately 0.0300 each, both below 0.05. But the matching interval for a 5% TOST is 90%, not 95%. With \(t_{0.975,19}\approx2.093\), the 95% interval is \(0.20-(2.093)(0.10)=-0.0093\) to \(0.20+(2.093)(0.10)=0.4093\) units. This wider interval crosses both margins, 0.00 and 0.40.
There is no inconsistency between the reported p-values and the 95% interval: they answer comparisons at different confidence levels. The correct audit is to construct the 90% interval for the 5% TOST; it is 0.0271 to 0.3729 units, and it lies within the margins. If a report explicitly claims that this 95% interval is the interval corresponding to the 5% TOST, that description is incorrect.
Scale, Tail, and Boundary Problems
A p-value can also look inconsistent with an interval because the report has used the wrong tail or compared quantities on incompatible scales. These problems are especially easy to miss when a ratio endpoint is analyzed on the log scale but reported as a ratio.
Worked Example: Auditing Results Reported on a Ratio Scale
Consider a hypothetical ratio analysis with geometric mean ratio \(1.10\), corresponding log estimate \(\hat{\theta}=\ln(1.10)\approx0.09531\), known standard error \(0.0800\) on the log scale, and margins 0.80 and 1.25 on the ratio scale. Suppose a large-sample normal procedure is used at \(\alpha=0.05\). Check the upper-margin p-value and the matching interval.
First express the margins on the log scale: \(\ln(0.80)\approx-0.22314\) and \(\ln(1.25)\approx0.22314\). The upper test statistic is \((0.09531-0.22314)/0.0800\approx-1.598\). Its p-value is the lower normal tail, \(P(Z\leq-1.598)\approx0.0550\), which exceeds 0.05. Equivalence therefore fails, regardless of the lower-margin test.
The matching 90% log-scale interval uses \(z_{0.95}\approx1.645\). Its endpoints are \(0.09531-(1.645)(0.0800)\approx-0.0363\) and \(0.09531+(1.645)(0.0800)\approx0.2269\). Exponentiating gives a ratio interval of approximately 0.9644 to 1.2547. Its upper endpoint exceeds 1.25, agreeing with the upper-test result.
A report that instead compares the log-scale upper endpoint, 0.2269, directly with the ratio margin 1.25 has mixed scales and reached no meaningful conclusion. Likewise, using the upper tail for the upper-margin test would calculate the wrong p-value: that test asks whether the estimate is sufficiently below the upper limit, so its p-value is a lower-tail probability.
Boundary conventions matter too. Under the strict decision rule \(p<\alpha\), a p-value exactly equal to \(\alpha\) does not meet the rejection criterion. Its corresponding interval endpoint is exactly on a margin, not strictly inside. This agreement can be obscured by rounding: an unrounded p-value of 0.0504 may display as 0.05, while an endpoint just inside a margin may display as equal to it. Report enough precision to make borderline results auditable, and state that the decision uses unrounded calculations.
Common Sources of Apparent Disagreement
The central diagnostic principle is to compare like with like. In particular, check whether the displayed results share the same endpoint, contrast, model, standard error, degrees of freedom, margins, and nominal test level. A difference in any of these can produce legitimately different numbers.
| What appears inconsistent | Likely issue | Diagnostic correction |
|---|---|---|
| Both p-values are below \(\alpha\), but an interval crosses a margin | The reported interval may be 95% rather than \(1-2\alpha\), or may use different uncertainty inputs. | Check the interval confidence level, \(SE\), and \(df\). |
| A p-value and endpoint seem to imply opposite decisions near a margin | Rounding, an equality convention, or a transcription error may be involved. | Compare unrounded values and apply one stated rule consistently. |
| A ratio interval looks acceptable but a p-value does not | Margins or estimates may have been compared on different scales, or a tail may be reversed. | Transform the estimate, limits, and interval to one scale and verify the relevant tail. |
| Two reports give different TOST conclusions for the same endpoint | They may use different margins, analysis populations, models, standard errors, or degrees of freedom. | Reconcile the analysis specifications before interpreting the discrepancy. |
A frequent mistake is to treat any interval crossing a margin as proof that the p-values are wrong. That reasoning omits the interval's confidence level: a 95% interval is not the matching interval for a 5% TOST. The correction is to establish the test level first, then examine the corresponding \(1-2\alpha\) interval using the same estimator and uncertainty calculation.
These checks diagnose internal consistency; they do not establish that the chosen margins, model, or analysis population are scientifically appropriate. Nor does agreement between two displays validate their common inputs. A mistaken standard error or inappropriate contrast can yield perfectly consistent but invalid results. The audit is most useful when its scope is clear: first determine whether the reported quantities agree mathematically, then assess whether the analysis choices are justified.
Check Your Understanding
Use the stated decision rule and show how each result relates to its margins.
- A 5% TOST is reported with a 95% interval. What confidence level should the matching two-sided interval have?
- For a lower margin \(L\), the test statistic is \(t_L=1.8\) with \(df=19\). Is the lower-margin p-value an upper-tail or lower-tail probability?
- A 5% TOST has an unrounded upper p-value of 0.0502. Does it meet the stated \(p<\alpha\) rule?
- A ratio analysis reports log-scale limits but compares them directly with a ratio-scale interval endpoint. What is the problem, and how should it be corrected?
Solutions. (1) The matching interval is 90%, because \(1-2(0.05)=0.90\). (2) It is the upper tail, \(P(T_{19}\geq1.8)\), because evidence for the alternative requires an estimate above the lower margin. (3) No. Since \(0.0502>0.05\), the lower or upper test in question does not reject at the stated level; equivalence requires both tests to reject. (4) The comparison mixes scales and is invalid. Transform the ratio endpoint or the margins so that the estimate, interval, and margins are all on the same scale before comparing them.