What a Confidence Interval Says in TOST
A confidence interval is central to interpreting a TOST result, but its percentage is easy to misstate. A 90% confidence interval does not mean that, after observing the data, there is a 90% probability that the fixed treatment effect lies between the observed endpoints. In frequentist analysis, the unknown parameter is fixed; the interval is random because it depends on the sample.
For a ratio endpoint, such as a geometric mean ratio, TOST is commonly conducted on the log scale. Let \(\theta\) denote the population log treatment-to-reference ratio, and let \(\hat{\theta}\) be its estimate. The corresponding ratio parameter is \(\exp(\theta)\). If the prespecified ratio limits are \(L\) and \(U\), the log-scale equivalence limits are \(\ln(L)\) and \(\ln(U)\). A confidence interval entirely within those limits supports an equivalence conclusion under the TOST procedure.
This definition describes the long-run performance of the procedure, not a probability distribution for the parameter after one particular sample has been observed. Once the data are collected, the calculated interval has definite endpoints. In the frequentist interpretation, it either contains the fixed parameter or it does not; the nominal confidence level does not tell us which is true.
Why the Coverage Interpretation Follows
The distinction can be seen directly from the construction of a symmetric confidence interval. Suppose an estimator has a standard error \(SE\), and the standardized estimation error follows a standard normal distribution under the model. If \(c\) is the critical value for the desired confidence level, the interval is \(\hat{\theta}\pm cSE\). The event that this interval covers \(\theta\) is equivalent to the estimator being within \(cSE\) of the fixed parameter.
Here \(Z\sim N(0,1)\), and the probability is taken over repeated samples generated under the model with the same fixed \(\theta\). For a 90% interval, \(c\) is chosen so that the middle 90% of the standard normal distribution lies between \(-c\) and \(c\). When a standard error is estimated or the estimator is only approximately normal, the corresponding \(t\)-based or approximate procedure requires its own assumptions; the coverage interpretation remains about repeated sampling, with exact or approximate coverage as appropriate.
For TOST, the interval is used to assess whether the plausible values supported by the procedure lie within the equivalence limits. This is a decision rule based on the observed estimate and its uncertainty. It does not turn the parameter into a random quantity or give the parameter a probability of being in the margins.
Worked Example: Interpreting a 90% Log-Ratio Interval
Suppose a hypothetical analysis estimates the population log ratio as \(\hat{\theta}=0.03922\), with \(SE=0.04000\). Assume the estimator is adequately approximated by a normal distribution, the standard error is valid for the analysis, and a 90% interval using \(c=1.6449\) is appropriate. The hypothetical equivalence limits are 0.80 and 1.25 on the ratio scale.
Calculate the log-scale interval. Its half-width is \(1.6449(0.04000)=0.06580\). Therefore,
The log-scale equivalence limits are \(\ln(0.80)=-0.22314\) and \(\ln(1.25)=0.22314\). The interval lies inside these limits. Back-transforming gives an estimated ratio of \(\exp(0.03922)=1.040\) and interval \[ [\exp(-0.02658),\exp(0.10502)]=[0.9738,\ 1.1107]. \] The ratio interval is contained within [0.80, 1.25], so this hypothetical result supports equivalence under the specified TOST analysis.
Interpretation. The 90% figure describes the repeated-sampling coverage of the interval procedure under the stated assumptions. It is not correct to say, “There is a 90% probability that the true ratio is between 0.9738 and 1.1107.” A correct statement is: “The estimated ratio was 1.040, with a 90% confidence interval of 0.9738 to 1.1107; the interval was contained within the prespecified equivalence limits.”
Repeated Samples: What the Percentage Refers To
Imagine repeating the same hypothetical study many times, with the same design, sample size, analysis method, and fixed population parameter. Each repetition would produce a different estimate and usually a different interval. If the procedure has exact 90% coverage, about 90% of those intervals would include the fixed parameter in the long run. The percentage is not a promise that exactly 90% of any finite batch of intervals will cover it.
Worked Example: A Hypothetical Coverage Exercise
Suppose, purely for illustration, that a valid procedure is applied independently to 1,000 hypothetical repeated samples and has exact 90% coverage for the same fixed log-ratio parameter. Let \(C\) be the number of intervals that contain that parameter. Each interval has probability 0.90 of covering it under this repeated-sampling setup, so \(C\sim\mathrm{Binomial}(1000,0.90)\).
The expected number of covering intervals is \[ E(C)=1000(0.90)=900. \] The standard deviation of this count is \[ \sqrt{1000(0.90)(0.10)}=\sqrt{90}=9.487. \] Thus, 900 is the expected count, not a guaranteed count. In any single repetition, the calculated interval either contains the fixed parameter or does not. The coverage statement describes the procedure across repetitions; it does not assign a 90% probability to the parameter being inside one observed interval.
This thought experiment also clarifies why coverage is conditional on the sampling setup and analysis being appropriate. If the model gives a poor description of the data, the standard error is wrong, or the stated sampling assumptions fail, the actual long-run coverage may differ from the nominal level. The interval’s label alone cannot establish that the procedure has achieved its intended coverage.
Careful Language for an Interval Near an Equivalence Limit
A confidence interval can be used to make a frequentist equivalence decision, but the decision and a probability claim are different things. In TOST, an interval wholly within the prespecified limits supports equivalence at the corresponding testing setup. An interval extending beyond a limit does not establish equivalence by that rule. Neither outcome tells us the probability that the fixed parameter is inside or outside the margins.
Worked Example: Reporting an Interval That Extends Beyond a Margin
Consider a hypothetical 90% ratio-scale confidence interval of [0.78, 1.18], with prespecified equivalence limits of [0.80, 1.25]. To compare on the log scale, transform the observed endpoints: \[ \ln(0.78)=-0.24846,\qquad \ln(1.18)=0.16551. \] The log-scale interval is therefore approximately [-0.24846, 0.16551], while the log-scale limits are \(\ln(0.80)=-0.22314\) and \(\ln(1.25)=0.22314\). The interval extends below the lower equivalence limit, so it is not wholly contained within the limits and does not support an equivalence conclusion under this rule.
The appropriate interpretation is that the interval procedure produced a range extending beyond the lower margin; the TOST criterion for establishing equivalence was not met. It is not appropriate to infer that there is a 90% probability that the true ratio is below 0.80, or that there is a 90% probability that it is within the margins. The interval’s coverage level supplies neither probability.
What a Confidence Interval Is—and Is Not
Several statistical summaries can look like ranges while answering different questions. Distinguishing their targets prevents a confidence interval from being given an interpretation it was not designed to support.
| Summary | What it describes | Parameter probability in the frequentist interpretation? |
|---|---|---|
| Confidence interval | A repeated-sampling procedure for estimating a fixed parameter | No |
| TOST confidence interval | Uncertainty in an estimate assessed against equivalence limits | No; it supports a decision rule |
| Bayesian credible interval | A posterior distribution for a parameter, conditional on data and prior assumptions | Yes, as a posterior probability statement under that Bayesian model |
| Prediction interval | Uncertainty about a future observation or outcome | It concerns a future outcome, not probability for the fixed parameter |
A Bayesian credible interval can support a statement about posterior probability, but it relies on a prior distribution and a probability model for the parameter. It is not simply another interpretation of a frequentist confidence interval. A prediction interval has a different target again: future observations, which include both estimation uncertainty and outcome variability.
Diagnostics, Limitations, and Reporting
Before interpreting a TOST confidence interval, verify that it is the interval for the intended estimand, that its scale is clear, and that its method and assumptions are suitable. For a ratio endpoint, distinguish the log-scale parameter and limits from their exponentiated ratio-scale versions. A numerically correct interval can still be misreported if its scale or units are mislabeled.
- Check the model and standard error. A normal or \(t\)-based interval depends on the relevant sampling assumptions and a valid estimate of uncertainty. Sparse data, strong departures from the assumed model, or dependence handled incorrectly can compromise nominal coverage.
- Keep the estimand fixed. An interval for one treatment contrast or endpoint does not automatically describe another. State which population parameter the estimate targets.
- Separate coverage from the TOST decision. The confidence level describes the interval procedure; containment within the margins is the criterion used for the equivalence conclusion.
- Avoid posterior wording. Do not write “the parameter has a 90% chance of being in this interval” unless the analysis is explicitly Bayesian and the statement refers to its posterior distribution.
A frequentist interval also does not describe the range containing a given percentage of individual patients’ exposures. It concerns uncertainty about the specified population parameter. Nor does it establish the probability that a future study will reach the same TOST decision. Those questions require different summaries or analyses.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A valid 95% confidence-interval procedure is repeated many times under the same model. What does the 95% refer to?
- After observing one 90% confidence interval, can a frequentist say there is a 90% probability that the fixed parameter lies inside it? Why or why not?
- A ratio-scale TOST interval is wholly contained within the prespecified equivalence limits. What conclusion is supported, and what does the confidence level not mean?
- Why might a procedure labelled 90% fail to have 90% coverage in practice?
- Which type of interval can support a posterior probability statement about a parameter, given its model and prior assumptions?
Solutions. (1) Across repetitions, about 95% of intervals produced by the procedure contain the fixed parameter, assuming the procedure’s model and conditions hold. (2) No. The parameter is fixed in the frequentist framework; the observed interval either contains it or does not. The 90% refers to the long-run coverage of the procedure. (3) The result supports equivalence under the specified TOST rule. The confidence level does not mean there is that percentage probability that the fixed parameter lies inside the observed interval. (4) The model, sampling assumptions, or standard-error calculation may be inadequate, so actual coverage can differ from nominal coverage. (5) A Bayesian credible interval, interpreted conditional on its prior and probability model.