Interpreting Test–Reference Variability Comparisons

Why Compare Test and Reference Variability?

For a narrow therapeutic index (NTI) drug, a small exposure difference may have clinical importance. A replicate crossover study can therefore provide information about two distinct features of a test formulation: its mean exposure relative to the reference, and the variability of repeated responses within the same subject. A regulator may require both a mean-exposure equivalence assessment and a comparison of within-subject variability. The two assessments answer different questions and must be interpreted separately.

The variability comparison asks whether repeated administrations of the test product are more variable within a subject than repeated administrations of the reference product. It does not ask whether the average test exposure is close enough to the average reference exposure. Nor does it replace the mean-based confidence-interval criterion. The specific requirement, threshold, confidence level, and permitted model depend on the applicable jurisdiction and study guidance; they should be specified before analysis.

Some regulatory procedures express their requirement as an upper confidence bound on \(q\). For example, where the applicable FDA NTI recommendation specifies that the upper bound of a 90% confidence interval for \(\sigma_{WT}/\sigma_{WR}\) must not exceed 2.5, the variability criterion is met only when that upper bound is at most 2.5. This is a particular regulatory criterion, not a universal threshold for every jurisdiction or every NTI study. Always use the rule applicable to the product and submission.

What the Model Estimates

A replicate crossover design administers each product more than once to at least some subjects. Repeated administrations make it possible to estimate within-subject variability separately for test and reference. A suitable model for a log-transformed endpoint can include fixed effects for sequence, period, and formulation, a subject effect reflecting repeated observations, and formulation-specific within-subject residual variances. In schematic form, an observation can be written as a mean structure plus a subject contribution and a within-subject residual; the residual variance is allowed to differ by formulation.

The comparison concerns these formulation-specific within-subject variances, not the variation between different subjects in their typical exposure. It also does not compare the total observed spread of all test observations with the total spread of all reference observations: those spreads can combine between-subject and within-subject sources. The replicate structure and the specified model are needed to separate the relevant components.

The assumptions must be appropriate for the study and model. The endpoint and transformation should be defined in advance; the replicate design must contain enough repeated observations to estimate each product's within-subject variance; and the model must represent the sequence, period, subject, and residual structure relevant to the design. The variance estimates must be positive and estimable. Inference also depends on the model's approximation for the sampling distribution of the variance estimates. If observations are missing, the design is unbalanced, or the model does not fit the data structure, a simple balanced-design calculation may not be valid.

Constructing and Interpreting the Confidence Bound

A common model-based approach forms an interval for the log ratio of within-subject standard deviations. Let \(\widehat{\sigma}_{WT}^{\,2}\) and \(\widehat{\sigma}_{WR}^{\,2}\) be model estimates of the two within-subject variances. The estimated ratio is \(\hat{q}=\sqrt{\widehat{\sigma}_{WT}^{\,2}/\widehat{\sigma}_{WR}^{\,2}}\). On the log scale, \(\log(\hat{q})\) is half the difference between the log variance estimates. If the model provides an estimated standard error \(SE\{\log(\hat{q})\}\), an approximate two-sided 90% confidence interval is calculated as follows, using the appropriate critical value and degrees of freedom for the prespecified method.

The interval is for a ratio of standard deviations, not a ratio of variances. The corresponding variance ratio is \(\sigma_{WT}^{2}/\sigma_{WR}^{2}=q^2\). A threshold of 2.5 on the standard-deviation ratio therefore corresponds to 6.25 on the variance-ratio scale; comparing a variance ratio directly with 2.5 would apply the wrong threshold.

The interval calculation is approximate when it uses a normal or \(t\)-based interval for the log ratio. The standard error must reflect the model and, where relevant, the covariance between the estimated log variances. The actual regulator-specified method takes precedence over this generic form. In particular, do not assume that two variance estimates from the same subjects are independent merely because they refer to different formulations.

Worked Interpretations

Why the Log-Ratio Interval Has This Form

The confidence interval follows from expressing the target ratio on a scale where multiplicative comparisons become additive. Define \(\theta=\log(q)\). Since \(q=\sigma_{WT}/\sigma_{WR}\), taking logarithms gives \(\theta=\log(\sigma_{WT})-\log(\sigma_{WR})\). Since \(\log(\sigma)=\tfrac{1}{2}\log(\sigma^2)\), the estimated log ratio is

If the model-based sampling distribution of \(\hat{\theta}\) is adequately approximated by a \(t\) distribution with \(df\) degrees of freedom, a two-sided 90% interval for \(\theta\) is \(\hat{\theta}\mathbin{\pm}t_{0.95,df}SE(\hat{\theta})\). The exponential function is increasing, so exponentiating both endpoints preserves their order and produces an interval for \(q\). This justifies evaluating the upper endpoint on the ratio scale against a ratio threshold. The approximation can be poor with small samples or unstable variance estimates; the assumed sampling distribution and interval method must therefore be suitable for the actual design.

Diagnostics, Alternatives, and Limitations

Before interpreting the bound, check that the analysis estimates formulation-specific within-subject variability on the intended scale. Confirm that the replicate observations needed to estimate both variances are present, that the model corresponds to the actual sequence and period structure, and that the reported ratio is a standard-deviation ratio if the regulatory threshold is defined that way. Review convergence, boundary estimates, large standard errors, and sensitivity to reasonable model specifications. Such checks flag possible problems; they do not authorize replacing the prespecified primary analysis with whichever result is most favorable.

The model-based confidence interval is not interchangeable with a comparison of raw standard deviations, a test of equal variances, or a comparison of total observed variances. A raw comparison can mix within- and between-subject variation, while a conventional equal-variance test answers a different null-hypothesis question and may not implement a regulator's upper-confidence-bound rule. Conversely, the confidence-bound approach directly evaluates the specified limit while reflecting estimation uncertainty. Its limitations include dependence on the replicate design, variance model, distributional approximation, sample size, and applicable regulatory guidance.

A failure to meet the upper-bound requirement is not proof that the test product is intrinsically more variable, and meeting it does not prove the two variabilities are exactly equal. The decision is about whether the data support a particular regulatory condition. It is also not a clinical statement that a formulation causes harm. Clinical interpretation and the consequences of variability require context beyond this statistical comparison.