Detecting Unequal Residual Variances Across Formulations

Why Compare Residual Variances by Formulation?

A crossover analysis commonly models a log-transformed pharmacokinetic endpoint, such as log AUC or log Cmax. Its mean model accounts for effects such as formulation, period, and sequence, while its covariance model accounts for repeated observations from the same subject. A further assumption may be that the remaining, conditional residual variability is the same under test and reference. If that assumption is poor, standard errors and likelihood-based comparisons that rely on the common-variance model may be unreliable.

The diagnostic question is specific: after fitting an appropriate mean and within-subject covariance structure, do test and reference observations have different residual variances? A difference in observed spread alone does not answer this. Unequal means, an omitted treatment-by-period effect, influential subjects, or a misspecified covariance structure can also produce formulation-specific patterns in residuals.

Model Setup and Assumptions

Let \(y_{ij}\) be the log endpoint for subject \(i\) in period \(j\), and let \(F_{ij}\) identify the formulation administered. A convenient illustration is a mixed model with a subject random intercept:

Here \(\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta}\) represents the fixed-effects mean structure, \(b_i\) induces within-subject correlation, and the residual variance is indexed by formulation. In the common-variance model, \(\sigma_T^2=\sigma_R^2=\sigma^2\). In the formulation-specific model, \(\sigma_T^2\) and \(\sigma_R^2\) are estimated separately. More elaborate crossover models can use another prespecified subject covariance structure; the comparison still concerns the residual variances while holding the mean and the rest of the covariance specification fixed.

The comparison assumes that the mean model is adequate, subjects are independent of one another, and the within-subject covariance structure is specified appropriately. For a Gaussian likelihood analysis, the conditional residual distributions are also assumed normal. The variance model must be identifiable: the data need enough observations under each formulation, with the design and covariance structure able to distinguish residual from subject-level variability. In small samples, likelihood-ratio reference distributions can be inaccurate, so the test should be treated cautiously or calibrated by a suitable method such as a parametric bootstrap.

A random subject intercept contributes to the marginal variance of every observation, while the formulation-specific residual term contributes an additional variance that depends on treatment. Consequently, a difference in total observed spread is not automatically a difference in residual variance. This distinction matters especially in replicate designs or models with random treatment effects.

Diagnose Before Testing

Begin with residuals from the fitted crossover model, not with unadjusted log observations. Compare residual spread by formulation using plots and summaries, while retaining subject and period information. A residual-versus-fitted plot coloured by formulation can reveal whether one formulation has a wider band of residuals across the fitted range. A subject-linked plot can show whether an apparent difference is driven by a few paired observations or by a general pattern.

Useful summaries include formulation-specific residual standard deviations and their ratio. They are descriptive diagnostics, not automatically unbiased estimates or a formal test: fitted residuals have constraints imposed by estimation, and repeated observations from a subject are correlated. Interpret the plot alongside the mean-model diagnostics already used for log-scale crossover models. In particular, a systematic residual trend or period-specific pattern suggests investigating the mean or covariance model before attributing the spread to formulation.

Comparing the Nested Variance Models

To formally compare the models, fit them with the same fixed-effects mean structure and the same subject covariance structure. The common-variance model is nested in the formulation-specific model because it is obtained by imposing the single constraint \(\sigma_T^2=\sigma_R^2\). Let \(\ell_C\) and \(\ell_S\) denote the maximized log likelihoods under the common and separate variance models. The likelihood-ratio statistic is

Under the null hypothesis \(H_0:\sigma_T^2=\sigma_R^2\), the alternative is \(H_a:\sigma_T^2\ne\sigma_R^2\). With regularity conditions and enough independent subjects, \(\Lambda\) is approximately distributed as a chi-squared random variable with one degree of freedom under \(H_0\). The one degree of freedom comes from releasing one equality constraint. The approximation is not a guarantee of accurate calibration in a small study.

A short justification follows from the nested models: the separate-variance model includes every parameter setting allowed by the common-variance model, so its maximized likelihood cannot be smaller. Thus \(\ell_S\geq\ell_C\) and \(\Lambda\geq0\). Under the null and regular large-sample conditions, the likelihood gain from estimating one additional free parameter has the chi-squared-one reference distribution. This is an approximate model-comparison test, not a test of whether the formulations have equal mean exposure.

Maximum likelihood can be used for both fits. Restricted maximum likelihood (REML) can also be used for this comparison when the fixed-effects design is identical in the two models; changing the fixed-effects structure changes the restricted likelihood basis and makes that comparison inappropriate. Whichever method is chosen, keep all other model features fixed so that a likelihood difference is attributable to the variance specification rather than to several simultaneous changes.

Why the Simple F Test Is Often Not the Answer

For two independent samples of normally distributed observations, an exact variance-ratio test can use \(s_T^2/s_R^2\) and an F distribution with degrees of freedom based on the two sample sizes. Its exactness depends on independent observations, normal populations, and sample variances calculated from those independent observations. Crossover residuals generally do not meet this setup: each subject can contribute both formulations, residuals are estimated after fitting a model, and their covariance may reflect repeated measures.

Therefore, applying an ordinary F test to the two residual columns can give a misleading p-value even if the variance ratio itself is a useful descriptive summary. A model-based likelihood comparison preserves the specified subject covariance structure. Robust alternatives such as Levene-type tests can be useful in appropriate independent-group settings, but applying them directly to correlated crossover residuals does not automatically solve the dependence problem. The analysis must match the sampling and covariance structure.

Failure Modes, Interpretation, and Limitations

• Comparing raw spreads. Raw observations combine mean differences, subject effects, and residual variation. Compare residuals conditional on the fitted model instead.
• Ignoring pairing. Treating test and reference residuals as independent can invalidate a simple F-test calculation when the same subjects contribute both formulations.
• Changing several model features at once. If the fixed effects or subject covariance structure also changes, the likelihood difference no longer isolates the variance assumption.
• Overinterpreting a small p-value. Statistical evidence for unequal variances does not show that the mean treatment effect is biased, nor does it establish clinical importance. Report the estimated variance ratio and its uncertainty.
• Using variance heterogeneity to mask a poor mean model. A variance model does not repair omitted period structure, treatment-by-period patterns, or influential observations. Inspect these possibilities before drawing a formulation-specific conclusion.

A formulation-specific residual model can improve the description of uncertainty when the spread genuinely differs, but it costs an additional parameter and may be unstable when data are sparse or highly unbalanced. Conversely, a common-variance model is more parsimonious and may be adequate even when sample variances look different by chance. Model choice should combine the likelihood comparison with residual patterns, design knowledge, parameter precision, and the consequences for the target inference. A non-significant test is not proof of equal variances; it may reflect limited information.