Mixed-Effects Models for NTI Replicate Crossover Studies

Why the Model Must Reflect the Crossover

A replicate crossover study measures each participant under more than one treatment and repeats at least one treatment within a participant. The resulting observations are not independent: measurements from the same participant can share a subject-specific baseline, and the period in which a treatment is given can matter. A mixed-effects model represents these features explicitly while estimating the treatment contrast needed for a bioequivalence analysis.

The focus here is the model specification: which terms describe the design, which term represents subject-to-subject variation, and what assumptions make the model interpretable. For illustration, assume a four-period, two-sequence design, with sequences TRTR and RTRT. Both test (T) and reference (R) are administered twice to each participant. The same framework applies to other replicate schedules, provided the model terms match the actual design.

Model Terms and Their Roles

A useful starting model is

Here, \(\mu\) is an overall mean; \(S_i\) is the fixed effect of sequence \(i\); \(P_k\) is the fixed effect of period \(k\); and \(\tau_{d_{ik}}\) is the fixed treatment effect for the treatment given in that period. The term \(b_{j(i)}\) is the random effect for subject \(j\) nested in sequence \(i\). The residual \(\varepsilon_{ijk}\) is the within-subject deviation not explained by those terms.

Sequence, period, and treatment are fixed effects because they are design factors whose specific levels are of interest in this study. Subjects are typically modeled as random effects: the enrolled subjects are viewed as a sample from a broader population of potential participants, and the model represents their differing baseline levels. Nesting matters because each participant follows one assigned sequence; a subject number such as “subject 3” identifies a participant within that sequence, not a participant shared across sequences.

One common distributional specification is \(b_{j(i)}\sim N(0,\sigma_b^2)\), with independent subject effects across participants. Conditional on the subject effects and fixed terms, residuals are assumed to have mean zero, to be independent across subjects, and to follow a prespecified within-subject covariance structure. A simple version uses independent normal residuals with one common variance. If the protocol or analysis plan calls for different residual variances by treatment, that structure must be specified explicitly; it is not implied merely by calling the analysis a mixed-effects model.

The model also assumes that the specified fixed-effect structure is adequate. In the basic additive model, treatment effects do not change by period or sequence, and there is no unmodeled carryover effect that distorts the current-period response. Appropriate washout and study conduct are therefore substantive assumptions, not just software settings. The endpoint and its transformation must also be suitable: for the usual bioequivalence interpretation, the response is commonly the natural logarithm of a positive PK endpoint.

Identifying the Treatment Contrast

Use a reference-coded treatment effect by setting \(\tau_R=0\) and \(\tau_T=\delta\). Then \(\delta\) is the model-based test-minus-reference difference on the log scale. Its exponentiation is a ratio of geometric means on the original scale:

The treatment contrast is a fixed-effect contrast, not a comparison of two unrelated groups. Subject effects account for the shared baseline of repeated observations, while period and sequence terms account for design features that might otherwise be mixed into the treatment estimate. The estimate and its standard error depend on the design, covariance specification, and estimation method. The model by itself does not determine the acceptance limits or replace the applicable bioequivalence decision rule.

A Small Numerical Illustration of the Contrast

In a complete, balanced design with equal residual variances, a treatment contrast can be illustrated by the marginal means of the log responses. In a real analysis, the mixed model estimates the contrast using all observations and the specified covariance structure; the simple calculation below illustrates its scale and interpretation.

Why Repeated Measurements Need a Subject Effect

A short variance calculation shows why within-subject dependence matters. For a simplified subject-intercept model, write one log response under each treatment as \(Y_T=\mu_T+b+\varepsilon_T\) and \(Y_R=\mu_R+b+\varepsilon_R\). If the subject effect \(b\) has variance \(\sigma_b^2\), and residuals have variance \(\sigma_e^2\) and are independent conditional on \(b\), then the covariance between the two responses is \(\sigma_b^2\). Subtracting the responses cancels the shared subject effect:

If those observations were incorrectly treated as independent, their marginal variances would each be \(\sigma_b^2+\sigma_e^2\), and the variance of their difference would be calculated as \(2(\sigma_b^2+\sigma_e^2)\). That calculation fails to account for the positive covariance from the shared subject effect. Replicate designs have more observations per subject, so the appropriate covariance structure must reflect the actual repeated-treatment schedule, rather than treating every period as a new independent participant.

Estimation, Diagnostics, and Alternatives

Mixed-effects models are commonly fitted by maximum likelihood or restricted maximum likelihood (REML). Fixed-effect estimates provide the treatment contrast; inference also requires a standard error and degrees-of-freedom approach appropriate to the fitted model and analysis plan. Software output should be checked against the design: verify the treatment sequence in every period, the subject-within-sequence identifiers, the reference coding, the covariance specification, and the contrast direction before interpreting estimates.

Useful diagnostics include checking residual patterns against fitted values and period, reviewing the distribution of subject-level effects where relevant, and confirming that the model converged without warnings. Examine whether treatment and period combinations are represented as expected and whether missing observations have left a design cell unsupported. A singular fit, an implausible variance estimate, large residual patterns by period, or convergence warnings can indicate that the covariance structure is too complex, data are insufficient for it, or the model does not match the study.

A paired analysis can be a reasonable simplification for a simple two-treatment, two-period crossover when its assumptions and design are appropriate. It does not directly represent a multi-period replicate schedule with period and sequence effects. A fixed-subject model can also represent participant differences, but it treats the observed participants as fixed levels and may consume many parameters; the random subject-within-sequence term is the usual population-oriented representation. Neither alternative excuses checking the design or covariance assumptions.

The basic random-intercept model has limitations. It may be inadequate when within-subject correlation is not captured by a shared intercept, when residual variability differs in a way the model omits, or when period-related patterns remain. Adding terms indiscriminately is not a solution: model complexity must be supported by the design and prespecified analysis. In particular, a treatment contrast estimated from an incorrectly coded or poorly supported model can be precise-looking but misleading.