Why Model Selection Starts With the Design
A crossover model should reflect how treatments were assigned and how observations are related within each participant. A residual plot cannot repair a model that ignores the assignment structure, and a software convergence message cannot establish that the model represents the study well. A sound workflow therefore starts with the protocol and data layout, then checks the fitted mean and variance structures, and finally assesses the stability of the computation.
The earlier material on residual diagnostics, unequal residual variances, and mixed-model output provides tools for reading individual checks. Here the task is to combine them into a defensible model-selection workflow for log-transformed exposure outcomes such as AUC or Cmax. The goal is not to pick the model that produces a preferred treatment result. It is to choose a model that is estimable, consistent with the design, and supported by the observed data and analysis assumptions.
Write Down the Candidate Models
Let \(y_{ij}\) be the log exposure for participant \(i\) in period \(j\). In a two-treatment, two-period crossover, a model for the mean can include sequence, period, and treatment. One mixed-effects specification is
Here \(b_i\) is the participant-specific random intercept, and \(\varepsilon_{ij}\) is the period-specific residual. The usual random-intercept model assumes that the participant effects are independent across participants, the residuals are independent across observations conditional on the random effects, and the random effects and residuals are independent. It also assumes a common residual variance \(\sigma_e^2\) for all observations. The normal distributions are model assumptions for likelihood-based inference, not statements that every observed value must look perfectly normal.
A fixed-subject alternative includes a separate fixed effect for each participant rather than \(b_i\). In a standard crossover, each participant belongs to just one sequence. Consequently, sequence membership is determined by participant identity: a fixed effect for every participant and a fixed sequence effect are aliased, so both cannot be separately estimated in the same model. With a complete, balanced two-period crossover, analyzing within-participant differences is another way to remove participant-level differences. These approaches can give useful treatment estimates, but their assumptions, estimable terms, and handling of incomplete records differ.
A random-intercept model implies that two observations from the same participant have covariance \(\sigma_b^2\) and marginal variance \(\sigma_b^2+\sigma_e^2\), giving within-participant correlation
This is a particular covariance structure, not a universal description of every crossover. If the data suggest different residual variances by treatment or a different within-participant covariance pattern, the model may need an explicitly changed variance or covariance structure. Such changes should be motivated by the design and diagnostics, rather than added simply because one fit gives a more favorable result.
Worked Example: Letting an Incomplete Design Guide the Model
Suppose a hypothetical two-period study assigns 12 participants to sequences TR and RT, with six assigned to each. Ten participants have log exposures from both periods; two have only one observed period because a period measurement is missing. The analysis targets the mean treatment contrast while accounting for period and sequence.
Design check. Treatment changes within participant across the two periods, while sequence is between participants. Two-period complete pairs permit a within-participant contrast. The two incomplete participants do not have such a contrast, because each contributes only one treatment observation.
Candidate model. A mixed model with fixed sequence, period, and treatment effects and a participant random intercept can use the observed records, provided the model is identifiable and the missingness assumptions are defensible. A fixed-subject model can represent the observed participants, but it cannot separately estimate fixed sequence effects alongside fixed effects for each participant. The missing records do not create within-participant contrasts for the two incomplete participants.
Conclusion. The mixed model is a natural candidate when retaining incomplete participants matters, but it is not automatically correct merely because the data are unbalanced. The analyst must check that treatment and period effects are estimable, examine the covariance assumptions, and consider whether the missingness process makes the likelihood analysis appropriate. If only complete pairs are analyzed, the target population and precision may differ; that choice should be justified rather than made silently.
A Diagnostic Workflow
A useful workflow separates structural questions from checks that can be made after fitting. The first three steps establish whether the proposed analysis makes sense; the later steps assess the fitted model.
List the sequences, periods, assigned treatments, observed outcomes, and missing records. Confirm that the proposed treatment and period effects can be distinguished in the observed design.
Use fixed participant effects or within-participant contrasts when that representation matches the estimand and available records. Consider a mixed model when a population-level participant variation model is appropriate, especially with incomplete or unequal records.
Specify whether residual variance is common across treatments and whether the within-participant dependence is represented by a random intercept or another defensible structure.
Plot residuals against fitted values, period, treatment, and sequence. Look for curvature, systematic shifts, changing spread, clusters, and observations that merit investigation. A pattern suggests a model check; it does not by itself identify the cause.
Review convergence messages, parameter boundaries, standard errors, and whether plausible alternative specifications materially change the treatment estimate or its uncertainty.
Residual and Variance Patterns
For a fitted model, a residual is an observed log exposure minus its fitted value. Residual-versus-fitted plots can reveal a systematic mean pattern that the fitted terms have not captured. Residuals grouped by treatment, period, or sequence can show shifts or differences in spread. These checks need to be interpreted in light of the design: a sequence group contains different participants, whereas period and treatment observations may be repeated within participants.
A fan-shaped residual plot or consistently wider spread for one formulation calls the common-variance assumption into question. A rough comparison of residual variances can help quantify the pattern, but it is not a stand-alone decision rule. Small samples make variance estimates noisy, and residuals from a fitted model are not generally independent raw observations. A variance difference should be assessed alongside the plot, design, fitted covariance structure, and sensitivity of the treatment estimate.
If a treatment-specific residual variance is justified, the model can allow separate residual variances by treatment while retaining the design terms. The change must be specified and fitted explicitly; simply observing unequal sample spreads does not mean the model has already accounted for them. Normality checks and influential-participant checks also remain relevant, but neither justifies automatic deletion nor replaces checking the mean and covariance structure.
Worked Example: Quantifying a Residual-Spread Pattern
In a hypothetical diagnostic summary, five residuals for treatment T are \(-0.08,\ 0.02,\ 0.04,\ -0.03,\ 0.05\), and five for treatment R are \(-0.16,\ 0.04,\ 0.08,\ -0.06,\ 0.10\). Treat these as illustrative fitted residuals on the log scale, and calculate the sample variance within each group as a rough descriptive comparison.
For T, the residual mean is zero, so
For R, the residual mean is also zero, and
The descriptive variance ratio is \(s_R^2/s_T^2=0.0118/0.00295=4.00\). Thus the residual spread for R is substantially larger in this small illustration. The calculation flags a common-variance assumption for review; it does not prove that the population residual variances differ by exactly a factor of four. Inspect the residual plots and fitted model, account for the small groups and repeated-measures structure, and assess whether allowing treatment-specific variance changes the treatment estimate or its standard error.
Why Within-Participant Contrasts Help Explain the Model
A short derivation clarifies why a participant random intercept represents shared participant-level variation and why it disappears from a simple paired contrast. In the two-period model, suppose participant \(i\) receives T once and R once. Write the two observations as
Subtract the R observation from the T observation. The common participant effect cancels:
If the two residuals are independent and each has variance \(\sigma_e^2\), then
This result explains the efficiency of pairing when participant-level variation is substantial. It also shows a limitation: if residual variances differ by treatment, the variance becomes \(\sigma_T^2+\sigma_R^2\), not \(2\sigma_e^2\). Further, the simple subtraction does not itself account for period effects; treatment and period must be represented consistently with the assignment design.
Worked Example: Checking a Convergence Warning
A hypothetical mixed-model fit reports a participant variance estimate of \(0.0002\) and a residual variance estimate of \(0.0400\), both on the log scale squared. The optimizer reports convergence but also warns that the participant variance is near the boundary. Calculate the implied participant standard deviation, residual standard deviation, and intraclass correlation under the random-intercept model.
The standard deviations are the square roots of the variance estimates:
The implied correlation is
The estimated shared participant component is very small relative to residual variation, and the variance estimate is close to its allowable lower boundary of zero. This does not by itself invalidate the treatment estimate or prove that a random effect is unnecessary. Check that the optimizer converged to a stable solution, that the variance estimate is not exactly constrained at a boundary, and that standard errors and estimates agree across reasonable starting values or fitting options. Compare with a design-appropriate simpler analysis, such as a complete-pair analysis when its limitations are acceptable. Report a simpler model only if its assumptions and estimand are suitable, not just because it avoids a warning.
Convergence, Comparison, and Failure Modes
Convergence is a computational condition: the fitting algorithm has met its stopping criteria for the specified model. It is not evidence that the model is correctly specified. Conversely, a warning does not automatically mean every estimate is unusable, but it requires investigation. Check whether the fixed-effects design matrix is estimable, whether variance estimates are on boundaries, whether standard errors are finite, and whether different fitting settings lead to materially different results. A singular or near-singular covariance estimate can indicate that the data provide little information for one or more variance components.
When comparing candidate models, compare only models that represent defensible alternatives for the design and target. For likelihood-based comparisons, use the same outcome records and likelihood basis when the comparison requires it; in particular, restricted maximum likelihood likelihoods should not be compared across models with different fixed-effects structures. A likelihood criterion can support a comparison of covariance structures, but it cannot make a structurally inappropriate model appropriate. Also report whether the treatment estimate and its uncertainty are sensitive to the choice.
| Approach | Useful when | Key caution |
|---|---|---|
| Complete-pair contrast or fixed participant effects | Participants have complete treatment comparisons and participant-level control is central. | Incomplete participants may not contribute a within-participant treatment contrast; fixed participant effects cannot be separately estimated with fixed sequence effects. |
| Mixed model with participant random effect | Repeated observations and unequal or incomplete records make a population model for participant variation useful. | Depends on the random-effects and residual covariance assumptions, as well as defensible assumptions about missing records. |
| Mixed model with an alternative variance structure | Design or diagnostics provide a credible reason for unequal residual variance or a different covariance pattern. | More flexibility can be weakly supported in small samples and can complicate estimation. |
The principal failure modes are treating convergence as validation, choosing a covariance structure by the treatment p-value, interpreting a noisy variance ratio as conclusive, and adding or removing terms without considering estimability. A sequence coefficient is a between-participant comparison and should not be read as evidence of a treatment carryover effect merely because it is statistically noticeable. Likewise, residual plots diagnose model-data disagreement; they do not identify a unique correction without design knowledge and further assessment.
What the Workflow Can and Cannot Establish
This workflow helps determine whether a fixed-effects or mixed-effects representation is appropriate for the observed crossover structure and whether its assumptions are plausible. It cannot guarantee that missingness is harmless, that unobserved design problems are absent, or that a chosen covariance structure is the only reasonable one. With few participants, variance components and residual patterns can be especially uncertain. A transparent analysis records the model specification, the reason for it, the diagnostic findings, convergence issues, and sensitivity of the treatment estimate and standard error to plausible alternatives.
A useful final check is to ask whether every fixed term corresponds to a design feature or analysis objective, whether every random term has a meaningful sampling interpretation, and whether the observations support the covariance parameters being estimated. Model choice is part of the statistical argument; it is not a software setting to be selected after looking for the most appealing result.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In a crossover where every participant belongs to exactly one sequence, why can fixed participant effects and a fixed sequence effect be aliased?
- A residual plot shows greater spread for one formulation. What is the appropriate next step, and what conclusion should not be drawn from the plot alone?
- A mixed model converges but estimates a participant variance near zero. Does convergence establish that the model is adequate?
- Under a random-intercept model with independent residuals, explain why a complete within-participant treatment difference removes the participant intercept.
Solutions. (1) Sequence is constant within participant and each participant appears in only one sequence. A separate indicator for that participant therefore contains the same sequence membership information, so the sequence effect cannot be separately identified. (2) Review residuals by treatment and other design factors, quantify the spread cautiously, and assess a treatment-specific variance model if justified. The plot alone does not prove a population variance difference or specify the correct model. (3) No. Convergence reports an algorithmic stopping condition, not model adequacy. Check boundary and stability diagnostics, estimability, standard errors, and sensitivity to suitable alternatives. (4) If \(y_{iT}=\mu_T+b_i+\varepsilon_{iT}\) and \(y_{iR}=\mu_R+b_i+\varepsilon_{iR}\), then \(y_{iT}-y_{iR}=(\mu_T-\mu_R)+(\varepsilon_{iT}-\varepsilon_{iR})\); the shared term \(b_i\) cancels.