Assessing Normality of Crossover Model Residuals

Why Residual Normality Is the Relevant Question

A crossover study can have strongly right-skewed exposure measurements on the original scale and still be adequately described by a model whose errors are approximately normal on the log scale. Conversely, log transformation does not guarantee normal model errors. The diagnostic question is therefore not whether the observed AUC or Cmax values look bell-shaped; it is whether the residuals are consistent with the error assumptions of the fitted crossover model.

In a log-scale crossover analysis, the model represents each log exposure using specified effects, such as treatment, period, sequence, and subject. Residuals are the differences between observed log exposures and the values fitted by that model. Normality is an assumption about the model errors after accounting for those effects, not a requirement that every set of observations or every covariate be normally distributed.

The interpretation depends on the model. In a fixed-effects crossover ANOVA, normal errors are commonly assumed to be independent with a common variance after conditioning on the design effects. In a mixed model, the assumptions also concern the random effects and the residual covariance structure; a marginal residual plot must be interpreted in light of that structure. A residual diagnostic cannot validate independence, variance assumptions, or the chosen covariance structure by itself.

Quantile Plots and Their Justification

A normal quantile-quantile plot compares ordered observed residuals with the ordered quantiles expected from a normal distribution. If the assumed model is reasonable, the points should lie approximately along a straight line. The line need not pass through zero in a finite sample, although residuals from models with an intercept often sum to zero.

One way to construct the plot is to order the \(n\) residuals, \(e_{(1)} \le \cdots \le e_{(n)}\), and pair each with a theoretical normal score. A common plotting position is \(p_i=(i-0.375)/(n+0.25)\), with score \(z_i=\Phi^{-1}(p_i)\), where \(\Phi^{-1}\) is the standard normal quantile function. Plot \(e_{(i)}\) against \(z_i\). Other standard plotting-position rules give slightly different scores, especially in small samples, but should not change the main diagnostic pattern.

This distinction explains why a small amount of scatter around a line is expected, and why a formal goodness-of-fit calculation that treats fitted residuals as independent normal observations may not have its nominal reference distribution. In mixed-model software, conditional and marginal residuals can also differ; use residuals appropriate to the fitted model and understand which type the software reports.

Targeted Checks Beyond the Plot

A quantile plot is most informative when interpreted alongside checks aimed at specific departures. Examine whether residuals show asymmetry, unusually heavy or light tails, or a pattern tied to design features. For crossover data, useful displays can distinguish periods, treatments, or sequences when the model and design permit those comparisons. Such grouping can reveal a systematic pattern that a pooled plot conceals. This is a targeted assessment of the normality assumption; it is not a substitute for assessing variance or covariance assumptions.

Descriptive summaries can help describe a pattern but should not become rigid pass/fail rules. For example, the moment-based skewness \(g_1=m_3/(m_2)^{3/2}\), where \(m_2\) and \(m_3\) use divisor \(n\) for the second and third centered moments, summarizes asymmetry. Its value is sensitive to extreme observations and sample size. It should be read with the plot and the study context, not used alone to certify normality.

Do Not Confuse Raw-Scale Skewness with Residual Nonnormality

Exposure measurements are positive and often vary multiplicatively. Their raw-scale distribution can be right-skewed even when log-scale errors are reasonably symmetric. The treatment of log exposures in crossover analysis separates a subject’s overall exposure level and modeled design effects from the remaining deviations. Residual normality concerns those deviations, not the raw distribution across subjects.

Alternative Checks, Failure Modes, and Limits

A histogram can reveal a broad asymmetry or multiple modes, but with a small crossover sample its shape depends heavily on bin choices. A normal quantile plot usually preserves more information about tail departures and is therefore a useful primary display. Formal tests, such as the Shapiro–Wilk test, provide a test statistic and p-value for a specified normality null under the test’s assumptions. A large p-value does not establish normality; it can reflect low power, particularly in small studies. A small p-value indicates incompatibility with the null under that test, but does not identify the cause or quantify its effect on the treatment estimate.

Look for patterns rather than a single numerical verdict. Curvature across most of the plot can suggest asymmetry or a distribution with a different shape. A departure concentrated at one end suggests a tail issue. Separate clusters or systematic shifts by period or treatment may point to an unmodeled design feature rather than a single common error distribution. A fan-shaped pattern primarily raises a variance question, and should be evaluated using variance diagnostics rather than labeled a normality failure.

Residual plots also have limits. Fitting the model constrains residuals and changes their variances; a small sample may not reveal important departures. A diagnostic pattern can be caused by model misspecification, a data-processing issue, or an unusual subject. It is not a license to remove observations. Check data and model construction, assess whether the pattern is systematic, and report material concerns. Normality diagnostics alone do not determine which subject is influential or justify automatic deletion.