Why Mixed-Model Residuals Need Careful Interpretation
A residual plot can reveal a misspecified biomarker trajectory, changing residual spread, or a distributional departure. In a linear mixed model, however, there is more than one useful residual because observations vary both across subjects and within subjects. A deviation from a population trajectory may reflect a subject’s random effect, not an unusual measurement error. The diagnostic question therefore matters: are we checking the population-level fit, the within-subject error model, or the combined distribution of observations?
The earlier material on marginal covariance decomposition and BLUPs supplies the model ingredients used here. This tutorial focuses on choosing and interpreting diagnostics, not on fitting the model or selecting its covariance structure.
The marginal residual compares the observed measurements with the population-level fitted mean. It includes subject-to-subject departures as well as within-subject deviations. The conditional residual compares observations with fitted means that include estimated subject effects, so it is more directly aimed at the level-one error pattern. Neither is simply the unobserved error \(\varepsilon_i\): both use estimated model quantities, and the conditional residual additionally uses a shrunken estimate of \(b_i\).
A Short Derivation: Why Conditional Residuals Are Shrunken
First suppose the parameters and fixed-effects mean are known, and set \(u_i=y_i-X_i\beta\). Under the model, \(u_i=Z_i b_i+\varepsilon_i\), with covariance \(V_i\). The Gaussian conditional expectation of the random effects is \(\hat b_i=GZ_i^\mathsf{T}V_i^{-1}u_i\). Thus the conditional residual is \(r_{ci}=u_i-Z_i\hat b_i\). Substituting the expression for \(\hat b_i\) gives
The final equality follows because \(V_i=Z_iGZ_i^\mathsf{T}+R_i\), so \(I-Z_iGZ_i^\mathsf{T}V_i^{-1}=R_iV_i^{-1}\). More directly, \(r_{ci}=E(\varepsilon_i\mid y_i)\): it is the conditional expectation of the level-one error, not the actual error itself. Its covariance across repeated samples, when the parameters are known, is
which is generally not \(R_i\). Estimating fixed effects and covariance parameters adds further complications. Consequently, raw conditional residuals can have reduced spread and dependence induced by fitting. A residual scale or reference distribution should account for the fitted model; dividing every raw residual by one common residual standard deviation is not automatically adequate.
Match the Plot to the Diagnostic Question
| Diagnostic question | Useful display | What to look for |
|---|---|---|
| Is the mean trajectory adequate? | Conditional residuals versus time or fitted values; smooth trend or binned means | Systematic curvature, a trend, or repeated runs above and below zero |
| Is residual spread stable? | Conditional residuals versus fitted values, time, or covariates | A funnel, changing band width, or distinct spread across ranges |
| Is a Gaussian approximation plausible? | Q–Q plot of appropriately scaled conditional residuals; marginal residual Q–Q plot for the joint marginal distribution | Systematic curvature, tail departures, or isolated extreme points |
| Are subjects’ observed levels described? | Marginal residuals by subject or over time | Subject-level offsets or structure not captured by the fixed mean |
For a Q–Q plot, order the residuals and compare them with theoretical quantiles from the proposed reference distribution. If the points follow a roughly straight line, that is compatible with the distributional assumption; bends in the middle or departures in the tails can indicate skewness, heavy tails, or other mismatch. A Q–Q plot is a diagnostic, not a formal proof of normality. In particular, the residuals are estimated and often dependent, so ordinary independent-sample intuition about the plot must be used cautiously.
The model’s marginal distribution is Gaussian if both random effects and errors are Gaussian. Marginal residuals can therefore be examined for overall marginal distributional departures, but they are correlated within subject. Conditional residuals are more targeted to the error distribution, though shrinkage and fitting also affect their distribution. If residuals are standardized, the standardization should reflect the relevant fitted covariance and leverage; a marginal residual \(e_{ij}\), for example, may be divided by \(\sqrt{\hat V_{i,jj}}\) for a basic marginal scale comparison. This diagonal scaling does not remove within-subject correlation.
Worked Example: Separating Population and Subject-Specific Residuals
Suppose a hypothetical biomarker model has fitted population trajectory \(\hat y=10+2t\), with estimated random intercepts \(+1\) for subject A and \(-1\) for subject B. At times \(t=0,1,2\), the observed values are A: \(11.2,13.0,14.8\), and B: \(9.1,11.2,13.3\). The conditional fitted trajectories add the estimated subject effects to the population fit.
For subject A, the population fitted values are \(10,12,14\). Therefore its marginal residuals are \(11.2-10=1.2\), \(13.0-12=1.0\), and \(14.8-14=0.8\). The conditional fitted values are \(11,13,15\), giving conditional residuals \(11.2-11=0.2\), \(13.0-13=0.0\), and \(14.8-15=-0.2\).
For subject B, the population fitted values are again \(10,12,14\), so the marginal residuals are \(9.1-10=-0.9\), \(11.2-12=-0.8\), and \(13.3-14=-0.7\). Its conditional fitted values are \(9,11,13\), giving conditional residuals \(0.1,0.2,0.3\).
The marginal residuals show that A is generally above and B generally below the population trajectory; those offsets are represented by the random intercepts. Conditional residuals are much closer to zero because the subject effects have been included. This small example illustrates why a large marginal residual is not necessarily evidence of an unusually large within-subject error.
Worked Diagnostic Decisions
Worked Example: Detecting a Mean-Structure Pattern
A hypothetical fitted biomarker model is assessed using conditional residuals in four equally sized time bands. The average residuals, in biomarker units, are \(-0.50,-0.10,0.40,1.00\), from earliest to latest. The pattern is not a random fluctuation around zero: the averages rise by \(0.40,0.50,\) and \(0.60\) units between successive bands.
The positive progression suggests that the fitted mean may be too low at later times relative to earlier times. A reasonable next check is a conditional-residual-versus-time plot, including a smooth trend, and a comparison with a scientifically plausible alternative mean shape. The pattern alone does not prove that a particular polynomial or transformation is correct; the model should be reconsidered and refitted, then diagnostics repeated. This interpretation assumes the residuals are calculated on the same biomarker scale and that time bands contain observations over comparable ranges of other modeled covariates.
Worked Example: Reading a Residual Q–Q Plot
Suppose a hypothetical fitted model yields six standardized conditional residuals, ordered as \(-1.40,-0.80,-0.20,0.10,0.50,1.80\). For a normal Q–Q plot, approximate standard-normal quantiles at plotting positions \((j-0.5)/6\) are \(-1.383,-0.674,-0.210,0.210,0.674,1.383\), for \(j=1,\ldots,6\).
The first five ordered residuals are reasonably close to the corresponding reference quantiles: their differences are \(-0.017,-0.126,0.010,-0.110,-0.174\). The largest residual is \(1.80\), which exceeds its reference quantile \(1.383\) by \(0.417\). This suggests checking the upper tail and the observation’s context, but six points are far too few to establish a tail-distribution failure. The conclusion is therefore “inspect further,” not “normality is rejected.” This assessment also presumes the residuals were scaled appropriately; a Q–Q plot of raw residuals with a non-unit expected scale would not support the same comparison.
Practical Workflow, Alternatives, and Limitations
Use conditional residuals to look for within-subject error patterns after accounting for fitted subject effects. Use marginal residuals to see departures from the population mean, remembering that random-effect variation contributes to them.
Inspect residuals against fitted values, time, and important covariates. Look for systematic structure in the center and changing spread, and consider subject grouping so repeated measurements are not visually treated as unrelated.
Use a Q–Q plot of suitably scaled residuals. If Gaussian assumptions are central to the analysis, compare the observed diagnostic pattern with reference behavior that accounts for the model’s estimation and dependence where feasible.
If a pattern appears, consider whether the mean, covariance, or distributional specification is implicated. Refit a defensible model and repeat the diagnostics rather than interpreting a single plot as a definitive test.
These displays complement, rather than replace, model-based checks. A plot of conditional residuals emphasizes the within-subject component, whereas a marginal residual plot includes random-effect departures. A Q–Q plot assesses shape; it does not reveal whether residual spread changes with time. Conversely, a residual-versus-time plot can identify a trend but does not establish Gaussian tails. No single plot checks every assumption.
There are also limits to visual diagnosis. With few subjects, apparent tail behavior is unstable; with many observations, small deviations can look conspicuous. Correlation means repeated residuals do not provide the same independent information as the same number of unrelated observations. Conditional residuals can be especially misleading if interpreted as observed errors, and marginal residuals can appear structured simply because subject-level variation is present. For difficult cases, simulation-based residual diagnostics under the fitted model can provide a reference that reflects estimation and dependence more fully.
Common Mistake
Wrong reasoning: “The conditional residuals are close to zero, so the model’s error variance must be small and its assumptions are satisfied.” Conditional residuals are pulled toward zero because subject effects are estimated from the same observations, and fitting can also reduce residuals. Their small magnitude alone is not evidence of good fit.
Correction: Check the pattern, scale, and distribution of residuals using diagnostics appropriate to the question. Compare conditional and marginal displays, examine time and fitted-value structure, and interpret Q–Q departures in light of sample size, correlation, and estimation. A visually unstructured plot supports the model but cannot prove it is correct.
Check Your Understanding
Answer each question, then compare with the solutions.
- In one sentence, distinguish a marginal residual from a conditional residual.
- A conditional-residual plot shows a widening band as fitted biomarker values increase. What assumption or feature should be investigated?
- Why is it incorrect to treat a conditional residual as the observed level-one error?
- A marginal residual Q–Q plot is curved. Does that alone establish that the within-subject errors are non-Gaussian?
Solutions. (1) A marginal residual subtracts the fixed-effects fitted mean, while a conditional residual also subtracts the estimated subject-specific random-effects contribution. (2) Investigate whether the residual variance changes with fitted value, and consider whether the variance specification is adequate. (3) It uses an estimated, shrunken random effect; with known parameters it equals the conditional expectation of the level-one error given the observations, not the error itself. (4) No. The marginal distribution also reflects random effects, residual dependence, and parameter estimation; inspect conditional diagnostics and the model’s assumptions before drawing that conclusion.