Residual Diagnostics for Genetic Mixed Models

Why Residual Diagnostics Matter

A genetic mixed model represents quantitative phenotypes using fixed effects, genetic or other random effects, and residual variation. A fitted model can produce plausible estimates while still missing a feature of the observed data: the mean may vary systematically with fitted values, the residual spread may change across the phenotype range, or the tails may be heavier than a Gaussian model implies. Residual diagnostics help reveal such mismatches.

In a mixed model, the word residual can refer to more than one quantity. A residual after fitting only the fixed effects retains the fitted random-effect contribution. A residual after subtracting predicted random effects does not. Neither is automatically an independent, equal-variance sample of errors. The distinction matters when interpreting plots, flagging unusual observations, or assessing Gaussian assumptions.

Model and Residual Definitions

Consider the Gaussian linear mixed model

where \(y\) is the vector of quantitative phenotypes, \(X\) is the fixed-effect design matrix, and \(\beta\) is the fixed-effect coefficient vector. The columns of \(Z\) associate observations with random effects \(b\), which may include polygenic effects with a covariance structure based on a genetic relationship matrix. The residual vector \(\varepsilon\) has covariance \(R\). The model assumes that \(b\) and \(\varepsilon\) are independent and that both are Gaussian with the stated means and covariance matrices. Consequently,

In practice, variance components and hence \(V\) are estimated. For definitions below, treat the fitted covariance parameters as given and let \(\hat{\beta}\) be the generalized least-squares estimate. Let \(\hat{b}\) be the best linear unbiased predictor (BLUP) of the random effects under the fitted model.

“Marginal” refers to the model that averages over the random effects; it does not mean that the residuals are independent. “Conditional” refers to predictions that include estimated random effects, not to a guarantee that each residual is the true individual-level error. In particular, \(\hat{b}\) is predicted from the same phenotype data, and prediction typically shrinks random-effect estimates toward zero.

Residual Covariance and Scaling

Even under a correctly specified model, residuals need not have a common variance or be mutually independent. For example, marginal residuals retain variation associated with the random effects. Moreover, estimating fixed effects imposes constraints on the residual vector and changes its covariance.

If the covariance parameters are treated as known, the GLS estimate is \(\hat{\beta}=(X'V^{-1}X)^{-1}X'V^{-1}y\). Define \(C=(X'V^{-1}X)^{-1}\). The marginal residual covariance is

This matrix is generally not \(V\): the fitted fixed effects have been estimated from the same observations. Its diagonal entries give the model-based marginal residual variances. A basic standardized marginal residual divides each residual by the square root of its corresponding diagonal entry. This puts the residual on a scale measured in approximate model-based standard deviations, but does not remove correlation between observations.

The conditional residuals also have a model-dependent covariance. With known covariance parameters and the GLS fit above, the BLUP satisfies \(\hat{b}=GZ'V^{-1}r_m\). Since \(V=ZGZ'+R\),

Thus, with fixed effects estimated, their covariance is \(RV^{-1}\operatorname{Var}(r_m)V^{-1}R\), not generally \(R\). This is one reason that dividing conditional residuals by a single residual standard deviation can be misleading. When covariance parameters are estimated, the displayed covariance formulas are conditional on their fitted values and do not account fully for their estimation uncertainty.

Residual-versus-Fitted Plots

A residual-versus-fitted plot places each residual on the vertical axis and the corresponding fitted value on the horizontal axis. Use marginal residuals against marginal fitted values, or conditional residuals against conditional fitted values, and label the choice. Points scattered around zero without a systematic curve are consistent with an adequately modeled mean. A curved trend can indicate that the fixed-effect mean structure is inadequate; a funnel-shaped spread can indicate that the variance changes across the fitted range.

In genetic data, plots can also be stratified or coloured by features relevant to the fitted model, such as a categorical covariate, batch, or family. A pattern confined to a group may be obscured in the overall plot. These displays are clues, not automatic tests: finite samples produce irregular patterns, and an apparent pattern should be considered in light of sample size, design, and the model assumptions.

Conditional residual plots require particular care. The conditional fitted values include predicted random effects estimated from the same outcomes. Those predictions can absorb some observed structure, so a visually quiet conditional plot does not by itself show that the random-effect covariance structure is appropriate. Comparing the marginal and conditional views can help describe where the model accounts for variation, but neither view alone proves that the model is correct.

Outliers and Distributional Diagnostics

An outlier check asks whether an observation is unusual relative to the fitted model, not whether it should automatically be deleted. Examine standardized residuals and consider their context, measurement quality, covariates, and related observations. A large residual can arise from a recording problem, an omitted feature, a genuine rare phenotype, or chance. Its size alone does not determine which explanation is correct.

A normal Q–Q plot compares ordered standardized residuals with theoretical quantiles from a standard normal distribution. If the points follow an approximately straight line, the marginal shape is broadly compatible with Gaussian behavior. Curvature in the tails can suggest heavier or lighter tails; an asymmetric pattern can suggest skewness. These are diagnostic descriptions, not proof of a particular alternative distribution.

Because residuals are correlated and affected by estimating fixed effects and covariance parameters, ordinary independent-sample reference arguments for a Q–Q plot or a formal normality test need not apply directly. When a more calibrated reference is needed, a parametric bootstrap can simulate outcomes under the fitted model, refit the model in each simulation, and compare the observed diagnostic with the resulting simulated diagnostics. This comparison must reproduce the fitting and diagnostic steps; treating correlated residuals as independent can give an inappropriate reference.

A Practical Diagnostic Sequence

A disciplined check keeps the residual definition, scaling, and question explicit. The following sequence can organize the assessment without turning any single plot into a pass-or-fail rule.

Common Pitfall and Interpretation

A common mistake is to treat conditional residuals as the original independent errors \(\varepsilon\). The reasoning is that subtracting the fitted random effects leaves only residual error. The correction is that random effects are predicted, not observed: \(r_c\) includes prediction error and is computed using the same data being diagnosed. Its variability and dependence therefore reflect both the covariance model and the fitting procedure.

A second mistake is to interpret a clean residual plot as proof that every aspect of the model is correct. Diagnostics can reveal some departures, but limited sample size, dependence, and the choices made in fitting can conceal others. They support a reasoned assessment of Gaussian mixed-model fit; they do not establish that the model is uniquely correct or that an observed genetic association is causal.