What a Residual Plot Can—and Cannot—Show
A residual plot asks whether the model’s remaining errors show a pattern related to what it predicts or to a measured predictor. In a generalized linear mixed model, that question needs care: outcomes may be discrete, observations may share a cluster, and the plotted fitted mean may be conditional on estimated random effects or averaged over them. A line or band in a plot can be evidence worth investigating, but it is not by itself proof that a model assumption has failed.
For observation \(i\) in cluster \(j\), write its response as \(Y_{ij}\) and its fitted mean as \(\hat{\mu}_{ij}\). A residual \(r_{ij}\) compares the observed response with a model-based expectation. The diagnostic question here is whether those residuals change systematically as \(\hat{\mu}_{ij}\), or a covariate \(x_{ij}\), changes.
The last distinction is central. For a conditional GLMM, a conditional mean uses the estimated random effect for the observation’s cluster; a marginal mean averages over the random-effects distribution. These are different predictions and answer different diagnostic questions. Residuals constructed from one target should not be casually paired with fitted means from the other. The earlier material on conditional and marginal fitted values and on residual definitions establishes these distinctions; here the emphasis is on reading the plots.
Expected Patterns Under a GLMM
Suppose that, conditional on the random effects and covariates, \(Y_{ij}\) has mean \(\mu_{ij}\) and variance \(V(\mu_{ij})\), with any dispersion factor included where appropriate. A Pearson residual based on that conditional target is
For a Bernoulli response with success probability \(p_{ij}\), \(V(p_{ij})=p_{ij}(1-p_{ij})\). For a Poisson count with mean \(\mu_{ij}\), \(V(\mu_{ij})=\mu_{ij}\). This scaling can make residual magnitudes easier to compare across fitted means, but it does not make discrete residuals normally distributed or independent. For grouped binomial data with \(m_{ij}\) trials, the response is a count and its variance is \(m_{ij}p_{ij}(1-p_{ij})\).
If the conditional model and parameters were known, the response residual \(Y_{ij}-\mu_{ij}\) would have conditional mean zero. Dividing by its true standard deviation gives a Pearson residual with conditional mean zero and variance one:
This provides a useful benchmark, not a promise that a finite-sample scatterplot will look like an even cloud. The actual plot uses estimated fixed effects, estimated random effects, and estimated variance parameters. Residuals within a cluster may remain related, and the discreteness of the response can produce stripes, asymmetry, or unequal-looking scatter even when the model is adequate.
Worked Example: A Binary Residual Plot with Two Stripes
Suppose a correctly specified Bernoulli GLMM gives a conditional fitted probability \(\hat{p}=0.05\) for 100 observations. The residuals below are Pearson residuals, calculated with the Bernoulli variance.
For an observation with \(y=0\),
For an observation with \(y=1\),
The plot can therefore show many points near \(-0.2294\) and a few near \(4.359\), rather than a symmetric band around zero. A fitted probability of \(0.05\) predicts about five successes in 100 observations, so seeing a small number of large positive residuals is an expected consequence of the binary scale. The two stripes alone do not diagnose lack of fit. Their frequency and location should be considered against the model’s predictive distribution, with clustering handled appropriately.
Reading Plots Against Fitted Means
A residual-versus-fitted plot is useful for looking for broad changes across the range of model predictions. With a link function, a pattern may indicate that the mean structure or link does not capture the relationship adequately. A curved smooth, a sustained run of positive residuals, or a change in their average across fitted values is more concerning than the familiar stripes or changing vertical range of raw residuals for binary and count outcomes.
The vertical spread may change naturally with the fitted mean. For example, the Bernoulli variance \(p(1-p)\) changes with \(p\), while the Poisson variance increases with its mean. Pearson scaling accounts for the specified variance function, but it does not remove all discreteness, skewness, or dependence. Also check whether a few fitted-mean regions contain very few observations: a jagged smooth in a sparse region is weak evidence of a systematic pattern.
A fitted-mean plot can also hide problems. If most observations have similar fitted means, the plot has little range over which to reveal a trend. If fitted means are conditional on estimated cluster effects, apparent variation between clusters may partly reflect those estimated effects. When the diagnostic question concerns population-average predictions, use residuals and fitted means defined for that marginal target, rather than treating conditional and marginal quantities as interchangeable.
Worked Example: A Trend Across Grouped Binomial Predictions
Consider four covariate levels in a hypothetical clustered binary-outcome analysis. At each level, 100 conditionally independent Bernoulli observations are summarized. Assume the listed probabilities are the model’s conditional fitted probabilities after accounting for the relevant cluster effects. The table gives the success count \(y\) and its expected count \(100\hat{p}\).
| Covariate level | \(\hat{p}\) | Observed successes \(y\) | Expected successes \(100\hat{p}\) |
|---|---|---|---|
| 0 | 0.10 | 12 | 10 |
| 1 | 0.20 | 23 | 20 |
| 2 | 0.30 | 42 | 30 |
| 3 | 0.40 | 55 | 40 |
For each grouped count, the Pearson residual is the observed-minus-expected count divided by the binomial standard deviation. At level 2, for example,
The four residuals are \(2/\sqrt{9}\approx0.667\), \(3/\sqrt{16}=0.750\), \(12/\sqrt{21}\approx2.619\), and \(15/\sqrt{24}\approx3.062\). Their upward progression is a systematic-looking pattern, not merely the two horizontal stripes expected for individual Bernoulli residuals. It suggests that the model underpredicts success increasingly at higher covariate levels. Possible explanations include an inadequate functional form, an omitted interaction, or an incorrectly specified relationship between the predictor and outcome. These summaries do not establish which explanation is correct; the pattern should be checked with the underlying observations and other diagnostics. The grouped calculation also assumes the conditional independence and fitted probabilities stated above; dependence left within the groups would change the variability of the totals.
Reading Plots Against Covariates
Residuals against a predictor can reveal structure that is difficult to see against fitted means. Use this plot to ask whether the model’s errors change with a predictor’s value, not to infer causation. A smooth trend can suggest a missing nonlinear term; distinct bands can point to a missing category effect or interaction; a change that appears only in one range may suggest that the current functional form is inadequate there.
For a continuous covariate, a smooth curve can help summarize a large cloud, but it is exploratory. Its apparent shape depends on the smoother and on how densely observations cover the predictor. For a categorical covariate, compare residual distributions or averages across its levels, while recognizing that sparse levels produce unstable summaries. For a covariate that is constant within clusters, residuals within a cluster do not provide independent replication of its effect. The number and diversity of clusters matter for judging a pattern.
Worked Example: A Possible Nonlinear Count Relationship
Suppose a Poisson GLMM has conditional fitted means \(\hat{\mu}=2,3,4,5\) at four values of a continuous predictor \(x=0,1,2,3\). For a hypothetical sequence of observations \(y=2,4,7,9\), the Pearson residuals are calculated using \(V(\hat{\mu})=\hat{\mu}\).
The residuals become increasingly positive as \(x\) increases. In this small illustration, that is consistent with underprediction at larger values of the predictor and motivates checking the shape of the mean relationship. It does not prove that a nonlinear term is needed: four observations cannot separate a genuine trend from sampling variation, and the pattern should be assessed across the full data and cluster structure. A Poisson variance assumption and the stated conditional means are also part of this calculation; if the conditional distribution or random-effects structure is inadequate, that can affect the plot as well.
A Practical Reading Sequence
Check whether the residuals and fitted means are both conditional on estimated random effects or both defined marginally. Record the residual type and its scale.
Ask whether residuals stay roughly centered around zero across fitted values and predictor values, or whether there are sustained trends, curves, or group differences.
For binary and count outcomes, expect ties, stripes, skewness, and mean-dependent spread. Compare a suspected pattern with what the fitted discrete model can generate.
Notice sparse predictor regions, repeated measurements, and covariates that vary only between clusters. A collection of points is not automatically a collection of independent pieces of evidence.
Use a pattern to form a specific hypothesis about mean structure or distributional fit, then check it with an appropriate residual or simulation-based diagnostic. A plot alone does not identify the cause.
Alternatives, Failure Modes, and Limits
Residual-versus-fitted and residual-versus-covariate plots are flexible visual checks. They are especially useful for locating a problem: a trend against one predictor can suggest where to inspect the model formula, while a pattern across fitted means can point to a broader mismatch. They do not, on their own, provide a calibrated test with a universal rejection threshold.
Other diagnostic displays answer related but different questions. Randomized quantile and simulation-based residuals can make discrete outcomes easier to compare with a reference distribution, provided their construction matches the conditional or marginal predictive target. Predictive checks compare features of observed outcomes with replicated data from the fitted model. These approaches can help assess whether an apparent feature is plausible under the fitted model; they do not replace careful plots against the predictors most relevant to the scientific model.
- Calling every funnel a variance failure. The variance of a Bernoulli or Poisson response depends on its mean. A changing spread in raw residuals can therefore be expected. Check the residual scale and the model’s variance function before interpreting it as a defect.
- Calling every stripe a trend. Discrete outcomes produce repeated residual values. Look for systematic changes in their location or frequency across fitted values or covariates, rather than expecting a continuous cloud.
- Overreading a smoother. A wavy line through sparse data may be unstable. Inspect the number of observations and clusters across the predictor range and treat the smoother as a guide, not a formal test.
- Ignoring clustering. Residuals from the same cluster may be associated. A plot may reveal a pattern, but its visual strength should not be interpreted as if every point were independent evidence.
- Mixing conditional and marginal quantities. A residual and fitted value aimed at different targets can create misleading comparisons. Align their definitions before diagnosing a trend.
These plots have further limits: estimated residuals inherit uncertainty from the fitted model, sparse data can conceal meaningful departures, and several misspecifications can produce similar visual patterns. A lack of visible structure is not proof that the GLMM is correct. The defensible conclusion is narrower: given the chosen residuals and plotting scale, the display does or does not show a clear pattern that merits follow-up.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For a Bernoulli observation with \(y=0\) and fitted probability \(\hat{p}=0.20\), calculate the Pearson residual. What would the residual be if \(y=1\)?
- A residual-versus-fitted plot has a funnel shape for a Poisson GLMM, but its residuals are unscaled response residuals. Why is the funnel not sufficient evidence of misspecification?
- Residuals are mostly positive at high values of a continuous predictor and mostly negative at low values. Name one possible model issue and one reason the plot alone cannot confirm it.
- Why should conditional residuals not be casually plotted against marginal fitted means?
Solutions. (1) For \(y=0\), \(r^{(P)}=(0-0.20)/\sqrt{0.20(0.80)}=-0.20/0.40=-0.500\). For \(y=1\), \(r^{(P)}=(1-0.20)/0.40=2.000\). (2) The Poisson response variance equals its mean, so raw residual spread can change as fitted means change; a funnel may be expected. Assess the pattern using an appropriate scale and the fitted model’s predictive behavior. (3) A nonlinear mean relationship or an omitted interaction could cause the trend. Sampling variation, sparse coverage, or cluster dependence could also produce it, so further checks are needed. (4) Conditional residuals compare outcomes with predictions that include cluster-specific random effects, whereas marginal means average over those effects. Mixing targets can produce a plot whose apparent departures do not correspond to either intended diagnostic comparison.