Residual Patterns That Separate Link and Family Problems

Why the Shape of a Residual Pattern Matters

A continuous-response GLM specifies both a conditional mean and a conditional variance. Its mean is connected to the linear predictor through a link function; its variance is described by a variance function and dispersion parameter. A poor fit can therefore arise because the fitted mean curve is wrong, because the assumed variability around that curve is wrong, or because both are wrong.

Residual plots help separate these possibilities, but they do not identify them automatically. The key distinction is between a systematic signed pattern, which is evidence that the conditional mean has not been captured, and a pattern in residual magnitude, which is evidence about the variance or response distribution. These are diagnostic clues, not proofs that a particular link or family is correct.

Definition: For observation \(i\), let \(Y_i\) be the response, \(\mu_i=E[Y_i\mid x_i]\) its conditional mean, and \(\hat{\mu}_i\) the fitted mean. The raw residual is \(e_i=Y_i-\hat{\mu}_i\). If the fitted model specifies \(\operatorname{Var}(Y_i\mid x_i)=\phi V(\mu_i)\), its Pearson residual is \(r_{Pi}=(Y_i-\hat{\mu}_i)/\sqrt{\hat{\phi}V(\hat{\mu}_i)}\), where \(\hat{\phi}\) estimates the dispersion. A residual plot displays residuals against fitted values, covariates, or another quantity relevant to the model.

For examining whether the mean is systematically high or low, use a signed residual such as the raw or Pearson residual. The sign indicates direction: a positive residual means the observed response exceeds the fitted mean, and a negative residual means it falls below it. To examine spread, look at residual magnitude, such as \(|r_{Pi}|\), or squared Pearson residuals. A signed residual plot and a spread plot answer different questions.

What Signed Residuals Say About the Mean

Suppose the true conditional mean at covariates \(x\) is \(m(x)=E[Y\mid X=x]\), while a model supplies a candidate mean \(\mu_0(x)\). The raw residual from that candidate mean is \(Y-\mu_0(X)\). Its conditional expectation is exactly the difference between the true and candidate means.

Result: If the conditional expectation exists, then \(E[Y-\mu_0(X)\mid X=x]=m(x)-\mu_0(x)\). Consequently, a persistent nonzero conditional signed-residual pattern indicates that the candidate mean function is not the true conditional mean. A variance misspecification alone does not change this conditional expectation.

This result follows directly from linearity of conditional expectation. It explains the central diagnostic distinction: under a correct mean, residuals should not systematically sit above or below zero as a function of the covariates, even if their variance changes. The result concerns population conditional expectations. In a finite sample, residual patterns fluctuate, and fitted residuals are affected by parameter estimation.

Check the sign pattern.
Look for sustained curvature, runs of positive or negative residuals, or a trend across a covariate or fitted-value range. Such structure suggests the mean function needs attention.

Check the magnitude pattern.
Look for residuals that become systematically more or less dispersed as fitted values or covariates change. This points toward the variance specification or response family.

Reassess the whole model.
Mean and variance problems can occur together. A spread pattern is not a reason to ignore a signed trend, and a mean correction does not guarantee the variance is appropriate.

What Residual Magnitudes Say About the Variance

The Pearson residual scales the raw residual by the standard deviation implied by the fitted variance function. If the mean is correct but the assumed variance is not, the signed residual can still average to zero while its squared magnitude changes systematically. The following calculation makes that distinction precise.

Result: Suppose \(E[Y\mid X=x]=\mu(x)\) and \(\operatorname{Var}(Y\mid X=x)=\sigma^2(x)\). For a specified positive variance scale \(a(x)\), define \(R=(Y-\mu(X))/\sqrt{a(X)}\). Then \(E[R\mid X=x]=0\) and \(E[R^2\mid X=x]=\sigma^2(x)/a(x)\). Thus a correct mean does not ensure that standardized residuals have the intended spread: that also requires the variance scale \(a(x)\) to match the true conditional variance.

For a GLM using \(a(x)=\phi V(\mu(x))\), a correct variance specification gives \(E[R^2\mid X=x]=1\). If the true variance is larger than the assumed variance in some region, squared Pearson residuals will tend to be larger there. Estimating the mean and dispersion changes the exact behavior of fitted residuals, so this is a population diagnostic principle, not an exact finite-sample identity for every fitted residual.

Worked Example: Curvature in the Mean, Not Just Changing Spread

Suppose a positive continuous response has true conditional mean \(m(x)=1+x^2\) at \(x=0,1,2\). A candidate log-link mean curve is \(\mu_0(x)=\exp(a+bx)\), with \(a=0\) and \(b=\log 5/2\). This curve matches the true mean at the two endpoints: \(\mu_0(0)=1\) and \(\mu_0(2)=5\).

At the middle value, the candidate mean is \(\mu_0(1)=\exp(\log 5/2)=\sqrt{5}\approx2.236\), whereas the true mean is \(m(1)=2\). Therefore the conditional expected raw residuals are \(m(0)-\mu_0(0)=0\), \(m(1)-\mu_0(1)\approx-0.236\), and \(m(2)-\mu_0(2)=0\).

The negative residual expectation in the middle reflects mean-curve mismatch: the candidate curve lies above the true mean there. The example does not claim that these endpoint-matched parameters would be maximum-likelihood estimates. It isolates the diagnostic logic: a systematic signed departure across \(x\) is about the conditional mean, and could motivate a different link or a richer predictor.

Worked Example: Correct Mean With a Changing Variance

Let \(X\) take values \(0,1,2\), and suppose the true mean is \(\mu(x)=2+2x\). Consider a Gamma response with dispersion \(\phi=0.25\) and variance function \(V(\mu)=\mu^2\). Its conditional variance is \(\phi\mu(x)^2\).

At the three covariate values, the means are \(2,4,6\), and the variances are \(0.25(2^2)=1\), \(0.25(4^2)=4\), and \(0.25(6^2)=9\). The raw residual \(Y-\mu(X)\) has conditional expectation zero at each value, but its conditional standard deviations are \(1,2,3\).

If one instead used a constant-variance model with variance one and examined residuals scaled by that assumed standard deviation, the conditional second moments of those scaled residuals would be \(1,4,9\). The residuals would fan out as the mean increased, although there is no signed mean trend in the population. That pattern points to a variance mismatch; it does not, by itself, establish that the Gamma family is the only suitable response distribution.

Reading Residuals Against Fitted Values and Covariates

A plot against fitted values is useful because the conditional variance in many GLMs depends on the mean. If the residual cloud has a systematic curve above and below zero, the fitted mean relationship may be inadequate. If the cloud remains centered near zero but its vertical spread changes, the variance function or family may be inadequate. A plot against an individual covariate can reveal structure that is hidden when the horizontal axis is the fitted mean.

Neither plot is conclusive on its own. A plot against a single covariate averages over the other covariates. A mean error that differs across combinations of predictors can cancel in that average, leaving little visible trend. Conversely, an apparent trend in a small sample may be random. Use the scientific role of the covariate, the fitted mean structure, and more than one diagnostic view when interpreting a pattern.

Worked Example: Separate the Signed Trend From the Spread

Suppose that at a particular covariate value \(x=1\), the fitted mean is \(4\). Four observed responses are \(2,3,5,6\), giving raw residuals \(-2,-1,1,2\). Their signed average is \((-2-1+1+2)/4=0\); their mean squared residual is \((4+1+1+4)/4=2.5\).

Now suppose that at \(x=2\), the fitted mean is \(6\), and responses are \(2,4,8,10\). The residuals are \(-4,-2,2,4\). Their signed average is \((-4-2+2+4)/4=0\), while their mean squared residual is \((16+4+4+16)/4=10\).

In this small illustrative sample, both groups are balanced around their fitted means, but the second group has much greater residual magnitude. The pattern motivates checking whether the variance changes with \(x\). It does not show that the mean is correct in the population: four observations per group cannot establish that, and a different sample could produce a signed imbalance by chance.

Common Mistakes and Limits of the Diagnosis

Treating any fan shape as a mean problem. Increasing residual magnitude with fitted values can occur under a correct mean when the variance grows with the mean. Check signed residuals separately from their magnitudes.
Treating a signed trend as a family-only problem. If the residual expectation changes systematically with a covariate, the conditional mean is not captured. A different variance function alone does not remove that population mean discrepancy.
Claiming the plot names the exact cause. A signed pattern may arise from an unsuitable link, an omitted nonlinear term, an omitted interaction, or another mean-structure error. A spread pattern may reflect an unsuitable variance function, distributional family, or unmodeled structure. The plot signals where to investigate; it does not uniquely select a remedy.
Interpreting fitted residuals as independent observations. Fitted means are estimated from the same data, so residuals can be constrained or correlated. Leverage and parameter estimation can affect visible patterns, especially in small samples.
Assuming a flat plot proves adequacy. A residual plot can miss patterns, especially when it averages over covariates or has limited data. Absence of an obvious trend is not proof of a correct mean, variance, or distribution.

For signed mean checks, raw residuals have a direct interpretation; Pearson residuals also express departures in units of the model-implied standard deviation. Deviance residuals are useful as a model-based diagnostic, but their nonlinear construction means their signed conditional expectation is not generally the simple difference between true and fitted means. Do not apply the raw-residual expectation identity to every residual definition.

Key takeaway: A systematic signed residual trend is evidence against the fitted conditional mean, including its link and predictor structure. Changing residual magnitude with no signed trend instead raises a variance or family question. Both patterns can coexist, and neither plot uniquely identifies the correction.

Check Your Understanding

Use the distinction between conditional mean and conditional variance to answer the following.

If raw residuals are systematically positive at low fitted values and negative at high fitted values, which part of the model should be investigated first, and why?
Can a model have zero conditional mean residuals at every covariate value and still show a fan-shaped residual plot? Explain.
If the true conditional variance is four times the variance used to scale a Pearson-type residual, what is the conditional second moment of that scaled residual, assuming the mean is correct?
Why can a residual plot against one covariate fail to reveal a mean error involving an interaction between two covariates?
What does a residual pattern suggest, and why does it generally not prove which particular link or response family should replace the fitted one?