Distinguishing Family Mismatch From Unmodeled Structure

Why the Source of a Residual Pattern Matters

A fitted model can show more residual variation than expected, or residuals can appear skewed, even when the chosen response family is not the main problem. If the mean model omits curvature, a group effect, or another systematic component, observations with similar covariate values may have residuals of the same sign. Pooling those residuals can make the response look more variable or differently shaped than it is after the mean structure is accounted for.

The useful question is therefore not simply whether residuals look unusual. It is whether their behavior changes systematically with covariates, groups, or fitted means. A persistent conditional mean pattern points first to the systematic part of the model. If the mean is adequately represented but the residual spread or shape still differs from what the candidate family predicts, family mismatch becomes a more plausible explanation. These clues guide model checking; they do not prove a unique cause.

Conditional Mean and Residual Definitions

Let \(Y\) be a continuous response and \(X\) the covariates. Write the true conditional mean and variance as \(\mu_0(X)=E[Y\mid X]\) and \(\operatorname{Var}(Y\mid X)\). A fitted GLM specifies a mean function \(m(X)\), typically through a linear predictor and link, and a candidate family supplies a variance function. Write the fitted variance benchmark as \(v(X)>0\). For an observation evaluated at these fixed functions, define the raw residual \(e=Y-m(X)\) and the standardized residual \(R=e/\sqrt{v(X)}\).

In practice, \(m\) and \(v\) are estimated, so residuals use fitted values such as \(\hat{\mu}_i\) and an estimated dispersion. The identities below describe fixed functions, or predictions evaluated on observations independent of the data used to fit them. They clarify the population patterns to look for. In-sample fitted residuals also reflect parameter estimation and leverage, so their appearance is not a direct, unaltered view of the population residuals.

What Residual Patterns Say About the Mean

Proof. Conditional second-moment finiteness implies conditional first-moment finiteness almost surely. Since \(m(X)\) and \(v(X)\) are functions of \(X\), they are fixed when conditioning on \(X\). Therefore, $$ E[R\mid X] =\frac{E[Y-m(X)\mid X]}{\sqrt{v(X)}} =\frac{\mu_0(X)-m(X)}{\sqrt{v(X)}}. $$ For the second identity, write $$ Y-m(X)=\{Y-\mu_0(X)\}+\{\mu_0(X)-m(X)\}. $$ On squaring and taking conditional expectations, the cross term is zero because \(E[Y-\mu_0(X)\mid X]=0\). The remaining terms give $$ E[(Y-m(X))^2\mid X] =\operatorname{Var}(Y\mid X)+\{\mu_0(X)-m(X)\}^2. $$ Dividing by \(v(X)\) proves the stated expression for \(E[R^2\mid X]\). \(\square\)

This result explains a central diagnostic distinction. A systematic trend in residual averages across a covariate, a group, or fitted means is evidence to investigate the mean structure. A correct mean model has \(E[R\mid X]=0\), even if the variance function is wrong. Conversely, an omitted mean component adds a nonnegative squared discrepancy to the residual second moment, so residuals may look excessively variable even when the conditional response family is otherwise appropriate.

Group Structure Can Masquerade as Extra Variation

An omitted group effect creates a related pattern. Residuals can average near zero overall while being predominantly positive in one group and negative in another. Pooling the groups conceals this conditional mean pattern and increases the overall residual spread. The variance decomposition makes the mechanism explicit.

Derivation. Let \(\mu_G=E[Y\mid G]\). Add and subtract \(\mu_G\) inside the squared deviation: $$ Y-E[Y]=(Y-\mu_G)+(\mu_G-E[Y]). $$ The cross term has expectation zero, since \(E[Y-\mu_G\mid G]=0\). Taking expectations of the two squared terms gives \(E[\operatorname{Var}(Y\mid G)]\) and \(\operatorname{Var}(E[Y\mid G])\), respectively. This proves the decomposition. \(\square\)

When the Mean Looks Adequate, Examine Conditional Shape

After checking for residual trends across covariates and groups, examine spread and shape within reasonably comparable portions of the data. A variance pattern that changes with fitted means in the way a candidate family does not predict can indicate a variance-family mismatch. Persistent asymmetry or tail behavior within covariate and fitted-mean ranges can likewise motivate reconsidering the family. A pooled residual histogram is less informative when observations come from different mean levels: mixing conditional distributions can itself create asymmetry or heavy tails.

A Practical Order for Interpreting Patterns

These steps are a way to organize evidence, not a mechanical selection rule. Sparse groups, influential observations, limited sample size, and uncertainty in estimated fitted means can obscure or imitate patterns. Replicated predictive checks and substantive knowledge remain useful complements, especially when several explanations fit the same residual evidence.

Common Mistake

Wrong reasoning: “The pooled standardized residuals have a large second moment, so the response family must allow more variation.” This skips the possibility that residual means differ across covariates or groups. As the examples show, squared residuals include a contribution from the squared mean error, and mixing groups adds between-group variation. Correction: first check conditional residual averages and whether spread changes systematically with covariates, groups, or fitted means. Then evaluate variance and shape relative to a family after the mean structure has been examined. Neither a large spread nor a skewed pooled plot identifies its cause by itself.