Omitted Covariates and Apparent Overdispersion

Why an Omitted Predictor Can Leave a Residual Signature

A fitted generalized linear model can show extra residual variation even when the response follows the assumed distribution after conditioning on the predictors that actually determine its mean. One explanation is that a relevant predictor was left out. If that predictor varies among observations with the same included covariates, their response means differ even though the fitted model assigns them a common mean. If its distribution also changes across included-covariate groups, the omission can distort the fitted mean pattern as well as increase residual variation.

The key comparison is conditional. Suppose \(X\) denotes the included covariates and \(Z\) an omitted predictor. The reduced model is assessed against the response distribution given \(X\), while the more detailed response mechanism depends on both \(X\) and \(Z\). The calculations below show how to compare these two levels of conditioning. They do not, by themselves, establish that omission is the cause of an observed diagnostic signal.

The Conditional-Variance Calculation

Assume \(Y\) has a finite second moment and that, given \(X\) and \(Z\), its mean and variance are \(m(X,Z)=E[Y\mid X,Z]\) and \(v(X,Z)=\operatorname{Var}(Y\mid X,Z)\). The reduced conditional mean is \(\bar m(X)=E[m(X,Z)\mid X]\). The variance given \(X\) separates into variation remaining within \(X,Z\) strata and variation in the stratum means.

For completeness, the cross-product vanishes because \(m(X,Z)-\bar m(X)\) is determined by \(X,Z\), and \(E[Y-m(X,Z)\mid X,Z]=0\). Thus its conditional expectation given \(X\) is \(E[(m(X,Z)-\bar m(X))E[Y-m(X,Z)\mid X,Z]\mid X]=0\). The two remaining terms are exactly the two terms in the theorem. \(\square\)

The theorem does not say that every reduced GLM must show overdispersion. It identifies a potential contribution: variation of the omitted-predictor-specific means within \(X\). Whether the reduced model’s variance benchmark is exceeded depends on the response family and on how its variance is evaluated at the reduced mean.

Poisson and Binomial Consequences

For a Poisson response, suppose \(Y\mid X,Z\) is Poisson with mean \(\mu(X,Z)\). Its conditional variance is also \(\mu(X,Z)\). Applying the theorem gives \(E[Y\mid X]=\bar\mu(X)=E[\mu(X,Z)\mid X]\) and \(\operatorname{Var}(Y\mid X)=\bar\mu(X)+\operatorname{Var}(\mu(X,Z)\mid X)\). The first term is the Poisson variance at the reduced mean; the second is the excess caused by variation in stratum means.

For a binomial response with a fixed number \(n\) of trials, suppose \(Y\mid X,Z\) is Binomial\((n,p(X,Z))\). Put \(\bar p(X)=E[p(X,Z)\mid X]\). The mean is \(n\bar p(X)\), and the variance decomposition yields \[ \operatorname{Var}(Y\mid X) =nE[p(1-p)\mid X]+n^2\operatorname{Var}(p\mid X). \] Because \(E[p(1-p)\mid X]=\bar p(1-\bar p)-\operatorname{Var}(p\mid X)\), this becomes \[ \operatorname{Var}(Y\mid X) =n\bar p(1-\bar p)+n(n-1)\operatorname{Var}(p\mid X). \] The added term is nonnegative; for \(n=1\), it is zero. Thus this particular binomial comparison yields an excess over the Binomial\((n,\bar p)\) variance when \(n>1\) and the success probability varies with \(Z\) within \(X\) strata.

Worked Examples

Using Stratified Residual Checks

When a plausible predictor is available but omitted from the fitted model, inspect residuals across its levels while controlling, as far as the data allow, for the included predictors. Otherwise, a residual pattern across \(Z\) might simply reflect a different distribution of \(X\) among the \(Z\) groups. Useful displays include residuals against \(Z\) with separate symbols or panels for important \(X\) groups, and tables of residual summaries within joint \(X,Z\) strata.

Look for systematic differences in residual location, spread, or both. Location patterns suggest that the reduced mean does not capture the response mean across the strata. A spread pattern may be consistent with the conditional-mean variation derived above. Check that fitted means are being compared with observations on an appropriate scale, and remember that a residual’s realized value includes sampling noise.

This is a diagnostic argument, not a causal conclusion. Residual patterns can also reflect an incorrectly specified mean function, influential observations, or other departures from the working model. If \(Z\) is unmeasured, a residual plot alone cannot identify it as the missing cause. If \(Z\) is measured, fitting or evaluating an expanded model can help assess whether its inclusion is substantively defensible and improves the conditional description.

Common Pitfall

Wrong reasoning: “The reduced model’s average residual within each \(X\) group is near zero, so no omitted predictor can be responsible for extra variation.” This confuses a group average with the pattern within the group. In the Poisson example, positive and negative conditional residual means cancel within each \(X\) group, even though the variance exceeds the Poisson benchmark.

Correction: A zero average residual does not imply that every subgroup has the same response mean. Examine residuals within meaningful \(Z\) strata, preferably also within levels or ranges of \(X\). Conversely, a visible subgroup pattern is evidence to investigate, not proof that the omitted predictor is the sole source of extra variation.