Mean-Structure Misspecification and Apparent Overdispersion

Why the Mean Structure Matters

A dispersion diagnostic compares residual variation with the variation expected under a fitted generalized linear model. That comparison depends on both the response distribution and the fitted means. If the mean structure is wrong, residuals can show a systematic pattern: observations above the fitted mean in some parts of the predictor space and below it in others. This mismatch can enlarge Pearson or deviance statistics, even when the response variation around its true mean is exactly what the chosen family predicts.

Three common forms of mean-structure error are using the wrong link, fitting a linear covariate effect when the relationship is nonlinear, and omitting an interaction. Each can leave structure in residuals against fitted values, predictors, or scientifically relevant groups. These patterns are a reason to investigate the mean model, not automatic evidence that the response has extra variation.

Definition: Mean-structure misspecification occurs when the fitted model’s mean function cannot represent the true conditional mean over the predictor values of interest. For example, the fitted model may use the wrong link, omit curvature, or exclude an interaction needed to describe how predictors jointly affect the mean.

The distinction between the mean and the variance is central. A Poisson model specifies both a conditional mean and a conditional variance equal to that mean. A binomial model specifies a conditional mean and the corresponding binomial variance. If the mean is fitted incorrectly, residuals are centered around the wrong values. The next identity quantifies what that does to a Pearson statistic.

How Mean Error Enters the Pearson Statistic

Let $Y_i$ have true mean $m_i=E[Y_i]$ and true variance $v_i=\operatorname{Var}(Y_i)$. Suppose a fitted model supplies mean $\mu_i$ and uses variance benchmark $V(\mu_i)$ in its Pearson contribution. For this calculation, treat $\mu_i$ as fixed; this describes the contribution under a specified fitted mean, not the exact finite-sample distribution after estimating model parameters.

Result: Provided $V(\mu_i)>0$ and $Y_i$ has a finite second moment, $$ E\left[\frac{(Y_i-\mu_i)^2}{V(\mu_i)}\right] = \frac{v_i+(m_i-\mu_i)^2}{V(\mu_i)}. $$ The first term in the numerator is the true response variance. The second is squared mean error.

To prove the identity, write $Y_i-\mu_i=(Y_i-m_i)+(m_i-\mu_i)$, square, and take expectations. The cross-product has expectation zero because $E[Y_i-m_i]=0$:

\begin{aligned} E[(Y_i-\mu_i)^2] &=E[(Y_i-m_i)^2] +2(m_i-\mu_i)E[Y_i-m_i] +(m_i-\mu_i)^2\\ &=v_i+(m_i-\mu_i)^2. \end{aligned}

Dividing by the fixed positive benchmark $V(\mu_i)$ proves the result. In particular, the expected signed residual is $E[Y_i-\mu_i]=m_i-\mu_i$. A persistent sign across a range of predictor values is therefore evidence that the fitted means are systematically displaced there.

For a Poisson response, the fitted benchmark is $V(\mu_i)=\mu_i$. If the true response is also Poisson with mean $m_i$, the expected contribution is

\frac{m_i+(m_i-\mu_i)^2}{\mu_i}.

For a grouped binomial count $Y_i\sim\operatorname{Bin}(n_i,p_i)$, with fitted probability $q_i$, the contribution uses fitted variance $n_iq_i(1-q_i)$. Its expectation is

\frac{n_ip_i(1-p_i)+n_i^2(p_i-q_i)^2} {n_iq_i(1-q_i)}.

The squared mean-error term can be especially influential for a grouped count: a probability error of $p_i-q_i$ produces a discrepancy of $n_i(p_i-q_i)$ in the expected count. Notice also that the expectation depends on the true variance as well as the mean error. Mean misspecification does not guarantee that every individual Pearson contribution exceeds one; the balance depends on the true and fitted variances.

Worked Examples of Structured Mean Error

Worked Example: An Incorrect Link for a Binary Response

Suppose the true success probability follows a logistic curve, $p(x)=1/(1+\exp(-x))$, but a fitted probit model gives $q(x)=\Phi(x)$, where $\Phi$ is the standard normal cumulative distribution function. Consider three predictor values, $x=-1,0,1$, with 100 independent Bernoulli observations at each value. For a transparent calculation, treat the fitted probabilities as fixed at the stated probit values.

At $x=1$, $p\approx0.7311$ and $q\approx0.8413$. At $x=-1$, symmetry gives $p\approx0.2689$ and $q\approx0.1587$. At $x=0$, both probabilities equal $0.5$. The fitted probabilities are too extreme at the endpoints: residuals tend to be positive at $x=-1$ and negative at $x=1$, since $E[Y-q]=p-q$.

At either endpoint, the expected Pearson contribution for one Bernoulli observation is

\frac{p(1-p)+(p-q)^2}{q(1-q)} \approx \frac{0.1966+(0.1102)^2}{0.8413(0.1587)} \approx \frac{0.2087}{0.1335} \approx1.564.

At the center, the contribution is $0.25/0.25=1$. Thus the expected Pearson sum across one observation at each of the three predictor values is about $2(1.564)+1=4.128$, rather than 3. With 100 observations at each value, the corresponding expected sum is about $412.8$. This calculation illustrates how a wrong link can leave a systematic pattern and increase Pearson residual variation. It is not an exact finite-sample expectation for a statistic using estimated coefficients; the fitted probabilities were held fixed for the calculation.

Worked Example: A Missed Nonlinear Effect in a Poisson Model

Let $x=-1,0,1$, with 100 independent Poisson observations at each value. Suppose the true means are $4,2,4$, respectively, so the mean has a U-shaped pattern. Fit a log-linear mean with intercept and linear $x$, but no curvature. In the population fit for this balanced design, symmetry makes the slope zero, and the fitted mean at all three predictor values is the arithmetic average $(4+2+4)/3=10/3$.

For either endpoint, $m=4$ and $\mu=10/3$. Its expected Pearson contribution is

\frac{4+(4-10/3)^2}{10/3} = \frac{4+4/9}{10/3} = \frac{40}{9}\cdot\frac{3}{10} = \frac{4}{3}.

At the center, $m=2$, so the expected contribution is

\frac{2+(2-10/3)^2}{10/3} = \frac{2+16/9}{10/3} = \frac{34}{9}\cdot\frac{3}{10} = \frac{17}{15}.

The total expected Pearson sum is therefore $100(4/3+17/15+4/3)=100(57/15)=380$. There are $300$ observations and two fitted coefficients, giving nominal residual degrees of freedom $298$. The ratio of the expected sum to these degrees of freedom is $380/298\approx1.275$. This is a population-mean calculation, not an assertion that the expectation of the random fitted ratio is exactly $1.275$. In a residual plot, the same misspecification appears as positive residuals at both endpoints and negative residuals near zero.

Worked Example: An Omitted Interaction in Grouped Binomial Data

Consider a balanced $2\times2$ design with 20 Bernoulli trials in each cell. Suppose the true success probabilities are $0.2$ in the two diagonal cells and $0.8$ in the other two cells. This alternating pattern requires an interaction. An additive logistic model with an intercept and two main effects cannot represent it; by symmetry its population fitted probability is $q=0.5$ in every cell.

For a cell with $p=0.2$ and $q=0.5$, the observed count has true variance $20(0.2)(0.8)=3.2$, and its mean differs from the fitted count $20(0.5)=10$ by $20(0.2-0.5)=-6$. Its expected Pearson contribution is

\frac{3.2+(-6)^2}{20(0.5)(0.5)} = \frac{3.2+36}{5} = 7.84.

A cell with $p=0.8$ has the same contribution: its variance is also $3.2$, and its fitted-count discrepancy is $+6$. The expected total across the four cells is $4(7.84)=31.36$. The additive model has three coefficients and there are four grouped observations, so the nominal residual degrees of freedom are $4-3=1$. The large expected sum relative to that benchmark reflects the missing interaction and the coherent cell-level discrepancies. A plot or table of residuals by both factors would reveal the alternating pattern that a single overall dispersion ratio cannot explain.

Deviance and Residual Patterns

The residual deviance is another measure of departure from fitted means. For Poisson data, the contribution of an observed count $y$ and fitted mean $\mu>0$ is $2[y\log(y/\mu)-(y-\mu)]$, with the usual convention that $y\log(y/\mu)=0$ when $y=0$. It is not identical to the Pearson contribution. When $y$ is close to $\mu$, however, a Taylor expansion gives the local connection:

\begin{aligned} 2\left[y\log(y/\mu)-(y-\mu)\right] &=2\mu\left[(1+u)\log(1+u)-u\right],\qquad u=(y-\mu)/\mu,\\ &=\mu u^2+O(\mu|u|^3) =\frac{(y-\mu)^2}{\mu}+O(\mu|u|^3). \end{aligned}

Thus, for small relative residuals, deviance and Pearson contributions are approximately alike. For larger discrepancies they can differ, so one ratio need not mirror the other exactly. A misspecified mean can increase either statistic, but a large deviance ratio is not a mathematical proof of extra-Poisson or extra-binomial variation.

Diagnostic principle: A dispersion ratio summarizes the size of residual variation; residuals against fitted means, predictors, and relevant groupings help diagnose its source. A systematic pattern suggests that the mean structure deserves attention before the ratio is interpreted as evidence of extra variation.

Using the Diagnostics Carefully

When residuals vary systematically with a predictor, check whether the model’s functional form is plausible. Curvature can suggest a nonlinear effect; different slopes across groups can suggest an interaction; a consistent discrepancy between observed and fitted patterns can be compatible with a link that is too restrictive. These are diagnostic clues, not automatic instructions to add terms: scientific knowledge, design, and the range of observed data matter as well.

A common mistake is to reason, “The Pearson ratio is above one, so the response must be overdispersed.” The correction is to ask whether the conditional means are adequately represented. As the identity above shows, mean error contributes squared bias to the expected Pearson numerator. Conversely, a ratio near one does not certify a correct mean function: small errors, limited data, or variance-benchmark effects can obscure a pattern. Parameter estimation also affects residuals and their degrees of freedom, so the fixed-mean calculations here explain mechanisms rather than provide an exact reference distribution for every fitted ratio.

A practical interpretation therefore combines the summary statistic with residual structure. Look for signs, curvature, or grouping patterns that persist across predictor values, and distinguish them from isolated large residuals. If a plausible mean-structure adjustment removes a pattern, reassess the dispersion diagnostics under that model. If substantial unexplained variation remains after the mean structure is credible, extra variation may still be a concern.

Check Your Understanding

Use the identities and examples above to answer these questions.

For fixed fitted mean $\mu_i$, why does the expected squared residual contain both $v_i$ and $(m_i-\mu_i)^2$?
In the Poisson example, what residual sign pattern would you expect at the two endpoints and at the center?
Why can an omitted interaction have a particularly large effect on the Pearson statistic for grouped binomial counts?
Does a dispersion ratio above one establish that the response has extra variation? Explain what else should be examined.
Why are deviance and Pearson contributions similar for small relative residuals, but not necessarily identical for large residuals?