Effects of Distributional Misspecification on GLM Estimates

Why the Response Family Matters Even When the Mean Is Right

A GLM response family describes more than the conditional mean. It also specifies a distributional shape, a variance relationship, and, often, a tail behavior. Choosing the wrong family can therefore affect likelihood-based inference even when the fitted mean function is correct. The key distinction is between estimating the mean and describing the full conditional distribution.

For some GLM likelihoods, the score for the regression coefficients is a weighted residual. If the conditional mean model is correct, that score has expectation zero even if the weights, and hence the proposed variance relationship, are wrong. Under suitable conditions the coefficient estimates can still converge to the correct values. However, the likelihood’s curvature generally does not describe the true sampling variability. In other likelihoods, having the right conditional mean alone does not guarantee a zero expected score.

Definition: A response-family model is distributionally misspecified when its proposed conditional distribution is not the true conditional distribution. The conditional mean model is correctly specified when \(E[Y_i\mid x_i]=\mu_i(\beta_0)\) for the true coefficient vector \(\beta_0\) and the stated mean function and link. Correct specification of the mean does not imply correct specification of the full distribution.

When the GLM Score Still Targets the Mean

Consider independent continuous responses \(Y_i\), with fixed covariates \(x_i\). Suppose the candidate model is an exponential-dispersion GLM with mean \(\mu_i(\beta)\), link \(g(\mu_i)=x_i^\top\beta\), variance function \(V(\mu_i)\), and dispersion \(\phi\). Write \(D_i=\partial\mu_i(\beta)/\partial\beta\), a column vector. The candidate log-likelihood score for \(\beta\) is

U_n(\beta)=\sum_{i=1}^n \frac{D_i\{Y_i-\mu_i(\beta)\}}{\phi V(\mu_i(\beta))}.

This expression follows from the exponential-dispersion likelihood derivative with respect to the mean: \(\partial\ell_i/\partial\mu_i=(Y_i-\mu_i)/\{\phi V(\mu_i)\}\). It reveals why a wrong variance function need not move the population target of this score.

Result: If \(E[Y_i\mid x_i]=\mu_i(\beta_0)\), then \(E[U_n(\beta_0)]=0\), provided the displayed score has finite expectation. This is an unbiased estimating equation at \(\beta_0\); it is not a claim that the candidate density is correct.

Indeed, \(D_i\), \(V(\mu_i(\beta_0))\), and \(\phi\) are fixed given the covariates, so

E[U_n(\beta_0)] =\sum_{i=1}^n \frac{D_i\{E[Y_i\mid x_i]-\mu_i(\beta_0)\}} {\phi V(\mu_i(\beta_0))} =0.

An unbiased score at the target is an important protection, but it is not by itself a consistency theorem. Consistency also needs identification and regularity conditions that make the sample estimating equation converge to its population counterpart and ensure that the target is its unique relevant solution. Nor does the argument apply to every likelihood merely because its mean parameter is correct: it relies on the stated residual form of the GLM score.

Worked Example: A Gamma GLM for Lognormal Responses

Suppose \(Y_i\mid x_i\) is lognormal, with \(\log Y_i\mid x_i\sim N(\eta_i,\sigma^2)\), where \(\eta_i=x_i^\top\beta_0\) and \(\sigma^2\) is constant. Its conditional mean is \(\mu_i=\exp(\eta_i+\sigma^2/2)\). A Gamma GLM with a log link can represent this mean by setting its coefficient vector to \(\beta_0\) with the intercept increased by \(\sigma^2/2\).

The Gamma variance function is \(V(\mu)=\mu^2\). With the log link, \(D_i=\mu_i x_i\), so the Gamma coefficient score is

U_n(\beta)=\sum_{i=1}^n \frac{x_i(Y_i-\mu_i(\beta))}{\phi\mu_i(\beta)}.

At the stated coefficient vector, \(E[Y_i\mid x_i]=\mu_i\), and therefore each score contribution has conditional expectation \(x_i(\mu_i-\mu_i)/(\phi\mu_i)=0\). The lognormal and Gamma distributions have different shapes, but this score is still centered at the correct mean coefficients. Whether the fitted coefficients converge there additionally depends on identification and regularity.

The distributions are not interchangeable: the lognormal variable has \(\operatorname{Var}(Y_i\mid x_i)=\mu_i^2\{\exp(\sigma^2)-1\}\), whereas the Gamma candidate specifies \(\phi\mu_i^2\). Even if a single \(\phi\) matches this variance, the Gamma and lognormal shapes remain different. Mean-score robustness does not establish a correct likelihood.

Why the Model-Based Standard Error Can Be Wrong

For a likelihood score \(s_i(\beta)\), define the sensitivity matrix \(A\) as the probability limit of the negative average derivative of the score, and \(B\) as the probability limit of its average covariance. In a regular, correctly specified likelihood, the information identity gives \(A=B\). Under misspecification this equality generally fails. The resulting asymptotic covariance of the coefficient estimator is the sandwich \(A^{-1}BA^{-1}\), rather than just the inverse curvature \(A^{-1}\).

Theorem: Let \(\hat\beta\) solve \(\sum_{i=1}^n s_i(\beta)=0\), with independent observations and fixed coefficient dimension. Suppose the score has mean zero at a unique target \(\beta_0\); the normalized negative score derivative converges in probability to a nonsingular matrix \(A\); the normalized score sum satisfies a central limit theorem with covariance limit \(B\); and the differentiability, moment, and uniform convergence conditions for a Taylor expansion of the score hold. Then

\sqrt n(\hat\beta-\beta_0)\xrightarrow{d} N\!\left(0,A^{-1}BA^{-1}\right).

To derive this result, expand the normalized score about \(\beta_0\):

0=\frac{1}{\sqrt n}\sum_{i=1}^n s_i(\beta_0) +\left\{\frac{1}{n}\sum_{i=1}^n \frac{\partial s_i(\tilde\beta)}{\partial\beta^\top}\right\} \sqrt n(\hat\beta-\beta_0),

where \(\tilde\beta\) lies between \(\hat\beta\) and \(\beta_0\), as permitted by the assumed Taylor conditions. The term in braces converges in probability to \(-A\). The central limit theorem gives \(n^{-1/2}\sum_i s_i(\beta_0)\xrightarrow{d}N(0,B)\). Solving the expansion and applying Slutsky’s theorem gives the stated covariance. Under a correct likelihood, \(B=A\), reducing the expression to \(A^{-1}\); under misspecification, that simplification is not justified. \(\square\)

This theorem concerns the score being solved and its actual variability. In GLMs, a sandwich estimate uses the fitted score contributions to estimate both sensitivity and variability. If dispersion or other nuisance parameters are estimated, their scores should be included in the joint sandwich calculation, or their effect must otherwise be accounted for. The inverse Hessian alone is not a general substitute.

Worked Example: Correct Group Means, Different Standard Errors

There are \(m\) observations in each of two groups. In group 0, responses are exponential with mean 1 and variance 1. In group 1, they are exponential with mean 3 and variance 9. Thus the conditional mean is exactly represented by a two-group model. Fit a Gaussian GLM with separate group means and one common variance.

Because the two group means are separate parameters, the Gaussian maximum-likelihood estimates are the two sample means. Their probability limits are 1 and 3: the mean coefficients target the correct group means despite the wrong Gaussian shapes and common-variance assumption. The fitted common Gaussian variance is the pooled average of the two within-group variances, which converges to \((1+9)/2=5\).

For the estimated mean in group 1, the Gaussian model-based standard error is therefore approximately \(\sqrt{5/m}\). But the actual variance of that sample mean is \(9/m\), so its actual standard error is \(3/\sqrt m\). A sandwich calculation uses the group 1 residual variability and recovers this latter standard error asymptotically. The model-based standard error is too small by the factor \(\sqrt{5}/3\).

For example, under the null hypothesis that the group 1 mean equals its true value 3, the model-based standardized estimate has limiting variance \(9/5\), not 1. A nominal two-sided 5% Wald test using the model-based standard error consequently has limiting rejection probability \(P(|Z|>1.96\sqrt{5/9})\), where \(Z\sim N(0,1)\). Its threshold is about 1.46, giving a rejection probability of about 0.14 rather than 0.05. Correct mean estimates alone do not make this test calibrated.

When Even the Coefficient Target Can Change

A zero expected score follows from the residual-form GLM score, not from the phrase “correct mean” in isolation. A likelihood whose score depends on other features of the response can target a different quantity. For example, maximizing a Laplace location likelihood with fixed scale estimates a median, because its location score is proportional to the sign of the residual. A conditional mean does not generally make that score have expectation zero.

This example is not a standard exponential-dispersion GLM family; it illustrates the broader likelihood principle. Before interpreting coefficients from a misspecified continuous-response likelihood, ask which population equation the fitting procedure solves. If its expected score is zero at the true mean coefficients and that solution is identified, the mean target may be preserved. If not, the estimates can converge to a different, pseudo-true parameter: the value that maximizes expected candidate log-likelihood.

Worked Example: A Wrong Family With an Unbiased Mean Score

Return to the two exponential groups, but now fit a Gaussian GLM with separate means and group-specific variances. The Gaussian score for each group mean is the sum of its residuals divided by that group’s variance parameter. Setting the score to zero gives the group sample mean exactly, so the estimates target 1 and 3.

The candidate Gaussian variance estimates converge to 1 in group 0 and 9 in group 1. Thus the Gaussian fit gets these two conditional variances right, but it still does not reproduce the exponential response shape: a Gaussian distribution is symmetric and has support on the whole real line, while an exponential distribution is positive and right-skewed. This example separates a correct mean score, and even a matching variance, from correct distributional specification.

Practical Implications and a Common Pitfall

Likelihood-based coefficient estimates, standard errors, and tests answer different questions. A coefficient estimate solves a score equation; its large-sample target depends on the expected score. A model-based standard error uses the candidate likelihood’s curvature and its information identity. A test additionally relies on the reference distribution used to calibrate its statistic. Under misspecification, the first can remain reliable while the latter two fail.

A common mistake is: “The conditional mean model is correct, so the likelihood-based standard errors and tests are correct too.” The correction is that a correct mean can center a GLM residual score, but it does not imply the proposed variance or response shape is right. Estimate uncertainty using a justified sandwich covariance when its assumptions hold, and interpret tests using that covariance or another calibration justified for the misspecified setting. Likelihood-ratio statistics also need not have their usual reference distribution under misspecification; do not assume the ordinary likelihood theory applies automatically.

Key takeaway: Distributional misspecification can leave GLM mean coefficients consistent when the candidate score is a suitably weighted residual and the mean model is correct. It can still invalidate likelihood-based standard errors and tests. Correct conditional means are not evidence that the full response family is correct.

Check Your Understanding

For each question, distinguish what follows from a correct conditional mean from what requires a correctly specified distribution.

Why does the exponential-dispersion GLM coefficient score have expectation zero at the true mean coefficients?
In the Gamma-log GLM example, what does matching the conditional mean establish, and what does it not establish?
State the difference between the sensitivity matrix \(A\) and score covariance matrix \(B\) in sandwich inference.
In the two-group Gaussian example, why is the group 1 model-based standard error too small?
Why can a correct conditional mean fail to center a likelihood score that is based on a different feature of the response?