Large-Sample Likelihood Theory · Tutorial 255 of 981

Likelihood Limits Under Model Misspecification

Learn how a misspecified GLM likelihood selects its population-optimal parameter and when the fitted estimator converges to it.

Intermediate 12 min read

What You'll Learn

Define the pseudo-true parameter as the maximizer of expected working log-likelihood
Relate that maximization to minimizing expected Kullback–Leibler divergence
State conditions under which the misspecified maximum-likelihood estimator is consistent
Derive the expected-score equations at an interior pseudo-true parameter
Work through Bernoulli, Poisson, and Gaussian examples of misspecification
Distinguish a zero expected score from a global likelihood maximum

When the Working GLM Is Not the Data-Generating Model

A likelihood model can be useful even when it does not exactly describe how the data were generated. A GLM might omit a predictor, impose a linear relationship where the true mean is nonlinear, or use a response distribution that is only an approximation. In these cases, maximum likelihood does not usually converge to a “true” coefficient vector within the working model: there may be no such vector. Instead, it can converge to the parameter that makes the working model best, on average, for the population producing the observations.

That limiting target is called the pseudo-true parameter. It depends both on the working likelihood and on the actual distribution of the data, including the distribution of the covariates. The target is defined by expected log-likelihood, not by an assumption that the working model is correct.

Definition: Let $Z=(Y,X)$ have its actual population distribution $P_0$, and let $q_\beta(y\mid x)$ be a working GLM conditional density or mass function, indexed by $\beta\in\Theta$. Define the population log-likelihood criterion $$ Q(\beta)=E_0[\log q_\beta(Y\mid X)]. $$ If this expectation is finite and $Q$ has a unique maximizer in $\Theta$, that maximizer $\beta^*$ is the pseudo-true parameter: $$ \beta^*=\arg\max_{\beta\in\Theta} Q(\beta). $$

The expectation here is under the actual joint distribution of $(Y,X)$, not under the working GLM. In particular, changing the population distribution of $X$ can change $\beta^*$, even if the same conditional relationship between $Y$ and $X$ is considered. In a fixed-design setting, the analogous population criterion is obtained as a limit of the average log-likelihood over the design points, when that limit exists.

The Population Target and Expected Score

The pseudo-true parameter has a useful divergence interpretation. Suppose the actual conditional distribution of $Y$ given $X=x$ has density or mass function $p_0(y\mid x)$, and the relevant log expectations are finite. For each $x$, the Kullback–Leibler divergence from $p_0(\cdot\mid x)$ to the working distribution $q_\beta(\cdot\mid x)$ is $$ D_{\mathrm{KL}}\{p_0(\cdot\mid x)\,\|\,q_\beta(\cdot\mid x)\} =E_0\left[\left.\log\frac{p_0(Y\mid X)}{q_\beta(Y\mid X)}\right|X=x\right]. $$

Averaging over $X$, the part involving $\log p_0(Y\mid X)$ does not depend on $\beta$. Consequently, maximizing $Q(\beta)$ is equivalent to minimizing the expected conditional Kullback–Leibler divergence, provided these expectations are well-defined. This interpretation does not make the working model correct; it identifies which member of that model is closest in this particular average-divergence sense.

Result (expected-score equation): Suppose $\beta^*$ is an interior maximizer of $Q$, and differentiation can be interchanged with expectation in a neighborhood of $\beta^*$. More precisely, suppose the score $s_\beta(Y,X)=\nabla_\beta\log q_\beta(Y\mid X)$ exists there and is dominated componentwise by an integrable random variable in that neighborhood. Then $$ \nabla Q(\beta^*)=E_0[s_{\beta^*}(Y,X)]=0. $$

To justify the result, the domination condition permits differentiation under the expectation, so $\nabla Q(\beta)=E_0[s_\beta(Y,X)]$ near $\beta^*$. Since $\beta^*$ is an interior maximum of a differentiable function, each directional derivative there is zero. Hence every component of $\nabla Q(\beta^*)$, and therefore the expected score, is zero. $\square$

This equation characterizes an interior optimum, but it does not by itself establish that the parameter is a global maximum or that there is only one solution. Those properties require additional features of the population criterion, such as strict concavity or a separate uniqueness assumption. In a correctly specified model, the true parameter often has zero expected score; under misspecification, the equation instead balances the working model’s score over the actual population.

Why the Estimator Converges to This Target

For independent and identically distributed observations $Z_i=(Y_i,X_i)$, write the average working log-likelihood as $$ Q_n(\beta)=\frac{1}{n}\sum_{i=1}^n \log q_\beta(Y_i\mid X_i). $$ The maximum-likelihood estimator maximizes $Q_n$, while $\beta^*$ maximizes its population counterpart $Q$. A uniform law of large numbers makes these two criteria close over the parameter space; uniqueness and separation of the population maximum then force their maximizers close as well.

Theorem (consistency for a misspecified likelihood): Let $\Theta$ be compact, and suppose $Q(\beta)=E_0[\log q_\beta(Y\mid X)]$ is finite and continuous on $\Theta$, with unique maximizer $\beta^*$. Suppose $$ \sup_{\beta\in\Theta}|Q_n(\beta)-Q(\beta)|\xrightarrow{p}0. $$ If $\hat\beta_n$ is an approximate global maximizer satisfying $$ Q_n(\hat\beta_n)\geq \sup_{\beta\in\Theta}Q_n(\beta)-o_p(1), $$ then $\hat\beta_n\xrightarrow{p}\beta^*$. An exact global maximum satisfies the approximate-maximizer condition.

The uniform convergence assumption is typically established from regularity conditions on the log-likelihood, such as measurability, continuity in $\beta$, and an integrable envelope, together with an appropriate uniform law of large numbers. The theorem states the conditions needed for the consistency argument itself; it does not claim those conditions hold automatically for every GLM.

Proof. Fix $\varepsilon>0$. By continuity of $Q$, compactness of $\Theta$, and uniqueness of $\beta^*$, the set $A_\varepsilon=\{\beta\in\Theta:\|\beta-\beta^*\|\geq\varepsilon\}$, if nonempty, is compact and its maximum value is strictly below $Q(\beta^*)$. Thus for some $\delta>0$, $$ \sup_{\beta\in A_\varepsilon}Q(\beta)\leq Q(\beta^*)-\delta. $$ Let $r_n$ denote the nonnegative approximation error in the maximizer condition, so that $r_n=o_p(1)$. On the event that $\hat\beta_n\in A_\varepsilon$, $$ Q_n(\beta^*)-Q_n(\hat\beta_n) \leq r_n, $$ because $Q_n(\hat\beta_n)\geq Q_n(\beta^*)-r_n$. But the population gap and uniform convergence give $$ Q_n(\beta^*)-Q_n(\hat\beta_n) \geq \delta-2\sup_{\beta\in\Theta}|Q_n(\beta)-Q(\beta)|. $$ The right-hand side exceeds, for example, $\delta/2$ with probability tending to one, whereas $r_n<\delta/2$ with probability tending to one. Therefore the event $\{\hat\beta_n\in A_\varepsilon\}$ has probability tending to zero. This holds for every $\varepsilon>0$, proving $\hat\beta_n\xrightarrow{p}\beta^*$. $\square$

Worked Examples

Worked Example: An Intercept-Only Bernoulli Model

Let $X$ take values 0 and 1 with probability $1/2$ each. Suppose the actual conditional distribution is Bernoulli with $P_0(Y=1\mid X=0)=0.2$ and $P_0(Y=1\mid X=1)=0.8$. Fit an intercept-only Bernoulli GLM, $q_\beta(Y=1)=p_\beta=\operatorname{logit}^{-1}(\beta)$, which ignores $X$.

The population success probability is $$ E_0[Y]=\tfrac12(0.2)+\tfrac12(0.8)=0.5. $$ The working log-likelihood contribution is $Y\log p_\beta+(1-Y)\log(1-p_\beta)$. Its expected value is $$ Q(\beta)=0.5\log p_\beta+0.5\log(1-p_\beta). $$ Differentiating with respect to $\beta$, using $dp_\beta/d\beta=p_\beta(1-p_\beta)$, gives $$ Q'(\beta)=0.5-p_\beta. $$ Thus the expected-score equation is $p_\beta=0.5$, which gives $\beta^*=0$. Also $Q''(\beta)=-p_\beta(1-p_\beta)<0$ for finite $\beta$, so this is the unique maximum. The working model does not represent the two distinct conditional success probabilities; its pseudo-true intercept matches their population average on the probability scale.

Worked Example: A Misspecified Poisson Log-Linear Mean

Let $X$ be equally likely to be $-1,0,1$, and let the actual conditional distributions be Poisson with means $1,4,2$, respectively. Fit a Poisson GLM with $\log\lambda_\beta(x)=\alpha+\gamma x$. The true log means are not linear in $x$, so this working mean model is misspecified.

For a Poisson observation, the score for $(\alpha,\gamma)$ is $(Y-\lambda_\beta(X))(1,X)^\mathsf{T}$. The expected-score equations therefore require $$ \lambda_-+\lambda_0+\lambda_+=1+4+2=7,\qquad -\lambda_-+\lambda_+=-1(1)+0(4)+1(2)=1, $$ where $\lambda_-,\lambda_0,\lambda_+$ are the fitted means at $-1,0,1$. The log-linear form also requires $\lambda_0^2=\lambda_-\lambda_+$, since $\lambda_-=e^{\alpha-\gamma}$, $\lambda_0=e^\alpha$, and $\lambda_+=e^{\alpha+\gamma}$.

Set $t=\lambda_-$. The score equations give $\lambda_+=t+1$ and $\lambda_0=6-2t$. The log-linear constraint becomes $$ (6-2t)^2=t(t+1),\qquad 3t^2-25t+36=0. $$ The roots are $t=(25\pm\sqrt{193})/6$. The plus root makes $\lambda_0=6-2t<0$, which cannot be a Poisson mean. Thus $$ \lambda_-=\frac{25-\sqrt{193}}6\approx1.851,\quad \lambda_0=\frac{\sqrt{193}-7}3\approx2.297,\quad \lambda_+=\frac{31-\sqrt{193}}6\approx2.851. $$ They are positive, sum to 7, and have $\lambda_+-\lambda_-=1$, verifying both expected-score equations. The corresponding coefficients are $\alpha=\log\lambda_0\approx0.831$ and $\gamma=\tfrac12\log(\lambda_+/\lambda_-)\approx0.216$. These fitted means are a compromise: they obey the imposed log-linear restriction rather than reproducing all three true means.

Worked Example: A Linear Gaussian Fit to a Curved Mean

Let $X$ be equally likely to be $-1,0,1$, and let $Y=X^2+\epsilon$, where $\epsilon\sim N(0,1)$ independently of $X$. Fit a Gaussian GLM with variance fixed at 1 and working mean $\mu_{\alpha,\gamma}(X)=\alpha+\gamma X$. The true conditional mean $X^2$ is not linear in $X$.

Ignoring terms not involving $(\alpha,\gamma)$, the Gaussian score equations are $$ E_0[Y-\alpha-\gamma X]=0,\qquad E_0[X(Y-\alpha-\gamma X)]=0. $$ Since $E_0[Y]=E_0[X^2]=2/3$, $E_0[X]=0$, $E_0[X^2]=2/3$, and $E_0[X^3]=0$, the first equation gives $2/3-\alpha=0$. The second gives $0-\gamma(2/3)=0$. Hence $\alpha^*=2/3$ and $\gamma^*=0$. The population criterion is strictly concave in these coefficients because the design has positive variation in both the intercept and slope directions, so this solution is the unique maximum. At $X=0$, for example, the true mean is 0 while the fitted mean is $2/3$: the expected score is zero because the residuals balance across the covariate distribution, not because the fitted conditional mean is correct at every $X$.

Interpretation and a Common Pitfall

The pseudo-true parameter is a population target defined by a specified working likelihood and a specified data distribution. It is not automatically a scientifically meaningful “effect,” and it need not describe the response well at every covariate value. If the model is used to summarize a relationship, the target’s dependence on the covariate distribution matters: a different population mix can produce a different best-fitting coefficient.

Pitfall: “The expected score is zero, so the model must be correctly specified.” This is incorrect. The Bernoulli and Poisson examples have zero expected score at their pseudo-true parameters despite their misspecified conditional mean structures.
Correction: Zero expected score is a first-order condition for an interior optimum, not a test of model correctness. Correct specification would require the working conditional distribution to agree with the actual one (up to the relevant almost-sure qualifications). Under misspecification, the score can average to zero while conditional discrepancies remain.
Pitfall: “Any solution of the expected-score equations is the pseudo-true parameter.” This need not hold: a stationary point may fail to be a global maximum, and multiple maxima may exist.
Correction: Define the target by maximizing $Q$. Use the expected-score equation as a characterization only alongside conditions that justify differentiation and establish the relevant maximum and uniqueness.

Key takeaway: Under misspecification, maximum likelihood targets the maximizer of expected working log-likelihood. With suitable uniform convergence and a unique separated population maximum, the estimator converges to that pseudo-true parameter; when it is interior and differentiation is justified, its expected score is zero.

Check Your Understanding

Use the definitions and examples above to answer the following questions.

Why can the pseudo-true parameter depend on the population distribution of the covariates?
What assumptions allow the expected-score equation to be obtained by differentiating the population criterion?
In the Bernoulli example, why does the fitted intercept equal zero even though the conditional success probabilities are not both one-half?
Why does the zero expected score in the Poisson example not make the log-linear mean model correctly specified?
In the consistency theorem, what roles do uniform convergence and uniqueness of the population maximum play?

Back to Tutorials