Consistency of GLM Maximum-Likelihood Estimators

Why Consistency Requires More Than a Good Fit

A maximum-likelihood estimator is computed by maximizing the likelihood in the observed sample. Consistency asks a different question: as the sample grows, does that maximizer approach the parameter that generated the data? The key difficulty is that the maximizing parameter itself depends on the sample. Convergence of the log-likelihood at each fixed parameter does not, by itself, control its random maximizer.

A useful consistency argument combines two facts. First, the sample average log-likelihood must converge uniformly over the parameter space to its expectation. Second, the expected log-likelihood must have a uniquely separated maximum at the true parameter. Uniform convergence then prevents the sample maximizer from remaining far from that maximum.

A Uniform-Convergence Consistency Theorem

Let \(Z_1,\ldots,Z_n\) be independent and identically distributed observations, and let \(q_\theta(Z_i)\) be the log-likelihood contribution at parameter \(\theta\in\Theta\). Write the sample average and its expectation as

The theorem below makes explicit the conditions needed for the argument. Compactness is imposed here to give a simple separation and uniform-convergence result; applications with an unrestricted parameter space need additional conditions that keep the maximizer in a suitable bounded region.

The proof has two parts: a uniform law of large numbers and an argument showing that uniform approximation preserves the location of a separated maximum.

Proof of the Uniform Law of Large Numbers

Under the first two assumptions of the theorem,

To prove this, define the modulus \(\omega_\delta(z)=\sup\{|q_\theta(z)-q_{\theta'}(z)|:\theta,\theta'\in\Theta,\ \|\theta-\theta'\|\leq\delta\}\). For almost every \(z\), continuity on compact \(\Theta\) implies uniform continuity, so \(\omega_\delta(z)\to0\) as \(\delta\downarrow0\). Also, \(\omega_\delta(z)\leq2M(z)\). The dominated convergence theorem therefore gives \(E[\omega_\delta(Z_1)]\to0\).

Fix \(\delta>0\) and choose a finite \(\delta\)-net \(\theta_1,\ldots,\theta_K\) in \(\Theta\): every \(\theta\in\Theta\) lies within distance \(\delta\) of at least one net point. For such a point \(\theta_j\), the triangle inequality gives

For fixed \(\delta\), the first term converges to zero in probability by the weak law of large numbers at the finitely many net points. The second term converges in probability to \(E[\omega_\delta(Z_1)]\), again by the weak law, since \(\omega_\delta\leq2M\) is integrable. Given any positive tolerance, choose \(\delta\) so that \(E[\omega_\delta(Z_1)]\) is sufficiently small. The displayed bound then proves the asserted uniform convergence. \(\square\)

Why a Separated Maximum Gives Consistency

Fix \(\epsilon>0\), and consider the parameters at least \(\epsilon\) away from the true value. If this set is nonempty, it is a closed subset of compact \(\Theta\), hence compact. Continuity of \(Q\) and uniqueness of its maximum imply a strictly positive gap:

Continuity of \(Q\) follows from the continuity of \(q_\theta(Z_1)\) and the integrable envelope, by dominated convergence. Let \(U_n=\sup_{\theta\in\Theta}|Q_n(\theta)-Q(\theta)|\). On the event \(U_n<\Delta_\epsilon/4\) and \(r_n<\Delta_\epsilon/4\), any \(\hat{\theta}_n\) outside the \(\epsilon\)-ball would satisfy

This contradicts the approximate-maximizer condition, since \(\sup_{\theta\in\Theta}Q_n(\theta)\geq Q_n(\theta_0)\). Uniform convergence gives \(U_n\xrightarrow{p}0\), and \(r_n\xrightarrow{p}0\), so the probability that either required bound fails tends to zero. Thus \(P(\|\hat{\theta}_n-\theta_0\|\geq\epsilon)\to0\). If the outside set is empty, that probability is already zero. This proves consistency. \(\square\)

Checking the Argument for a Canonical-Link GLM

Consider independent and identically distributed random-design observations \(Z_i=(X_i,Y_i)\) from a GLM with known dispersion \(\phi\). Suppose the conditional response distribution is a one-parameter exponential family with log-density or log-mass

Here the canonical link makes the natural parameter \(\eta=x^\mathsf{T}\beta\). Assume the parameter space for \(\beta\) is compact, the true value \(\beta_0\) belongs to it, and all linear predictors in the parameter space lie in the natural-parameter domain. For a direct verification of the uniform law, assume \(X\) is bounded and \(E[|Y|]<\infty\). On the resulting bounded range of predictors, assume \(b\) is continuous and finite. The log-likelihood contribution, omitting the term \(c(Y,\phi)\), is

Bounded \(X\) and compactness of the parameter space bound \(|X^\mathsf{T}\beta|\). Consequently, for some finite constants \(C_1,C_2\), \(|q_\beta(X,Y)|\leq C_1|Y|+C_2\) for every \(\beta\). This is an integrable envelope. Continuity in \(\beta\) follows from continuity of \(b\). The uniform law therefore applies.

It remains to identify the unique expected maximum. For a regular exponential family, \(E[Y\mid X]=b'(X^\mathsf{T}\beta_0)\). Conditional on \(X=x\), the expected log-likelihood difference from the true value is

Convexity of \(b\) implies \(b(\eta)\geq b(\eta_0)+b'(\eta_0)(\eta-\eta_0)\), so this conditional difference is nonpositive. If \(b\) is strictly convex on the predictor range, equality holds only when \(\eta=\eta_0\). If \(E[XX^\mathsf{T}]\) is positive definite, then \(X^\mathsf{T}(\beta-\beta_0)=0\) almost surely is possible only when \(\beta=\beta_0\): indeed,

Thus the expected log-likelihood has a unique maximum at \(\beta_0\). Together with the uniform law, these conditions give consistency of the global maximum-likelihood estimator over the stated compact parameter space. The argument concerns a global maximizer; it does not claim that a numerical algorithm has found one.

Common Pitfall: Pointwise Convergence Is Not Enough

A tempting argument is: “For every fixed \(\theta\), \(Q_n(\theta)\) converges to \(Q(\theta)\), and \(Q\) is maximized at \(\theta_0\), so the maximizer of \(Q_n\) must approach \(\theta_0\).” This skips the key issue. Pointwise convergence controls each fixed parameter separately, not the maximum over parameters that can change with the sample. A random maximizer could select a location where the sample criterion deviates substantially from its expectation.

The correction is to establish uniform convergence over the parameter space and a positive expected-likelihood gap outside every neighborhood of \(\theta_0\). The compactness, continuity, and integrable-envelope conditions above are one standard route to uniform convergence. In applications where parameters are unrestricted, it is not valid simply to assume those compact-space arguments still apply: one must also justify that the estimator cannot escape to arbitrarily large parameter values.