When GLM Likelihood Equations Have Finite Solutions

Why a Score Equation May Have No Finite Solution

Likelihood equations describe where the likelihood is stationary, but they do not ensure that a stationary point exists. Even when the design identifies every coefficient and the likelihood is concave, its supremum may occur only as one or more coefficients tend to infinity. This distinction matters in practice: an optimization routine can keep moving toward a better likelihood without ever reaching a finite estimate.

We focus first on a canonical-link generalized linear model with independent observations and fixed dispersion. Let \(x_i\) denote the design vector for observation \(i\), and let \(\beta\in\mathbb{R}^p\) be the coefficient vector. A known offset can be included in the linear predictor as \(\eta_i=x_i^\mathsf{T}\beta+o_i\); it does not change the coefficient-design arguments below. Up to terms not depending on \(\beta\), the log-likelihood is

\ell(\beta)=\sum_{i=1}^n \frac{y_i\eta_i-b(\eta_i)}{a_i(\phi)}.

Here \(b\) is the cumulant function and \(a_i(\phi)>0\) is the dispersion scaling, including any known observation-specific weight. We assume throughout that each observed value is in the model's support and that the displayed likelihood is finite for every finite coefficient vector under consideration.

Definition: A finite interior maximum is a maximizer \(\hat\beta\in\mathbb{R}^p\) at which the likelihood is attained, rather than approached only along a sequence of coefficients whose norm diverges. A coefficient vector is identifiable from the design when \(X\beta\) determines \(\beta\) uniquely; for the fixed design matrix \(X\), this is equivalent to \(X\) having full column rank.

Concavity, Identifiability, and What They Do Not Guarantee

For a canonical link, the linear predictor is the canonical parameter. Differentiating the log-likelihood gives the score and Hessian

U(\beta)=\sum_{i=1}^n \frac{x_i\{y_i-b'(\eta_i)\}}{a_i(\phi)}, \qquad \nabla^2\ell(\beta)=- \sum_{i=1}^n \frac{b''(\eta_i)}{a_i(\phi)}x_ix_i^\mathsf{T}.

For a regular exponential family, \(b''(\eta_i)\) is the variance function at the corresponding mean and is positive at finite interior parameter values. The Hessian is therefore negative semidefinite, so \(\ell\) is concave. If \(X\) has full column rank and every \(b''(\eta_i)/a_i(\phi)>0\), the Hessian is negative definite: for every nonzero \(v\in\mathbb{R}^p\),

v^\mathsf{T}\nabla^2\ell(\beta)v = -\sum_{i=1}^n \frac{b''(\eta_i)}{a_i(\phi)} (x_i^\mathsf{T}v)^2 < 0.

The strict inequality follows because full column rank implies \(Xv\ne 0\) for \(v\ne0\), so at least one squared term is positive. Thus the log-likelihood is strictly concave, and it can have at most one finite maximizer. But strict concavity says nothing by itself about whether a maximum is attained: a strictly concave function can increase toward a limiting value at infinity.

Theorem: If \(\ell\) is differentiable and concave on \(\mathbb{R}^p\), any finite score root \(\hat\beta\) satisfying \(\nabla\ell(\hat\beta)=0\) is a global maximizer. If \(\ell\) is strictly concave, that maximizer is unique.

Proof. Fix any \(\gamma\in\mathbb{R}^p\) and consider \(g(t)=\ell(\hat\beta+t(\gamma-\hat\beta))\) for \(0\le t\le1\). Concavity of \(\ell\) implies concavity of \(g\). For a differentiable concave function on an interval, its secant slope from 0 to 1 is no greater than its right derivative at 0. Hence

\ell(\gamma)-\ell(\hat\beta) =g(1)-g(0) \le g'(0) =\nabla\ell(\hat\beta)^\mathsf{T}(\gamma-\hat\beta) =0.

So \(\ell(\gamma)\le\ell(\hat\beta)\) for every \(\gamma\), proving global maximality. If \(\ell\) is strictly concave, two distinct points cannot both maximize it, so the maximizer is unique. \(\square\)

This result gives a useful distinction: for a concave canonical-link likelihood, a finite score root is enough to certify a global maximum. But concavity and full rank do not create a score root. A separate existence condition is needed.

Theorem: Suppose \(\ell\) is continuous on \(\mathbb{R}^p\) and \(\ell(\beta)\to-\infty\) as \(\|\beta\|\to\infty\). Then \(\ell\) attains a finite global maximum.

Proof. Choose any \(\beta_0\). By the assumed limit, there is a radius \(R\) such that \(\|\beta\|>R\) implies \(\ell(\beta)<\ell(\beta_0)\). On the closed ball \(\{\beta:\|\beta\|\le R\}\), continuity and compactness imply that \(\ell\) attains a maximum. That maximum is at least \(\ell(\beta_0)\), while every point outside the ball has a smaller value. It is therefore a global maximum at a finite coefficient vector. \(\square\)

This is a sufficient condition, not a necessary one: a likelihood can attain a maximum even if it does not tend to minus infinity in every direction. Its practical lesson is that one must rule out escaping directions, rather than infer existence from curvature alone.

Worked Examples: Attainment and Escape Directions

Worked Example: A Finite Logistic Maximum

Consider an intercept-only Bernoulli logit model with four independent observations, two successes and two failures. The linear predictor is \(\eta_i=\beta\), and the log-likelihood, apart from constants, is

\ell(\beta)=2\beta-4\log(1+e^\beta).

Its score is \(U(\beta)=2-4e^\beta/(1+e^\beta)\). At \(\beta=0\), \(U(0)=2-4(1/2)=0\). The second derivative is \(-4e^\beta/(1+e^\beta)^2<0\) for every finite \(\beta\), so the likelihood is strictly concave. The score-root theorem shows that \(\hat\beta=0\) is the unique global maximum. Here the observed response pattern permits a finite estimate: the fitted success probability is \(e^0/(1+e^0)=1/2\), matching the observed fraction \(2/4\).

Worked Example: Complete Separation in Logistic Regression

Take two observations with a predictor \(x\): \((x_1,y_1)=(0,0)\) and \((x_2,y_2)=(1,1)\). Use the design vectors \(x_i=(1,x_i)^\mathsf{T}\), so \(X\) has rows \((1,0)\) and \((1,1)\) and has full column rank. With \(\beta=(\beta_0,\beta_1)^\mathsf{T}\), consider the coefficient path \(\beta(t)=(-t,2t)^\mathsf{T}\), where \(t>0\). The two linear predictors are \(-t\) and \(t\).

For the first observation, the probability of its observed failure is \(1-\operatorname{logit}^{-1}(-t)=\operatorname{logit}^{-1}(t)\). For the second, the probability of its observed success is \(\operatorname{logit}^{-1}(t)\). Therefore

\ell(\beta(t))=2\log\{\operatorname{logit}^{-1}(t)\} =2\log\left(\frac{1}{1+e^{-t}}\right).

This increases strictly with \(t\) and tends to 0 as \(t\to\infty\). At every finite \(t\), \(\operatorname{logit}^{-1}(t)<1\), so \(\ell(\beta(t))<0\). The supremum is not attained at any finite coefficient vector. The design identifies both coefficients and the likelihood is strictly concave, yet there is no finite maximum. The separating response pattern, not a rank defect, is the problem.

Worked Example: All-Zero Counts in an Intercept-Only Poisson Model

Let \(Y_i\sim\operatorname{Poisson}(\mu)\) independently, with \(\log\mu=\beta\), and suppose the observed counts are all zero. Ignoring constants, the log-likelihood is

\ell(\beta)=-n e^\beta, \qquad U(\beta)=-n e^\beta.

For every finite \(\beta\), \(U(\beta)<0\), so the score equation has no finite root. As \(\beta\to-\infty\), \(\ell(\beta)\to0\), which is its supremum, but \(\ell(\beta)<0\) at every finite value. Thus the MLE for the log-mean coefficient is not finite, despite the intercept-only design being identifiable. The fitted mean approaches zero without ever equalling zero at a finite log-mean.

Response Patterns and Logistic Separation

The logistic example illustrates a general diagnostic. For binary observations, define the signed design vector \(z_i=(2y_i-1)x_i\), where \(y_i\in\{0,1\}\). Under the logit link, the contribution to the log-likelihood can be written as \(\log\{\operatorname{logit}^{-1}(z_i^\mathsf{T}\beta)\}\). If some direction \(d\) satisfies

z_i^\mathsf{T}d\ge0\ \text{for every }i, \qquad z_i^\mathsf{T}d>0\ \text{for at least one }i,

then moving from \(\beta\) to \(\beta+td\), for \(t>0\), increases or preserves every observation's contribution and strictly increases at least one. This follows because \(u\mapsto\log\{\operatorname{logit}^{-1}(u)\}\) is strictly increasing. Such a direction rules out a finite global maximum. Strict separation is the special case where every signed projection is positive; weak inequalities with some equalities also signal a direction of nonattainment when at least one projection is positive. If all projections are zero, the direction changes no fitted linear predictor and instead reflects a design nonidentifiability.

This criterion is a way to detect failure, not a general existence test for every GLM. For logistic regression, separation is the central response-pattern issue. For other families, boundary patterns can have different forms; the all-zero Poisson example shows that a log-mean can escape toward minus infinity even in a one-parameter model.

Common Mistakes and Practical Interpretation

Assuming full rank guarantees an MLE. Full rank means the design can distinguish coefficient values through their linear predictors. It does not prevent the likelihood from improving along a direction that sends fitted means toward a boundary.
Treating negative curvature as proof of existence. Negative-definite Hessian gives strict concavity and at most one finite maximizer. It does not show that the likelihood reaches its supremum at a finite point.
Calling any score root a maximum. A score root is a stationary point. For a concave likelihood it is a global maximum, as proved above. In a nonconcave likelihood, stationarity alone does not establish maximality; curvature or a global argument is also needed.
Interpreting a large numerical estimate as proof of a finite MLE. An optimizer may stop at a large coefficient because of an iteration limit or tolerance even while the likelihood continues to improve toward a boundary. Examine the response pattern, score behavior, and likelihood along plausible escape directions.

The practical question is not just whether the likelihood equations can be written down, but whether their solution exists in the finite parameter space. Concavity makes a finite root easy to interpret, while design rank separates identifiability from existence. Response patterns determine whether the data actually keep the optimum away from the boundary.

Check Your Understanding

Use the definitions and results above to answer these questions.

Why does full column rank imply a negative-definite canonical-link Hessian when every variance factor is positive?
What does the concavity theorem allow you to conclude from a finite score root, and what additional property makes that root unique?
In the two-observation logistic example, what happens to each fitted response probability as \(t\to\infty\), and why is the supremum not attained?
For an intercept-only Poisson model with all observed counts zero, explain why the score has no finite root even though the design is identifiable.
Give a sufficient condition on the behavior of a continuous log-likelihood at infinity that guarantees a finite global maximum.