Concavity, Stationary Points, and Finite Likelihood Maxima

Why curvature is not the whole existence question

Newton and Fisher-scoring methods use derivatives to move toward a likelihood optimum. But a small score or a seemingly favorable curvature does not, by itself, answer the central question: does a finite coefficient vector actually maximize the likelihood? A likelihood can be concave and still increase toward a limiting value as one or more coefficients diverge. Conversely, if a concave log-likelihood has a stationary point in the parameter space, that point is already a global maximizer.

The distinction matters when interpreting an optimizer’s output. Curvature helps classify a point locally; concavity can extend that conclusion globally; existence asks whether the best value is attained at a finite parameter value at all.

Definition: Let \(\ell(\beta)\) be a twice differentiable log-likelihood on a convex parameter domain in \(\mathbb{R}^p\). Its score is \(U(\beta)=\nabla\ell(\beta)\), and its Hessian is \(H(\beta)=\nabla^2\ell(\beta)\). A point \(\hat\beta\) is stationary if \(U(\hat\beta)=0\). A finite maximum is attained at some \(\hat\beta\in\mathbb{R}^p\), rather than only approached along a sequence whose coefficients leave every bounded set.

What the Hessian says at a stationary point

For a twice differentiable function, a negative definite Hessian at a stationary point is a sufficient condition for a strict local maximum. A positive definite Hessian gives a strict local minimum. An indefinite Hessian gives directions of both increase and decrease, so the stationary point is a saddle point. A negative semidefinite Hessian is less decisive: it is consistent with a maximum, but by itself does not establish a strict maximum or rule out flat directions.

For example, \(f(x,y)=x^2-y^2\) has zero gradient at \((0,0)\) and Hessian with diagonal entries \(2\) and \(-2\). Moving along the \(x\)-axis increases \(f\); moving along the \(y\)-axis decreases it. The stationary point is therefore a saddle, not a maximum. This illustrates why “the score is zero” is not a classification.

A useful GLM setting is the canonical-link model for independent observations from an exponential family. Suppose the log-likelihood, apart from terms not depending on \(\beta\), is

\ell(\beta)=\sum_{i=1}^n \frac{y_i\theta_i-b(\theta_i)}{a_i}, \qquad \theta_i=x_i^\mathsf{T}\beta,\qquad a_i>0.

Here \(b\) is the cumulant function and \(x_i\) is the covariate vector for observation \(i\). For a regular exponential family, \(b''(\theta_i)>0\) wherever the natural parameter is in its allowed interior. Differentiating gives

U(\beta)=\sum_{i=1}^n x_i\frac{y_i-b'(\theta_i)}{a_i}, \qquad H(\beta)=-\sum_{i=1}^n x_i x_i^\mathsf{T}\frac{b''(\theta_i)}{a_i}.

For every vector \(v\in\mathbb{R}^p\),

v^\mathsf{T}H(\beta)v =-\sum_{i=1}^n \frac{b''(\theta_i)}{a_i}(x_i^\mathsf{T}v)^2\leq 0.

Thus the Hessian is negative semidefinite, and the log-likelihood is concave on any convex region where these derivatives exist. If the design matrix has full column rank, then for every nonzero \(v\) at least one \(x_i^\mathsf{T}v\) is nonzero. Since all weights \(b''(\theta_i)/a_i\) are positive, the displayed quadratic form is then strictly negative. The Hessian is negative definite, and the log-likelihood is strictly concave.

Worked Example: A Finite Logistic Intercept Maximum

Suppose three independent Bernoulli observations have outcomes \(1,1,0\), with a logit model containing only an intercept \(\beta\). Write \(p=\exp(\beta)/(1+\exp(\beta))\). The log-likelihood is

\ell(\beta)=2\log(p)+\log(1-p) =2\beta-3\log(1+\exp(\beta)).

The score and second derivative are

U(\beta)=2-3p,\qquad H(\beta)=-3p(1-p).

The stationary equation \(2-3p=0\) gives \(p=2/3\), hence \(\exp(\hat\beta)=2\) and \(\hat\beta=\log 2\). At this point \(H(\hat\beta)=-3(2/3)(1/3)=-2/3<0\), so it is a strict local maximum. More strongly, \(H(\beta)<0\) for every finite \(\beta\), so \(\ell\) is strictly concave. Its stationary point is therefore the unique global maximum. The attained log-likelihood is \(2\log(2/3)+\log(1/3)=\log(4/27)\).

Concavity turns stationarity into global optimality

The key consequence of concavity is a global one: at any point, the tangent plane lies above the function. This first-order inequality explains why a zero score is sufficient for global maximization when the log-likelihood is concave.

Theorem: Let \(\ell\) be differentiable and concave on an open convex domain \(D\subseteq\mathbb{R}^p\). If \(\hat\beta\in D\) satisfies \(\nabla\ell(\hat\beta)=0\), then \(\ell(\beta)\leq\ell(\hat\beta)\) for every \(\beta\in D\). Thus \(\hat\beta\) is a global maximizer. If \(\ell\) is strictly concave, this maximizer is unique.

Proof. Fix any \(\beta\in D\) and define \(g(t)=\ell(\hat\beta+t(\beta-\hat\beta))\) for \(0\leq t\leq1\). Convexity of \(D\) ensures that the line segment stays in \(D\). Concavity of \(\ell\) makes \(g\) concave. For \(0<t\leq1\), concavity gives \(g(t)\geq(1-t)g(0)+tg(1)\). Rearranging,

g(1)-g(0)\leq\frac{g(t)-g(0)}{t}.

As \(t\) decreases to zero, the right side converges to \(g'(0)=\nabla\ell(\hat\beta)^\mathsf{T}(\beta-\hat\beta)=0\). Therefore \(g(1)\leq g(0)\), which is \(\ell(\beta)\leq\ell(\hat\beta)\). If \(\ell\) is strictly concave, two distinct maximizers would have a midpoint with a strictly larger value, contradicting maximality. Hence the maximizer is unique. \(\square\)

This result establishes global optimality if a stationary point exists inside the domain. It does not assert that one must exist. Nor does a zero score guarantee uniqueness unless strict concavity (or another suitable condition) is available.

Worked Example: A Finite Maximum With Nonunique Coefficients

Take a Poisson model with log link and two coefficients, but let both columns of the design matrix be identical: \(x_i=(1,1)^\mathsf{T}\) for every observation. Then the linear predictor for each observation is \(\eta_i=\beta_1+\beta_2\). Suppose the observed counts are \(y_1=1\) and \(y_2=3\). Ignoring constants, with \(\lambda=\exp(\beta_1+\beta_2)\),

\ell(\beta_1,\beta_2)=4(\beta_1+\beta_2)-2\exp(\beta_1+\beta_2).

Let \(s=\beta_1+\beta_2\). Differentiating with respect to either coefficient gives \(4-2\exp(s)\). The stationary condition is \(\exp(s)=2\), so every finite pair satisfying \(\beta_1+\beta_2=\log 2\) is stationary. The Hessian is

H(\beta_1,\beta_2)=-2\exp(s) \begin{pmatrix}1&1\\1&1\end{pmatrix}.

Its quadratic form is \(-2\exp(s)(v_1+v_2)^2\leq0\), but it is zero when \(v_1=-v_2\). The likelihood depends only on \(s\), so it is strictly concave in \(s\) and is maximized at \(s=\log 2\); its maximum value, apart from the omitted constants, is \(4\log 2-4\). This is a finite global maximum, but the coefficient vector is not unique. The rank-deficient design leaves one direction of coefficients unidentified.

Concavity does not guarantee a finite maximum

A continuous function attains a maximum on a compact domain, but a GLM coefficient space is typically \(\mathbb{R}^p\), which is not compact. Concavity alone does not prevent the likelihood from approaching its supremum along an unbounded sequence. If that limiting value is never reached at a finite coefficient vector, there is no finite maximum, even though every finite point may have negative curvature in the relevant direction.

Worked Example: Logistic Likelihood Approaching a Boundary Supremum

Suppose a logistic model has an intercept only and \(n\) Bernoulli observations, all equal to 1. With \(p=\exp(\beta)/(1+\exp(\beta))\), the log-likelihood and score are

\ell(\beta)=n\log(p)=n\beta-n\log(1+\exp(\beta)), \qquad U(\beta)=n(1-p)>0

for every finite \(\beta\). Since the score is positive everywhere, the log-likelihood is strictly increasing and has no finite stationary point. As \(\beta\) tends to \(+\infty\), \(p\) tends to 1 and \(\ell(\beta)\) tends to 0. For every finite \(\beta\), \(p<1\), so \(\ell(\beta)=n\log(p)<0\). Thus 0 is the supremum, but it is not attained: the probability approaches the boundary value 1 only as the intercept diverges. This example shows why concavity and negative curvature do not, on their own, establish existence of a finite estimate.

Reading score behavior and avoiding a common mistake

In one dimension, concavity makes the score nonincreasing. If the score changes from positive to negative and is zero at a finite point, that point is a global maximum; strict concavity makes it unique. If the score remains positive at every finite parameter value, as in the all-successes example, the likelihood keeps increasing and a finite stationary maximum does not exist. In several dimensions, the theorem gives the corresponding conclusion once a stationary point is found, while the behavior along particular directions can help reveal whether values keep improving away from that point.

These statements concern the mathematical likelihood, not just the numerical status reported by an optimizer. An algorithm can stop because an update is tiny or because of numerical limits; that alone does not prove that a finite maximum exists. The score, curvature, parameter rank, and behavior toward the edge of the parameter space all inform the conclusion.

Common mistake: “The score is zero, so the fitted coefficients are the unique maximum.” A zero score only establishes stationarity. Without concavity it need not be a maximum; even with concavity, uniqueness requires strict concavity or another argument.
Correction: Check the curvature or establish concavity, then use the global-optimality result. Separately assess whether a finite stationary point exists and whether the model identifies every coefficient direction.
Common mistake: “A concave likelihood must have a finite maximum.” Concavity controls the shape, not whether the supremum is attained on an unbounded domain.

Key takeaway: A stationary point of a differentiable concave log-likelihood is a global maximum; strict concavity makes it unique. But concavity does not ensure that a stationary point or finite maximum exists, and a flat direction can leave multiple finite maximizers.

Check Your Understanding

Use the score, curvature, and domain together when you answer.

What does a negative definite Hessian establish at a stationary point, and how does that differ from a negative semidefinite Hessian?
Why does concavity make an interior stationary point a global maximizer?
In the Poisson example with identical design columns, why is the maximum finite but the coefficient vector nonunique?
For the all-successes logistic model, what is the supremum of the log-likelihood, and why is it not attained at a finite intercept?
Can a zero score alone establish that a GLM fit is a unique finite maximum? State the additional issues that must be checked.