Canonical-Link Hessians And Likelihood Curvature

Why Canonical-Link Curvature Has a Simple Form

A GLM likelihood describes how the observed responses vary as the coefficient vector changes. Its Hessian records the local curvature of that likelihood: negative curvature in every coefficient direction is a strong indication that a stationary point is a local maximum. Under a canonical link, the second derivative simplifies to a weighted cross-product of the design matrix. This makes the curvature’s sign and any flat directions visible through the weights and the rank of the design.

One detail matters at the outset. In the usual independent-observation exponential-family GLM, the canonical-link Hessian’s weights depend on the fitted means (and the dispersion), not on the realized response values. The responses appear in the score, but cancel from the Hessian. Calling these “response-dependent weights” would be misleading if “response” means the observed $y_i$; the derivation below makes the distinction explicit.

Setup and the Canonical-Link Hessian

Definition: Suppose $Y_1,\ldots,Y_n$ are independent, with log densities or log mass functions $$ \log f_i(y_i;\theta_i,\phi) = \frac{y_i\theta_i-b(\theta_i)}{a_i(\phi)}+c_i(y_i,\phi), $$ where $a_i(\phi)>0$, the support does not depend on $\theta_i$, and $b$ is twice differentiable on an open natural-parameter domain. A canonical-link GLM sets $\theta_i=\eta_i=x_i^{\mathsf T}\beta$. Write $X$ for the $n\times p$ design matrix with rows $x_i^{\mathsf T}$, $y=(y_1,\ldots,y_n)^{\mathsf T}$, and $\mu_i=b'(\eta_i)=E[Y_i]$. The dispersion values are treated as fixed when differentiating with respect to $\beta$.

The admissible coefficient set consists of $\beta$ for which every $x_i^{\mathsf T}\beta$ lies in the natural-parameter domain. We work on a convex subset of that set when discussing strict concavity. The variance relation is $\operatorname{Var}(Y_i)=a_i(\phi)b''(\eta_i)$. For a nondegenerate regular family, $b''(\eta_i)>0$ in the interior of the domain.

Ignoring terms that do not involve $\beta$, the log-likelihood is $\ell(\beta)=\sum_i [y_i x_i^{\mathsf T}\beta-b(x_i^{\mathsf T}\beta)]/a_i(\phi)$. Differentiate each summand: its derivative with respect to $\beta$ is $x_i[y_i-b'(\eta_i)]/a_i(\phi)$. Differentiating once more gives the following result.

Theorem: Under the setup above, the score and observed Hessian for the coefficient vector are $$ \nabla_\beta\ell(\beta) = X^{\mathsf T} \begin{pmatrix} (y_1-\mu_1)/a_1(\phi)\\ \vdots\\ (y_n-\mu_n)/a_n(\phi) \end{pmatrix}, \qquad \nabla_\beta^2\ell(\beta) = -X^{\mathsf T}W(\beta)X, $$ where $W(\beta)=\operatorname{diag}(w_1,\ldots,w_n)$ and $w_i=b''(\eta_i)/a_i(\phi)$. In particular, the Hessian does not depend on the realized values $y_i$.

Proof. The $i$th log-likelihood contribution that depends on $\beta$ is $[y_i\eta_i-b(\eta_i)]/a_i(\phi)$, with $\eta_i=x_i^{\mathsf T}\beta$. Since $\nabla_\beta\eta_i=x_i$, its gradient is $x_i[y_i-b'(\eta_i)]/a_i(\phi)$. Summing over $i$ gives the displayed score. Since $\nabla_\beta b'(\eta_i)=b''(\eta_i)x_i$, the derivative of that contribution’s gradient is $-b''(\eta_i)x_ix_i^{\mathsf T}/a_i(\phi)$. Summing these matrices gives $-X^{\mathsf T}W(\beta)X$. The weights contain $b''(\eta_i)$ and $a_i(\phi)$, but not $y_i$, proving the final claim. $\square$

Reading the Sign and Rank

If $b''(\eta_i)\geq 0$ and $a_i(\phi)>0$, all diagonal entries of $W$ are nonnegative. For any direction $v\in\mathbb R^p$, the second directional derivative is $$ v^{\mathsf T}\nabla_\beta^2\ell(\beta)v = -(Xv)^{\mathsf T}W(\beta)(Xv) = -\sum_{i=1}^n w_i(x_i^{\mathsf T}v)^2 \leq 0. $$ Thus the Hessian is negative semidefinite, and the log-likelihood is concave on any convex admissible region. A direction $v$ has zero curvature precisely when $x_i^{\mathsf T}v=0$ for every observation with $w_i>0$. With positive weights for all observations, this is exactly the condition $Xv=0$.

Theorem: At a coefficient vector where all $w_i>0$, the Hessian is negative definite if and only if $X$ has full column rank. If $b''>0$ throughout the natural-parameter values arising in a convex admissible region and $X$ has full column rank, the log-likelihood is strictly concave on that region.

Proof. For any $v$, the quadratic form of the negative Hessian is $\sum_i w_i(x_i^{\mathsf T}v)^2$. If $X$ has full column rank, $Xv\neq 0$ whenever $v\neq 0$; with every $w_i>0$, the sum is then strictly positive. Hence the Hessian is negative definite. Conversely, if $X$ does not have full column rank, there is a nonzero $v$ with $Xv=0$; the quadratic form is zero in that direction, so the Hessian is not negative definite.

For strict concavity on the stated region, take distinct $\beta_0,\beta_1$ and set $d=\beta_1-\beta_0\neq 0$. Along the line segment $\beta(t)=\beta_0+td$, $0\leq t\leq 1$, the second derivative of $g(t)=\ell(\beta(t))$ is $g''(t)=d^{\mathsf T}\nabla_\beta^2\ell(\beta(t))d<0$, by the first part and full column rank. A twice-differentiable function with strictly negative second derivative on the segment is strictly concave there. Thus $\ell$ is strictly concave on the region. $\square$

Both conditions in the theorem matter. Positive $b''$ gives positive weights, but cannot supply curvature in a coefficient direction that leaves every linear predictor unchanged. Full column rank supplies that missing condition. If some weights are zero, the relevant condition is instead that no nonzero $v$ satisfies $x_i^{\mathsf T}v=0$ for all observations with positive weights.

Worked Examples

Worked Example: Poisson Regression Curvature at a Given Coefficient

Consider independent Poisson responses with a log canonical link, an intercept and one predictor, and design rows $x_i^{\mathsf T}=(1,x_i)$ for $x=(0,1,2)$. At $\beta=(0,\log 2)^{\mathsf T}$, calculate the Hessian. For a Poisson distribution, $b(\eta)=e^\eta$, $b''(\eta)=e^\eta$, and $a(\phi)=1$.

The linear predictors are $0,\log 2,2\log 2$, so the fitted means and Hessian weights are $1,2,4$. Therefore $$ X^{\mathsf T}WX = \begin{pmatrix} 1&1&1\\ 0&1&2 \end{pmatrix} \begin{pmatrix} 1&0&0\\ 0&2&0\\ 0&0&4 \end{pmatrix} \begin{pmatrix} 1&0\\ 1&1\\ 1&2 \end{pmatrix} = \begin{pmatrix} 7&10\\ 10&18 \end{pmatrix}. $$ The observed Hessian is the negative of this matrix. Its determinant is $7(18)-10^2=26>0$, and its first diagonal entry is negative after applying the minus sign. Thus the Hessian is negative definite. The calculation does not use the observed counts: changing the counts while keeping $\beta$ fixed leaves this Hessian unchanged.

Worked Example: Gaussian Identity-Link Curvature

Let $Y_i$ be independent normal responses with mean $x_i^{\mathsf T}\beta$ and known variance $\sigma^2=4$. Take design rows $(1,0),(1,1),(1,2)$. In the natural-parameter representation, the Gaussian cumulant has $b''=1$, and $a(\phi)=\sigma^2=4$, so every Hessian weight is $1/4$.

The design cross-product is $$ X^{\mathsf T}X = \begin{pmatrix} 3&3\\ 3&5 \end{pmatrix}, \qquad \nabla_\beta^2\ell(\beta) = -\frac14 \begin{pmatrix} 3&3\\ 3&5 \end{pmatrix}. $$ The determinant of $X^{\mathsf T}X$ is $15-9=6>0$, so the design has full column rank and the Hessian is negative definite. Here the weights do not even vary with the fitted means. As in the general result, the observed responses affect the score but not this Hessian.

Worked Example: A Rank-Deficient Design Has a Flat Coefficient Direction

Consider Poisson regression with three columns: an intercept and two identical predictors $z$. Use observations with $z=0,1,2$, so the rows of $X$ are $(1,0,0),(1,1,1),(1,2,2)$. At $\beta=(0,0,0)^{\mathsf T}$, all fitted means, and therefore all weights, equal 1.

The nonzero vector $v=(0,1,-1)^{\mathsf T}$ satisfies $Xv=0$: each row has the same second and third entries. Consequently $X^{\mathsf T}WXv=0$, and the Hessian has zero curvature along $v$. More strongly, for any real $t$, $X(\beta+tv)=X\beta$. The fitted means and the likelihood are identical at $\beta$ and $\beta+tv$. The data can inform the combined effect of the two duplicate columns, but cannot identify their separate coefficients.

Curvature, Identifiability, and Maximum Likelihood

For positive weights, the Hessian is negative definite exactly when the design has full column rank. The same rank condition explains coefficient identifiability in this setting. If $Xv=0$ for a nonzero $v$, then $\beta$ and $\beta+tv$ produce the same linear predictors for every real $t$. They therefore specify the same natural parameters and, in an identifiable family, the same response distributions. No likelihood calculation can distinguish those coefficient vectors.

When $X$ has full column rank, the map $\beta\mapsto X\beta$ is one-to-one. If the natural-parameter model is identifiable and the weights are positive, the log-likelihood also has strictly negative curvature in every nonzero coefficient direction. On a convex admissible region, strict concavity implies that there can be at most one maximizer in that region. It does not guarantee that a finite maximizer exists: the likelihood may approach its supremum toward the boundary or as coefficients diverge.

At an interior stationary point, a negative-definite Hessian implies a strict local maximum. Indeed, continuity of the Hessian gives a neighborhood in which its quadratic form remains negative for every nonzero direction. Taylor’s theorem around the stationary point then gives a strictly smaller log-likelihood at every sufficiently close distinct coefficient vector. In contrast, negative semidefiniteness alone allows flat directions and is not enough to establish a strict local maximum.

Key takeaway: With a canonical link, the observed Hessian is $-X^{\mathsf T}WX$, where the weights are determined by the fitted natural parameters and dispersion—not by the realized responses. Positive weights plus full column rank give negative-definite curvature and rule out flat coefficient directions. Neither fact alone guarantees existence of a finite MLE.

Common Mistake

Wrong reasoning: “Every $b''(\eta_i)$ is positive, so the likelihood is strictly concave in $\beta$.” This overlooks the design matrix. If $X$ is rank deficient, some nonzero $v$ satisfies $Xv=0$, and then $$ v^{\mathsf T}\nabla_\beta^2\ell(\beta)v = -\sum_i w_i(x_i^{\mathsf T}v)^2 =0. $$ The likelihood is flat along that direction even though every weight is positive. The correction is to require both positive weights and full column rank for negative-definite curvature; for strict concavity on a convex region, these conditions must hold throughout that region.

Check Your Understanding

Use the Hessian and rank conditions to answer these questions.

In the canonical-link Hessian, which quantities determine the weights, and where do the observed responses enter the first derivative?
If every Hessian weight is positive but $X$ has a nonzero null vector, what happens to the likelihood along the corresponding coefficient direction?
Why does full column rank, together with positive weights, make the Hessian negative definite?
Does a negative-definite Hessian at one coefficient vector prove that a finite MLE exists? Explain.
For a design with two identical predictor columns, which coefficient feature can be identified, and which cannot?