Local Quadratic Geometry of a GLM Log-Likelihood

Local Shape: Slope and Curvature

A GLM fitting algorithm evaluates the log-likelihood at a current coefficient vector and uses its local shape to decide which direction to move. The score describes the first-order change, while the Hessian describes how that slope changes in different directions. Together they determine a quadratic surface that approximates the log-likelihood nearby.

The approximation is local: it describes the likelihood near the current coefficients, not necessarily across the whole parameter space. Its stationary point is therefore a candidate step, not a guarantee that the actual likelihood increases by the amount predicted—or even increases at all.

Definition: Let $\ell(\beta)$ be a twice continuously differentiable log-likelihood for a coefficient vector $\beta\in\mathbb{R}^p$. At a current value $\beta_0$, the score is $U(\beta_0)=\nabla\ell(\beta_0)$, and the Hessian is $H(\beta_0)=\nabla^2\ell(\beta_0)$. For a displacement $\delta\in\mathbb{R}^p$, the second-order Taylor approximation is $$ Q_{\beta_0}(\delta)=\ell(\beta_0)+U(\beta_0)^\mathsf{T}\delta+\frac{1}{2}\delta^\mathsf{T}H(\beta_0)\delta. $$

The constant term sets the height of the approximation at $\beta_0$. The linear term gives its slope there, and the quadratic term gives its curvature. In one dimension, a positive score means the log-likelihood initially rises as the coefficient increases; a negative Hessian means the local quadratic bends downward. In several dimensions, these statements apply along each direction $v$: the slope is $U^\mathsf{T}v$, and the curvature is $v^\mathsf{T}Hv$.

The Approximation for a GLM

Consider independent responses $Y_1,\ldots,Y_n$ from an exponential-dispersion family, with density or mass function $$ f(y_i;\theta_i,\phi)=\exp\left\{\frac{y_i\theta_i-b(\theta_i)}{a(\phi)}+c(y_i,\phi)\right\}. $$ Here $\phi$ is treated as fixed, $E[Y_i]=\mu_i=b'(\theta_i)$, and $\operatorname{Var}(Y_i)=a(\phi)V(\mu_i)$, where $V$ is the variance function. The linear predictor is $\eta_i=x_i^\mathsf{T}\beta+o_i$, with fixed design vector $x_i$ and offset $o_i$, and the link satisfies $g(\mu_i)=\eta_i$. Assume the inverse link is twice differentiable in the region considered and $V(\mu_i)>0$. These conditions make the derivatives below well-defined.

Write $d_i=d\mu_i/d\eta_i$ and $d_i'=d^2\mu_i/d\eta_i^2$. Differentiating the log-likelihood with respect to $\beta$ gives the following score and Hessian:

U(\beta)=\sum_{i=1}^n x_i\frac{(y_i-\mu_i)d_i}{a(\phi)V(\mu_i)}

H(\beta)=\sum_{i=1}^n x_ix_i^\mathsf{T} \left[ -\frac{d_i^2}{a(\phi)V(\mu_i)} +\frac{y_i-\mu_i}{a(\phi)} \left\{ \frac{d_i'}{V(\mu_i)} -\frac{d_i^2V'(\mu_i)}{V(\mu_i)^2} \right\} \right].

To see the derivative behind these formulas, differentiate the $i$th log-likelihood contribution with respect to $\eta_i$. Since $d\theta_i/d\mu_i=1/V(\mu_i)$, its derivative is $(y_i-\mu_i)d_i/[a(\phi)V(\mu_i)]$. Multiplication by $x_i$ gives the contribution to the score. Differentiating that scalar once more with respect to $\eta_i$ gives the bracketed term in the Hessian; the chain rule contributes $x_ix_i^\mathsf{T}$. Thus the observed Hessian generally depends on the observed response through $y_i-\mu_i$.

For the canonical link, $\eta_i=\theta_i$, so $d_i=V(\mu_i)$ and $d_i'=V'(\mu_i)V(\mu_i)$. The response-dependent part in braces cancels, leaving $$ H(\beta)=-\sum_{i=1}^n x_ix_i^\mathsf{T}\frac{V(\mu_i)}{a(\phi)}. $$ This is a useful special case, but the general quadratic approximation is defined by the actual score and Hessian at the current coefficient vector.

Worked Example: The Local Quadratic for a Logistic Intercept

Suppose four independent Bernoulli observations have three successes. Fit an intercept-only logistic model, so each success probability is $p=e^\beta/(1+e^\beta)$. Omitting no parameter-dependent terms, the log-likelihood is $$ \ell(\beta)=3\beta-4\log(1+e^\beta). $$

At $\beta_0=0$, $p=1/2$. Differentiation gives $U(\beta)=3-4p$ and $H(\beta)=-4p(1-p)$. Hence $U(0)=1$ and $H(0)=-1$. For displacement $\delta$, the quadratic approximation is $$ Q_0(\delta)=\ell(0)+\delta-\frac{1}{2}\delta^2. $$ Its slope at zero is 1, and its curvature is $-1$. Setting its derivative $1-\delta$ equal to zero gives the stationary displacement $\delta=1$.

The quadratic predicts a log-likelihood change of $1-\tfrac12=0.5$ at this displacement. The actual change is $$ \ell(1)-\ell(0)=3-4\log(1+e)+4\log 2\approx 0.5195. $$ The prediction is close, but not exact: the likelihood is not itself quadratic over this interval.

Why the Stationary Displacement Has This Form

When the Hessian at $\beta_0$ is nonsingular, differentiate the quadratic approximation with respect to $\delta$. Its gradient is $U(\beta_0)+H(\beta_0)\delta$. The stationary displacement therefore solves $$ H(\beta_0)\delta=-U(\beta_0). $$ This equation defines the local Newton displacement. It describes the stationary point of the quadratic approximation; whether that point is a maximum depends on the curvature.

Theorem: Suppose $\ell$ is twice continuously differentiable in a neighborhood of $\beta_0$. As $\delta\to 0$, $$ \ell(\beta_0+\delta)=\ell(\beta_0)+U(\beta_0)^\mathsf{T}\delta+\frac{1}{2}\delta^\mathsf{T}H(\beta_0)\delta+o(\|\delta\|^2). $$ Thus the score and Hessian give the first- and second-order local geometry of the log-likelihood.

Proof. Define the one-variable function $h(t)=\ell(\beta_0+t\delta)$ for $0\leq t\leq 1$, taking $\delta$ small enough that the line segment lies in the neighborhood. The chain rule gives $h'(0)=U(\beta_0)^\mathsf{T}\delta$ and $h''(t)=\delta^\mathsf{T}H(\beta_0+t\delta)\delta$. The fundamental theorem of calculus, applied twice, gives $$ h(1)=h(0)+h'(0)+\int_0^1(1-t)h''(t)\,dt. $$ Subtracting $\tfrac12\delta^\mathsf{T}H(\beta_0)\delta$ from the integral term leaves $$ \int_0^1(1-t)\delta^\mathsf{T}\{H(\beta_0+t\delta)-H(\beta_0)\}\delta\,dt. $$ By continuity of $H$, the norm of the matrix difference in braces tends uniformly to zero for $0\leq t\leq1$ as $\delta\to0$. The absolute value of the remainder is therefore at most $$ \frac{\|\delta\|^2}{2} \sup_{0\leq t\leq1}\|H(\beta_0+t\delta)-H(\beta_0)\|=o(\|\delta\|^2). $$ This proves the expansion. $\square$

Result: If $H(\beta_0)$ is negative definite and $U(\beta_0)\ne0$, the stationary point of $Q_{\beta_0}$ is a strict maximum and has a higher quadratic value than $Q_{\beta_0}(0)$.

Proof. Negative definiteness makes $H(\beta_0)$ nonsingular, so the stationary displacement $\delta_N=-H(\beta_0)^{-1}U(\beta_0)$ exists. The quadratic has negative definite curvature, so its stationary point is a strict maximum. At that point $U(\beta_0)=-H(\beta_0)\delta_N$, and therefore $$ Q_{\beta_0}(\delta_N)-Q_{\beta_0}(0) =U(\beta_0)^\mathsf{T}\delta_N+\frac12\delta_N^\mathsf{T}H(\beta_0)\delta_N =-\frac12\delta_N^\mathsf{T}H(\beta_0)\delta_N>0. $$ The inequality follows from negative definiteness and $\delta_N\ne0$, which is ensured by $U(\beta_0)\ne0$. This conclusion concerns the quadratic approximation only; the Taylor remainder means it does not by itself establish an increase in the actual log-likelihood. $\square$

Worked Example: Poisson Log-Link Curvature

Take two independent Poisson responses, $y_1=1$ and $y_2=2$, with an intercept-only log-link model. Each mean is $e^\beta$, and, up to a constant independent of $\beta$, $$ \ell(\beta)=3\beta-2e^\beta. $$

At $\beta_0=0$, the score is $3-2e^0=1$, and the Hessian is $-2e^0=-2$. Thus $$ Q_0(\delta)=\ell(0)+\delta-\delta^2. $$ Its derivative $1-2\delta$ vanishes at $\delta=1/2$, where the quadratic predicts a change of $1/2-(1/2)^2=0.25$. The actual change is $$ \ell(1/2)-\ell(0)=\frac32-2(e^{1/2}-1)\approx 0.2026. $$ Both values are positive, but they differ because the exponential likelihood has higher-order curvature that the local quadratic omits.

Worked Example: A Gaussian Case Where the Approximation Is Exact

Let $Y_1=1$ and $Y_2=3$ be independent normal observations with mean $\beta$ and known variance 1. The log-likelihood is $$ \ell(\beta)=-\frac12\{(1-\beta)^2+(3-\beta)^2\}-\log(2\pi). $$

At $\beta_0=1$, the score is $(1-1)+(3-1)=2$, and the Hessian is $-2$. The quadratic approximation is therefore $$ Q_1(\delta)=\ell(1)+2\delta-\delta^2. $$ Because the log-likelihood is already quadratic in $\beta$, this approximation is exact for every $\delta$. Its derivative $2-2\delta$ vanishes at $\delta=1$; the resulting coefficient is $\beta=2$, the sample mean. The log-likelihood change is exactly $2(1)-1^2=1$.

What the Local Geometry Does—and Does Not—Tell You

The score identifies the direction of immediate increase: for a small displacement $t v$, the first-order change is $tU^\mathsf{T}v$. The Hessian then describes how that directional slope changes. If its curvature is negative in every nonzero direction, the quadratic is concave and has a unique stationary maximum. If the Hessian is singular, the stationary equation may have no solution or multiple solutions; if it is not negative definite, the stationary point need not maximize even the quadratic.

A common mistake is to conclude that a negative definite Hessian guarantees that taking the full stationary displacement increases the actual likelihood. It guarantees an increase in the quadratic approximation when the score is nonzero, as proved above. The actual likelihood also includes the Taylor remainder, which can matter for a large displacement. Local curvature is essential for constructing and interpreting a step, but a separate check is needed before treating the step as a confirmed likelihood improvement.

This geometric viewpoint connects the score and Hessian to computation without confusing the fitted model with its approximation. In a GLM, the formulas show where the response, mean, link, and design enter the local shape. The same calculation also makes clear why the local approximation can be exact for one model and only approximate for another.

Check Your Understanding

Use the definitions and examples above to answer the following.

For a one-dimensional log-likelihood with score 2 and Hessian $-4$ at the current value, write its local quadratic in terms of a displacement $\delta$ and find the stationary displacement.
In the general GLM Hessian formula, which term makes the observed curvature depend on the response for a noncanonical link?
What does negative definiteness of the Hessian guarantee about the quadratic approximation, and what does it not guarantee about the actual log-likelihood?
Why is a nonsingular Hessian needed to obtain a unique stationary displacement from the quadratic approximation?
In the Gaussian example, why does the Taylor approximation remain exact even when the displacement is not small?

Local Quadratic Geometry of a GLM Log-Likelihood

What You'll Learn