Curvature Of The Poisson Regression Log-Likelihood

Why Curvature Matters

The score in Poisson regression describes the local change in log-likelihood as the coefficients move. The Hessian describes how that change itself varies: it records the log-likelihood's curvature in every coefficient direction. This matters for understanding the shape of the objective, for determining whether a direction has genuine curvature or is flat, and for numerical optimization.

For Poisson regression with a log link, the curvature has a particularly useful form: a negative weighted cross-product of the design matrix. The weights are the fitted conditional means, so they are positive at every finite coefficient vector. That positivity is the key to proving concavity and identifying exactly when the curvature is strict.

Log-Likelihood and Hessian

Suppose \(Y_1,\ldots,Y_n\) are independent responses with \(Y_i\sim\operatorname{Poisson}(\mu_i)\), where the fixed design row for observation \(i\) is \(\mathbf{x}_i^\top\), a vector of \(p\) covariates including any intercept. The log-link model is \(\log(\mu_i)=\mathbf{x}_i^\top\boldsymbol{\beta}\), with coefficient vector \(\boldsymbol{\beta}\in\mathbb{R}^p\). Thus \(\mu_i(\boldsymbol{\beta})=\exp(\mathbf{x}_i^\top\boldsymbol{\beta})\), which is positive and finite for every finite \(\boldsymbol{\beta}\).

Let \(\mathbf{X}\) be the \(n\times p\) design matrix with rows \(\mathbf{x}_i^\top\), and let \(\mathbf{y}\) contain the observed counts \(y_i\). For nonnegative integer observations, the log-likelihood, up to no omitted parameter-dependent terms, is

The factorial term does not depend on \(\boldsymbol{\beta}\). Differentiating the remaining terms gives the score \(\nabla\ell(\boldsymbol{\beta})=\sum_i\mathbf{x}_i(y_i-\mu_i(\boldsymbol{\beta}))\). To differentiate again, use \(\nabla\mu_i(\boldsymbol{\beta})=\mu_i(\boldsymbol{\beta})\mathbf{x}_i\). This yields the Hessian.

Each term \(\mathbf{x}_i\mathbf{x}_i^\top\) is a \(p\times p\) matrix, and each weight \(\mu_i(\boldsymbol{\beta})\) is positive. The observed counts do not appear directly in the Hessian. They affect the score, but at a fixed coefficient vector the curvature is determined by the design and the model-implied means.

Concavity and the Rank Condition

Proof. Let \(\mathbf{a}\in\mathbb{R}^p\) be any direction. By the Hessian expression, its quadratic form is

The inequality follows because every \(\mu_i(\boldsymbol{\beta})>0\) and every squared term is nonnegative. Therefore the Hessian is negative semidefinite. To show that this implies concavity here, take any two coefficient vectors \(\boldsymbol{\beta}_0,\boldsymbol{\beta}_1\) and define \(g(t)=\ell(\boldsymbol{\beta}_0+t(\boldsymbol{\beta}_1-\boldsymbol{\beta}_0))\) for \(0\leq t\leq1\). The second derivative is the Hessian quadratic form in direction \(\boldsymbol{\beta}_1-\boldsymbol{\beta}_0\), so \(g''(t)\leq0\). A twice differentiable function on an interval with nonpositive second derivative is concave; hence \(\ell\) is concave on \(\mathbb{R}^p\).

For strict negativity, the displayed quadratic form equals zero precisely when \(\mathbf{x}_i^\top\mathbf{a}=0\) for every \(i\), because all weights are strictly positive. This is equivalent to \(\mathbf{X}\mathbf{a}=\mathbf{0}\). Thus the Hessian is negative definite exactly when the only vector in the null space of \(\mathbf{X}\) is \(\mathbf{0}\), which is exactly full column rank. If \(\mathbf{X}\) has full column rank, then for any two distinct coefficient vectors the direction between them is nonzero and has strictly negative second derivative along the entire joining line. Therefore the log-likelihood is strictly concave. If \(\mathbf{X}\) is rank deficient, choose a nonzero \(\mathbf{a}\) with \(\mathbf{X}\mathbf{a}=\mathbf{0}\). Then \(\mathbf{X}(\boldsymbol{\beta}+t\mathbf{a})=\mathbf{X}\boldsymbol{\beta}\) for every real \(t\); all linear predictors and the log-likelihood remain unchanged along that line. Strict concavity fails. \(\square\)

Worked Curvature Calculations

Interpreting Strict and Non-Strict Curvature

The quadratic form offers a practical interpretation. A direction \(\mathbf{a}\) changes the linear predictor for observation \(i\) by \(\mathbf{x}_i^\top\mathbf{a}\). The curvature in that direction is the negative sum of the squared predictor changes, weighted by the fitted means. A direction that changes at least one predictor has strictly negative curvature; a direction changing none of them has zero curvature.

The weights also explain why the Hessian depends on the current coefficient vector: changing \(\boldsymbol{\beta}\) changes the fitted means and therefore changes how strongly each observation contributes to curvature. But for every finite coefficient vector, each weight remains positive, so this change cannot reverse the sign of curvature. This is why concavity is global, not merely a property at one candidate estimate.

Common Pitfall

• Wrong reasoning: “The Hessian is negative definite because the fitted means are positive.” Positivity of the means makes the weighted cross-product nonnegative in every direction, but it does not guarantee that every nonzero coefficient direction changes a predictor.
• Correction: Positive weights guarantee a negative semidefinite Hessian. Negative definiteness additionally requires \(\mathbf{X}\mathbf{a}\ne\mathbf{0}\) for every nonzero \(\mathbf{a}\), or equivalently full column rank. Duplicate or linearly dependent design columns create flat directions.
• Another common error: putting the observed counts \(y_i\) on the Hessian diagonal. The counts occur in the score, but differentiating the score replaces the fitted means with the weights in the Hessian.