Logistic Information and Likelihood Curvature

Why Likelihood Curvature Matters

In binary logistic regression, the likelihood records how well a coefficient vector agrees with the observed outcomes. Its slope indicates a direction in which the likelihood increases; its curvature describes how quickly that increase slows near a maximum. That curvature provides a direct measure of how much information the data contain about the coefficients and, under suitable conditions, an approximation to their sampling uncertainty.

The key calculation is especially useful in logistic regression: the negative Hessian of the log-likelihood has the form of a weighted cross-product of the design matrix. Each weight depends on a fitted probability, and is largest when that probability is near one half. We derive this form, establish the concavity it implies, and work through numerical information matrices.

Score, Hessian, and Fisher Information

Suppose \(Y_1,\ldots,Y_n\) are independent Bernoulli responses, with fixed predictor vectors \(\mathbf{x}_i\in\mathbb{R}^q\). Write the coefficient vector as \(\boldsymbol{\beta}\), the linear predictor as \(\eta_i=\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}\), and the success probability as \(p_i=\{1+\exp(-\eta_i)\}^{-1}\). Let \(X\) be the \(n\) by \(q\) matrix whose \(i\)th row is \(\mathbf{x}_i^\mathsf{T}\). The log-likelihood for the observed values \(y_i\in\{0,1\}\), omitting no terms that depend on the coefficients, is

Differentiating with respect to \(\boldsymbol{\beta}\) gives the score vector. Differentiating again gives the Hessian matrix, the matrix of second partial derivatives.

To derive the result, first note that \(d p_i/d\eta_i=p_i(1-p_i)\). The derivative of a single log-likelihood contribution with respect to \(\eta_i\) is \(y_i-p_i\). Since \(d\eta_i/d\boldsymbol{\beta}=\mathbf{x}_i\), summing the individual gradients gives

Differentiating this score once more, the derivative of \(p_i\) with respect to \(\boldsymbol{\beta}\) is \(p_i(1-p_i)\mathbf{x}_i\). Therefore

This Hessian does not depend on the observed \(y_i\) once \(\boldsymbol{\beta}\) is fixed. Consequently its negative equals its own expectation, giving \(\mathcal{I}(\boldsymbol{\beta})=X^\mathsf{T}WX\). The same expression also follows from the covariance of the score: the responses are independent, \(E[Y_i]=p_i\), and \(\operatorname{Var}(Y_i)=p_i(1-p_i)\), so

Thus, in this model, observed information and expected information coincide at every finite coefficient vector. This equality is a property of the logistic likelihood; it does not mean information is constant as the coefficients change, because the \(p_i\), and hence the diagonal entries of \(W\), change with \(\boldsymbol{\beta}\).

Concavity and What It Does Not Guarantee

The information matrix is positive semidefinite. For any vector \(\mathbf{v}\in\mathbb{R}^q\),

Because the Hessian is the negative of this matrix, the log-likelihood is concave over the coefficient space. For finite coefficients and finite predictor values, every \(p_i(1-p_i)\) is strictly positive. The Hessian is then negative definite at every coefficient vector precisely when \(X\) has full column rank. If \(X\) is rank deficient, there is a nonzero vector \(\mathbf{v}\) with \(X\mathbf{v}=0\); moving the coefficients along that direction does not change any linear predictor or the likelihood.

The rank condition gives a statement about curvature, not a guarantee that the data yield a finite maximum. For example, in a separable data set, coefficients can move farther in a direction that improves the likelihood while the fitted probabilities approach zero or one. The likelihood can approach its supremum without attaining it at any finite coefficient vector. In that situation, strict concavity at finite coefficient vectors does not produce a finite maximum.

From Information to a Covariance Approximation

Near a finite maximum-likelihood estimate \(\hat{\boldsymbol{\beta}}\), the score is zero. A second-order Taylor expansion of the log-likelihood around that estimate gives, for a nearby coefficient vector \(\boldsymbol{\beta}\),

The quadratic term describes the local curvature. A direction with large curvature is one in which a small coefficient change lowers the log-likelihood sharply; a direction with little curvature allows more variation without a comparable likelihood decrease. When the model is correctly specified and standard large-sample maximum-likelihood conditions hold, including a finite true coefficient vector, fixed dimension, full-rank design, and increasing information, this local quadratic approximation motivates the familiar covariance approximation

The approximation is not an exact finite-sample covariance identity. It uses curvature at the fitted coefficients as a measure of sampling uncertainty. It also requires an invertible information matrix. If the design is rank deficient, the inverse does not exist; if some fitted probabilities are very close to zero or one, their weights are small and the information may be weak in relevant directions.

Common Mistake: Treating Inverse Information as Exact

A tempting but incorrect argument is: “The inverse Hessian is the covariance matrix, so it is exact whenever the Hessian can be inverted.” The Hessian describes curvature of the observed log-likelihood at a particular coefficient vector; it does not by itself equal the finite-sample sampling covariance of the estimator. The correction is to call its inverse a local or large-sample covariance approximation, and to check that the fitted information is invertible and the model’s regularity conditions are plausible. A highly curved likelihood may still be a poor normal approximation in a small or unusual sample.