Information Equality and Sandwich Covariance

Why Likelihood Curvature May Not Describe Uncertainty

A likelihood fit summarizes the data using a specified model. If the data-generating distribution is not in that model, the fitted parameter typically converges to the parameter that maximizes expected log-likelihood, not to a true parameter of the fitted family. The curvature of that expected log-likelihood still describes how sharply the criterion is maximized. But the actual variation of the score across observations need not match that curvature.

The distinction matters for standard errors. In a correctly specified regular likelihood, expected curvature and score variance are equal, giving the familiar inverse-information covariance. Under misspecification they can differ. The asymptotic covariance then combines both matrices in a sandwich: curvature on the outside, score variance in the middle.

Definitions and the Information Equality

Let \(Z_1,\ldots,Z_n\) be independent and identically distributed observations from a distribution \(P_0\), which need not belong to the fitted model. For one observation, write the fitted log-likelihood contribution as \(q(Z;\theta)\), where \(\theta\in\mathbb{R}^p\). Define its score by \(s(Z;\theta)=\nabla_\theta q(Z;\theta)\), and its Hessian by \(H(Z;\theta)=\nabla_\theta^2q(Z;\theta)\). The expected log-likelihood is \(M(\theta)=E_0[q(Z;\theta)]\).

The expectation of the score is not automatically the derivative of the expected log-likelihood. To equate them, differentiation must be allowed to pass under the expectation. One sufficient condition is that \(q(Z;\theta)\) is differentiable near \(\theta^*\), and that each score component is dominated there by an integrable random variable. Under such a condition, differentiation under the integral or expectation gives \(\nabla M(\theta^*)=E_0[s(Z;\theta^*)]\). Since \(\theta^*\) is an interior maximizer, \(\nabla M(\theta^*)=0\). Thus the score is centered, and \(B=E_0[s(Z;\theta^*)s(Z;\theta^*)^T]\).

Under correct specification, the expected curvature and score variance agree, subject to regularity conditions. To see the source of the equality, let \(f_\theta(z)\) be a density or mass function relative to a fixed dominating measure, with support not depending on \(\theta\). Suppose it is twice differentiable, its first and second derivatives can be passed under the integral, and the resulting score and Hessian expectations are finite. Since \(\int f_\theta(z)\,dz=1\), differentiation once gives \(E_\theta[s(Z;\theta)]=0\). Also,

Integrating the identity, the left side is \(\nabla_\theta^2 1=0\), by the assumed interchange. Therefore \(E_\theta[H]+E_\theta[ss^T]=0\). The score has mean zero, so its second moment is its covariance, and consequently \(A=B\). This is the information equality. Under misspecification, the expectation is taken under \(P_0\), not the fitted density \(f_\theta\); the normalization argument for \(f_\theta\) no longer implies \(A=B\).

Deriving the Sandwich Limit

The next result states the usual independent-observation form. Its assumptions separate the statistical argument from the technical conditions that justify it.

The centering condition in assumption (1) is important: an interior maximum of the expected log-likelihood makes its derivative zero, but only valid differentiation under expectation identifies that derivative with the expected score. The derivation uses the multivariate central limit theorem for the centered scores and a Taylor expansion of the sample score.

The inverse-curvature approximation \(A^{-1}/n\) is therefore not generally the covariance of \(\hat\theta\). It is correct when \(B=A\), as under the information equality, but otherwise it omits the variability of the score under the actual distribution. In practice, the matrices are often estimated by

With an interior maximum, the sample scores sum to zero, so the displayed outer-product estimate is also the empirical covariance of the fitted scores using divisor \(n\). Under the theorem's regularity conditions and corresponding laws of large numbers, these estimates converge to \(A\) and \(B\). The factor \(1/n\) is essential: \(A\) and \(B\) are per-observation matrices.

Worked Examples

Interpretation and Common Pitfall

The matrices have distinct roles. \(A\) measures how quickly the expected fitted log-likelihood declines away from its pseudo-true maximizer; \(B\) measures how much the individual fitted scores vary under the actual data distribution. A steep criterion does not by itself guarantee a precise estimator if the scores are highly variable, and a shallow criterion does not by itself imply large actual variance if the scores vary little.

• Common mistake: using inverse curvature as the covariance under misspecification. The reasoning “the estimator maximizes a likelihood, so its covariance is inverse information” silently assumes information equality. Under misspecification, the curvature \(A\) and score variance \(B\) need not agree. Use \(A^{-1}BA^{-T}/n\), and use \(A^{-1}/n\) only when the equality is justified.
• Common mistake: declaring the expected score zero without checking differentiation. An interior maximum gives \(\nabla M(\theta^*)=0\). It gives \(E_0[s(Z;\theta^*)]=0\) only when differentiation under expectation is valid. A suitable integrable dominating function for the score near \(\theta^*\) is one standard sufficient condition.
• Common mistake: mixing total and per-observation information. The theorem uses \(A\) and \(B\) for one-observation quantities. The covariance of the estimator includes the additional factor \(1/n\).

In GLM work, the sandwich covariance provides asymptotic uncertainty for the likelihood estimator even when the specified response distribution is not exactly correct, provided the regularity conditions for the estimator and sandwich limit hold. It does not repair bias from an unsuitable mean structure, dependence between observations, or failure of the required regularity conditions; those require separate analysis.