Expected And Observed Information In Efficiency Calculations

Why Two Information Quantities?

Likelihood curvature measures how sharply a likelihood changes near a parameter value. But there are two related quantities to distinguish: expected Fisher information, which averages over possible samples under the model, and observed information, which is calculated from the sample actually obtained. They may be close in a large sample, but they are generally not the same number for a particular data set.

The distinction matters in first-order efficiency calculations. Under standard regularity conditions, expected information determines the asymptotic variance bound for regular estimators. Observed information is often useful as a data-based estimate of that bound, but the realized curvature is not itself the definition of the bound. In what follows, \(I(\theta)\) denotes information per observation and \(I_n(\theta)\) denotes expected information for a sample of size \(n\).

Definitions and the Information Identity

Suppose \(X_1,\ldots,X_n\) are independent and identically distributed with density or mass function \(f(x;\theta)\), where \(\theta\) is a scalar parameter in an open subset of the real line. Write the log-likelihood as \(\ell_n(\theta)=\sum_{i=1}^n \log f(X_i;\theta)\). The score is the derivative of the log-likelihood, and observed information is its negative second derivative.

The expectation in \(I(\theta)\) is taken over the distribution of \(X\) at the specified value of \(\theta\). In contrast, \(J_n(\theta)\) depends on the observed values \(X_1,\ldots,X_n\). For a vector parameter, the score is a column vector and the definitions become \(I(\theta)=E_\theta[u(X;\theta)u(X;\theta)^\mathsf{T}]\) and \(J_n(\theta)=-\nabla^2\ell_n(\theta)\); these are matrices.

The following identity is central to the distinction. It requires regularity; it should not be assumed merely because a likelihood can be differentiated. For a discrete model, sums replace the integrals below.

Proof. Since the density integrates to one, the assumed interchange of differentiation and integration gives \(\int f(x;\theta)\,dx=1\), \(\int \partial f(x;\theta)/\partial\theta\,dx=0\), and \(\int \partial^2 f(x;\theta)/\partial\theta^2\,dx=0\). Where \(f\) is positive, the score is \(u=(\partial f/\partial\theta)/f\). Therefore \[ E_\theta[u(X;\theta)] =\int \frac{\partial f(x;\theta)}{\partial\theta}\,dx=0. \] Also, differentiating the logarithm twice gives \[ \frac{\partial^2}{\partial\theta^2}\log f =\frac{\partial^2 f/\partial\theta^2}{f} -\left(\frac{\partial f/\partial\theta}{f}\right)^2. \] Multiplying by \(f\), integrating, and using \(\int \partial^2 f/\partial\theta^2\,dx=0\) yields \[ E_\theta\left[\frac{\partial^2}{\partial\theta^2}\log f(X;\theta)\right] =-\int \frac{(\partial f(x;\theta)/\partial\theta)^2}{f(x;\theta)}\,dx =-E_\theta[u(X;\theta)^2]. \] All integrals are well-defined under the stated conditions. This proves the identity. \(\square\)

For independent observations, additivity gives \(J_n(\theta)=\sum_{i=1}^n -\partial^2\log f(X_i;\theta)/\partial\theta^2\), and hence \(E_\theta[J_n(\theta)]=nI(\theta)\). The identity equates an expectation of curvature with a variance-like score quantity; it does not say that the curvature for each realized observation equals its expectation.

Worked Examples: Realized Information Can Differ

Information in First-Order Efficiency Calculations

Expected information, rather than the observed curvature of one sample, defines the usual first-order efficiency benchmark in a regular correctly specified likelihood model. In the scalar case, suppose the true parameter \(\theta_0\) is an interior point, \(0<I(\theta_0)<\infty\), and the usual differentiability, identifiability, and likelihood regularity conditions for maximum likelihood asymptotic normality hold. Then a consistent maximum likelihood estimator satisfies \[ \sqrt{n}(\hat\theta-\theta_0)\xrightarrow{d}N\left(0,I(\theta_0)^{-1}\right). \] The asymptotic variance of \(\hat\theta\) on the \(1/\sqrt n\) scale is therefore \(I(\theta_0)^{-1}\), and its approximate variance at sample size \(n\) is \(1/[nI(\theta_0)]\). For regular estimators in the same model, this is the corresponding scalar information-bound benchmark when the bound applies.

This statement concerns a repeated-sampling limit at the true parameter. It is not a claim that the variance of an estimator in every finite sample equals the inverse observed information, nor does it cover irregular situations in which the information bound fails. In a vector model, replace \(I(\theta_0)^{-1}\) by the inverse information matrix, assuming it is nonsingular; parameter-specific bounds may then be obtained from that matrix.

Proof. At the fixed true value, \(J_n(\theta_0)/n=n^{-1}\sum_{i=1}^n j(X_i;\theta_0)\). The summands are iid with finite expectation, so the law of large numbers gives convergence in probability to \(E_{\theta_0}[j(X;\theta_0)]\). By the information identity this expectation is \(I(\theta_0)\). For evaluation at \(\hat\theta\), pointwise convergence at \(\theta_0\) alone is insufficient because the evaluation point is random. The additional uniform law of large numbers controls the empirical average uniformly in a neighborhood of \(\theta_0\); consistency places \(\hat\theta\) in that neighborhood with probability tending to one, and continuity makes the limiting expectation at \(\hat\theta\) converge to its value at \(\theta_0\). Thus the plug-in convergence follows under the stated additional conditions. \(\square\)

The practical role of observed information follows: \(J_n(\hat\theta)^{-1}\), when well-defined, is commonly used to estimate the asymptotic covariance matrix of a maximum likelihood estimator. It is useful because it is calculable from the fitted model and the observed data. Its justification is consistency toward the expected information under appropriate conditions, not equality in every sample. Expected information is the theoretical target for a first-order efficiency comparison; observed information is a sample-dependent estimate of that target.

Common Mistake: Substituting Realized Curvature for the Bound

A common error is to calculate \(J_n(\hat\theta)\) for one data set and call its inverse the exact efficiency bound, then compare another estimator's finite-sample variance directly with that number. This mixes three different objects: an asymptotic bound defined through expected information, a random estimate of that information, and an estimator's finite-sample variance. They need not be equal.

The correction is to state which quantity is being used and why. For a theoretical first-order calculation, use \(I(\theta_0)\) (or \(nI(\theta_0)\) for the full sample) to obtain the asymptotic variance benchmark. For an estimated standard error, one may use observed information at a consistent fitted value, while recognizing that it is random and only asymptotically justified under regularity. If observed and expected information differ materially in a small sample, that is not an algebraic inconsistency: it may indicate ordinary sample variation, and it can also prompt a check of the model and calculation.