Expected, Observed, and Empirical Information

Three Quantities Called Information

In likelihood inference, an information matrix describes how much a likelihood can tell us about its parameter. There are three related quantities to distinguish. Expected Fisher information averages over possible data; observed information and the empirical outer product of scores are both calculated from the realized data, using curvature and score products, respectively. The latter two can differ substantially in a finite sample, even when all three have the same large-sample limit under appropriate conditions.

Consider independent observations $X_1,\ldots,X_n$ with densities or mass functions $f_i(x_i;\theta)$, with respect to fixed dominating measures. Let $\theta$ be a $p$-dimensional parameter in an open parameter space, and write the log-likelihood as $\ell(\theta)=\sum_{i=1}^n \ell_i(\theta)$, where $\ell_i(\theta)=\log f_i(X_i;\theta)$. The score contribution from observation $i$ is the column vector $s_i(\theta)=\partial\ell_i(\theta)/\partial\theta$; its Hessian is the matrix of second derivatives of $\ell_i$.

Definition: The expected Fisher information, observed information, and empirical outer product of scores are, respectively, $$ I_n(\theta)=\sum_{i=1}^n E_\theta[s_i(\theta)s_i(\theta)^\mathsf{T}],\qquad J_n(\theta)=-\frac{\partial^2\ell(\theta)}{\partial\theta\,\partial\theta^\mathsf{T}},\qquad K_n(\theta)=\sum_{i=1}^n s_i(\theta)s_i(\theta)^\mathsf{T}. $$ The expectation in $I_n(\theta)$ is over the model's possible data at parameter $\theta$. In contrast, $J_n(\theta)$ and $K_n(\theta)$ use the observed data. Their plug-in versions evaluate the matrices at an estimate such as the maximum-likelihood estimator $\hat\theta$.

The outer product $s_i s_i^\mathsf{T}$ is a $p$-by-$p$ matrix. The word “empirical” here means that the score products are summed over the realized observations; it does not mean that $K_n$ is an average over hypothetical samples. Some conventions divide $K_n$ by $n$ and call that the empirical information per observation. We use the sum convention above, so comparisons on a per-observation scale use $K_n/n$, $J_n/n$, and $I_n/n$.

Calculating the Matrices

For a generalized linear model, the score and Hessian contributions follow by differentiating each observation's log-likelihood. Expected information then requires averaging the score product under the specified model, whereas the observed matrix and empirical score product use the actual recorded responses. The examples show that the distinctions are visible even in simple models.

Worked Example: Bernoulli Logistic Intercept

Let $Y_i$ be independent Bernoulli observations with success probability $p=\{1+\exp(-\theta)\}^{-1}$, where $\theta$ is the logit intercept. Suppose $n=5$, the realized data contain one success and four failures, and we evaluate information at a fixed parameter with $p=0.4$. The score for one observation is $s_i(\theta)=Y_i-p$, because the derivative of the Bernoulli log-likelihood is $Y_i-p$.

For one observation, $E_\theta[(Y_i-p)^2]=\operatorname{Var}_\theta(Y_i)=p(1-p)=0.4(0.6)=0.24$. Therefore the expected information for all five observations is

I_5(\theta)=5(0.24)=1.20.

The second derivative of each log-likelihood contribution is $-p(1-p)$, independent of the observed $Y_i$. Thus the observed information is

J_5(\theta)=5p(1-p)=5(0.24)=1.20.

The realized empirical outer product is instead the sum of the five squared scores. One score is $1-0.4=0.6$, and four are $0-0.4=-0.4$, so

K_5(\theta)=(0.6)^2+4(-0.4)^2=0.36+0.64=1.00.

At this parameter value, expected and observed information happen to match, but the empirical outer product does not. The value $1.00$ is calculated from the realized responses, not by averaging over possible samples.

Worked Example: Poisson Log-Mean

Suppose $Y_1,Y_2,Y_3$ are independent Poisson variables with common mean $\lambda=\exp(\theta)$. Evaluate the three quantities at $\lambda=2$, for observed counts $0,1,4$. The log-likelihood contribution, omitting a term independent of $\theta$, is $\ell_i(\theta)=Y_i\theta-\exp(\theta)$. Hence $s_i(\theta)=Y_i-\lambda$, and the second derivative is $-\lambda$.

Since a Poisson variable has variance $\lambda$, the expected squared score for one observation is $E_\theta[(Y_i-\lambda)^2]=\lambda=2$. Summing gives $I_3(\theta)=3(2)=6$. The observed information is also $J_3(\theta)=3\lambda=6$, because each negative second derivative equals 2 for every possible count.

For the empirical outer product, the observed scores are $-2,-1,2$. Therefore

K_3(\theta)=(-2)^2+(-1)^2+2^2=4+1+4=9.

The data happen to produce score products summing to 9, while the expected and observed information are both 6. All three are information calculations, but only the first averages over possible data.

Worked Example: Normal Mean With Known Variance

Let $Y_1,Y_2,Y_3$ be independent $N(\mu,4)$ observations, with variance 4 known. Evaluate at $\mu=2$ for observed values $1,2,4$. The score contribution is $s_i(\mu)=(Y_i-\mu)/4$, and the second derivative is $-1/4$. Thus each observation contributes expected information $1/4$, and the total expected information is $I_3(\mu)=3/4=0.75$. The observed information is also $J_3(\mu)=3/4=0.75$.

The observed scores are $-1/4,0,1/2$, so the empirical outer product is

K_3(\mu)=\left(-\frac14\right)^2+0^2+\left(\frac12\right)^2 =\frac1{16}+\frac4{16}=\frac5{16}=0.3125.

This is another finite-sample difference: the observed curvature is fixed by the known variance and sample size, but the score products vary with the realized values.

Why the Expected Matrices Match Under Regularity

The expected score outer product and expected negative Hessian agree when the likelihood is sufficiently regular. This identity explains why expected information can be defined in either way in a regular, correctly specified model. The required conditions matter: differentiating under the integral without justification can invalidate the argument.

Theorem: Suppose one observation has density or mass function $f(x;\theta)$ with respect to a measure not depending on $\theta$, where $\theta$ lies in an open set. Assume $f$ is twice differentiable in $\theta$, and that, in a neighborhood of the parameter value considered, differentiation can be interchanged with integration for the first and second derivatives. Assume also that the score and the entries of the Hessian and score outer product have finite expectations. Then $$ E_\theta[s(\theta)]=0,\qquad E_\theta[s(\theta)s(\theta)^\mathsf{T}] =-E_\theta\!\left[\frac{\partial^2\ell(\theta)} {\partial\theta\,\partial\theta^\mathsf{T}}\right]. $$

Proof. Since $f$ is a density or mass function, its integral (or sum) over the sample space is 1. Differentiating this equality with respect to $\theta$, as permitted by the assumptions, gives

0=\int \frac{\partial f(x;\theta)}{\partial\theta}\,dx =\int f(x;\theta)s(x;\theta)\,dx =E_\theta[s(\theta)].

For the second derivative, use $\partial f/\partial\theta=f s$ and differentiate again. The product rule gives the matrix identity

\frac{\partial^2 f(x;\theta)}{\partial\theta\,\partial\theta^\mathsf{T}} =f(x;\theta)\left[ \frac{\partial^2\ell(x;\theta)}{\partial\theta\,\partial\theta^\mathsf{T}} +s(x;\theta)s(x;\theta)^\mathsf{T} \right].

The integral of the left side is the second derivative of 1, hence is the zero matrix. Integrating the right side therefore yields the expected Hessian plus the expected score outer product equal to zero. Rearranging proves the stated identity. $\square$

For independent observations, applying the identity to each contribution and summing gives $I_n(\theta)=-E_\theta[\partial^2\ell(\theta)/\partial\theta\,\partial\theta^\mathsf{T}]$. The theorem equates expectations; it does not assert that $J_n(\theta)=K_n(\theta)$ for each realized data set. The Poisson example illustrates the distinction: the expected score product is 2 per observation, but the observed score product for a particular count need not be 2.

When the Realized Matrices Agree Asymptotically

To compare the random matrices with expected information, it is useful to divide each by sample size. A simple setting is an independent, identically distributed, correctly specified model with parameter $\theta_0$. Let $s_i(\theta_0)$ be the individual scores. Suppose the regularity conditions above hold, the entries of $s_i(\theta_0)s_i(\theta_0)^\mathsf{T}$ and the individual Hessian have finite first moments, and the ordinary law of large numbers applies. Then, entry by entry,

\frac{K_n(\theta_0)}{n}\xrightarrow{p} E_{\theta_0}[s_1(\theta_0)s_1(\theta_0)^\mathsf{T}], \qquad \frac{J_n(\theta_0)}{n}\xrightarrow{p} -E_{\theta_0}\!\left[ \frac{\partial^2\ell_1(\theta_0)} {\partial\theta\,\partial\theta^\mathsf{T}} \right].

The identity just proved makes the two limits equal to the per-observation expected information. This is a direct consequence of applying the law of large numbers to the score products and Hessians separately.

Result: In an independent, identically distributed, regular, correctly specified model with finite first moments of the relevant matrix entries, $K_n(\theta_0)/n$ and $J_n(\theta_0)/n$ converge in probability to the same matrix as $I_n(\theta_0)/n$. If evaluated at a consistent estimator $\hat\theta$ instead, the same conclusion additionally requires appropriate continuity and uniform laws of large numbers in a neighborhood of $\theta_0$.

For fixed-design GLMs, observations need not be identically distributed. The analogous conclusion holds when the design and model satisfy conditions ensuring that the normalized sums of score products and Hessians converge to the same limiting matrix. Thus “asymptotically agree” means convergence after suitable normalization, not exact equality at every sample size. It also presumes correct specification and regularity; without those conditions, the common-limit argument need not apply.

Interpretation and a Common Pitfall

Expected information is a model-based quantity: at a specified parameter value, it describes average curvature or average score variation across repeated samples. Observed information describes the curvature in the particular likelihood produced by the data. The empirical outer product records the variation represented by the particular observed score contributions. In regular settings, either realized matrix is often used as a plug-in estimate of information for uncertainty calculations, but their finite-sample values can differ, as the examples show.

Pitfall: treating a zero total score as a zero score-product matrix. At an interior maximum-likelihood estimate, the total score satisfies $\sum_i s_i(\hat\theta)=0$. It does not follow that $\sum_i s_i(\hat\theta)s_i(\hat\theta)^\mathsf{T}=0$. The latter is a sum of outer products, not the outer product of the sum.
Pitfall: calling observed information an expectation. $J_n(\theta)$ uses the Hessian evaluated on the realized data. The expected negative Hessian is a different quantity, even though its value agrees with expected Fisher information under the theorem's assumptions.
Pitfall: expecting exact finite-sample equality. The identity equates expected quantities. The convergence result equates normalized large-sample limits. Neither says that observed information and empirical score products must match for the data in hand.

Choose the quantity you need.
Use expected information for a model average; use observed information for realized likelihood curvature; use the empirical outer product for realized score products.

Compute at a specified parameter.
Differentiate each log-likelihood contribution for its score and Hessian, and take an expectation only when calculating expected information.

Check the comparison being made.
Finite-sample matrices may differ. A common asymptotic limit requires regularity, correct specification, moment conditions, and a suitable large-sample regime.

Key takeaway: Expected Fisher information averages over possible data; observed information and the empirical outer product of scores are both computed from realized data, one from curvature and the other from score products. Regularity can make their normalized values converge to the same limit without making them identical in a finite sample.

Check Your Understanding

Use the definitions and conditions above to answer the following questions.

Which of the three information matrices averages over possible data, and which two are calculated from realized data?
In the Bernoulli example, why do expected and observed information equal 1.20 while the empirical outer product is 1.00?
What regularity step is needed to derive the expected negative-Hessian identity from the integral of a density?
Why does a zero total score at an interior maximum not imply that the empirical outer product is zero?
What does asymptotic agreement of normalized observed and empirical information mean, and why does it not imply finite-sample equality?