Expected And Observed Information

Two Curvatures of a Likelihood

A likelihood describes how compatible different parameter values are with the data that were observed. Its curvature can be summarized in two related ways. Observed information is the negative curvature for the particular data set in hand. Expected Fisher information averages that curvature over data sets generated by the model at a specified parameter value. They answer different questions: one concerns the realized likelihood, and the other concerns the model's average behavior before observing data.

This distinction matters in pharmacometrics and design. A design is chosen before its future observations are known, so an expected quantity is often relevant for planning. After data are collected, the realized likelihood may be sharper or flatter than its expected curvature. Neither quantity is automatically a better description for every purpose; the distinction depends on the question and on the model assumptions.

Definition: Let \(y\) denote data with likelihood \(L(\boldsymbol{\theta};y)\), and let \(\ell(\boldsymbol{\theta};y)=\log L(\boldsymbol{\theta};y)\). The observed information matrix is \(J(\boldsymbol{\theta};y)=-\nabla_{\boldsymbol{\theta}}^2\ell(\boldsymbol{\theta};y)\), the negative Hessian of the log-likelihood for the observed data. The expected Fisher information matrix is \(I(\boldsymbol{\theta})=E_{\boldsymbol{\theta}}[J(\boldsymbol{\theta};Y)]\), where the expectation is over data \(Y\) generated under the model at \(\boldsymbol{\theta}\). When the regularity conditions below hold, it also equals \(E_{\boldsymbol{\theta}}[\mathbf{s}(\boldsymbol{\theta};Y)\mathbf{s}(\boldsymbol{\theta};Y)^\mathsf{T}]\), where \(\mathbf{s}=\nabla_{\boldsymbol{\theta}}\ell\) is the score.

For a scalar parameter \(\theta\), the matrices reduce to numbers: \(J(\theta;y)=-\partial^2\ell/\partial\theta^2\) and \(I(\theta)=E_\theta[J(\theta;Y)]\). Both describe information on the same parameter scale, but the observed quantity depends on the particular data. Expected information depends on the model and parameter value, not on which realization happened to occur.

Conditions and the Information Identity

The equality between expected negative curvature and expected squared score is not automatic. For a density or probability mass function \(f(y;\boldsymbol{\theta})\), assume the model is correctly specified, the parameter is in the interior of its parameter space, the log-likelihood has the required derivatives, and differentiation can be interchanged with integration or summation. The support of the distribution should not change with the parameter, and the relevant expectations must exist. These regularity conditions allow the following derivation.

The expected score is zero.
Since \(f\) is normalized, differentiating \(\int f(y;\boldsymbol{\theta})\,dy=1\) gives \(E_{\boldsymbol{\theta}}[\mathbf{s}]=0\), because \(\nabla f=f\nabla\log f=f\mathbf{s}\).

Differentiate the zero-score identity.
For score components \(s_j\) and parameter component \(\theta_k\), differentiating \(E_{\boldsymbol{\theta}}[s_j]=0\) gives \(E_{\boldsymbol{\theta}}[\partial s_j/\partial\theta_k+s_js_k]=0\). The second term appears because the density inside the expectation also depends on \(\theta_k\).

Identify the Hessian.
Since \(\partial s_j/\partial\theta_k\) is the \(j,k\) entry of the log-likelihood Hessian, rearranging yields \(E_{\boldsymbol{\theta}}[-\nabla^2\ell]=E_{\boldsymbol{\theta}}[\mathbf{s}\mathbf{s}^\mathsf{T}]\).

This identity says that, under the stated conditions, the average negative curvature equals the average score outer product. It does not say that the observed negative Hessian equals the score outer product for each particular data set. Nor does it say that observed and expected information must be numerically equal in a finite sample.

For independent observations, the log-likelihood is a sum of individual log-likelihood contributions. Its score and Hessian are therefore sums as well. Consequently, expected information adds across independent observations under the model. The observed information also adds across those observations, but its realized value varies with the data.

Worked Example: Bernoulli Data at a Prespecified Probability

Suppose, hypothetically, that \(n=10\) independent outcomes follow a Bernoulli model with success probability \(p\), and that the observed data contain \(x=6\) successes. Calculate observed and expected information at the prespecified value \(p=0.4\). The assumptions are independent Bernoulli trials with a common probability, and \(p=0.4\) is interior to \(0<p<1\).

Ignoring the binomial coefficient, which does not depend on \(p\), the log-likelihood is \(\ell(p)=x\log p+(n-x)\log(1-p)\). Its first and second derivatives are

\frac{\partial\ell}{\partial p}=\frac{x}{p}-\frac{n-x}{1-p}, \qquad \frac{\partial^2\ell}{\partial p^2}=-\frac{x}{p^2}-\frac{n-x}{(1-p)^2}.

Thus the observed information is \(J(p;x)=x/p^2+(n-x)/(1-p)^2\). Substituting the data and \(p=0.4\),

J(0.4;6)=\frac{6}{0.4^2}+\frac{4}{0.6^2} =37.5+11.11=48.61.

The count \(X\) has \(E_p[X]=np\). Taking the expectation of the observed-information expression gives

I(p)=\frac{np}{p^2}+\frac{n(1-p)}{(1-p)^2} =\frac{n}{p(1-p)},\qquad I(0.4)=\frac{10}{0.4(0.6)}=41.67.

The observed value, 48.61 inverse probability-squared units, exceeds the expected value, 41.67 inverse probability-squared units. This is a realized difference, not a contradiction: expected information averages over possible success counts, whereas this data set has a particular count.

Why Finite-Sample Values Can Differ

Expected information is an average over repeated data sets under a specified model and parameter value. Observed information is one draw from the corresponding random curvature. An individual data set need not display the model-average curvature, even when the model is correct. The gap can be noticeable when the sample is small or when the likelihood's curvature depends strongly on the observations.

The Bernoulli example also shows why it is important to say where the information is evaluated. At the maximum-likelihood estimate \(\hat p=x/n\), the observed and expected expressions happen to agree for this model:

J(\hat p;x) =\frac{x}{(x/n)^2}+\frac{n-x}{(1-x/n)^2} =\frac{n}{\hat p(1-\hat p)} =I(\hat p),

provided \(0<x<n\). The equality follows by substituting \(x=n\hat p\) and \(n-x=n(1-\hat p)\). It is a feature of this likelihood and evaluation at its estimate, not a general rule that observed and expected information are equal at maximum likelihood estimates.

Worked Example: Poisson Data Show a Different Realization

Consider hypothetical independent counts \(1,0,2,3,1\), each modeled as Poisson with common mean \(\lambda\) per observation interval. There are \(n=5\) observations and their sum is 7. Assume the Poisson model is appropriate and \(\lambda>0\). Calculate both information quantities at the prespecified value \(\lambda=2\).

The log-likelihood, up to terms not involving \(\lambda\), is \(\ell(\lambda)=\left(\sum_i y_i\right)\log\lambda-n\lambda\). Therefore, \(J(\lambda;y)=\left(\sum_i y_i\right)/\lambda^2\). For these data,

J(2;y)=\frac{7}{2^2}=\frac{7}{4}=1.750.

Under the model, \(E_\lambda[\sum_iY_i]=n\lambda\). Hence \(I(\lambda)=n/\lambda\), and at \(\lambda=2\),

I(2)=\frac{5}{2}=2.500.

The observed curvature is lower than the model expectation for this realization. The two values have units of inverse rate-squared, where the rate is the expected count per observation interval. Here the observed total of 7 is below its model expectation of \(5(2)=10\), and the observed-information formula depends directly on that total.

For this example, the maximum-likelihood estimate is \(\hat\lambda=7/5=1.4\). At that estimate, \(J(\hat\lambda;y)=7/1.4^2=3.571\) and \(I(\hat\lambda)=5/1.4=3.571\). As in the Bernoulli example, they coincide at the estimate for this particular model; this coincidence should not be generalized to all likelihoods.

A Case Where They Agree for Every Data Set

Worked Example: Normal Data With Known Variance

Suppose, hypothetically, three independent measurements are \(1.8\), \(2.4\), and \(2.1\) response units, modeled as \(Y_i\sim N(\mu,1.5^2)\), with known standard deviation \(1.5\) response units and unknown mean \(\mu\). Assume the observations are independent and the normal model and known variance are appropriate. The log-likelihood terms involving \(\mu\) are \(-\sum_i(y_i-\mu)^2/(2(1.5)^2)\).

Differentiating twice gives \(\partial^2\ell/\partial\mu^2=-n/(1.5)^2\). The observed information is consequently

J(\mu;y)=\frac{3}{1.5^2}=\frac{3}{2.25}=1.333 \quad\text{inverse response-units squared}.

This expression contains no data values, so averaging it over the normal model leaves it unchanged: \(I(\mu)=3/1.5^2=1.333\) inverse response-units squared. In this known-variance normal model, observed and expected information agree for every sample, not just at an estimate. That result reflects the likelihood's constant curvature in \(\mu\), rather than a universal property of information.

Choosing and Checking an Information Quantity

Expected information is often used when the data have not yet been observed and the aim is to summarize what the assumed model predicts on average at a specified parameter value. Observed information describes the local curvature of the likelihood for realized data. It can be useful for understanding how strongly the observed likelihood concentrates near a parameter value. These interpretations require care: information is curvature on a chosen parameter scale, and a numerical value should not be interpreted without its units and evaluation point.

Several checks help reveal when a calculation or interpretation may be going wrong:

Check the evaluation point. State the parameter value used for each calculation. Comparing \(J(\hat\theta;y)\) with \(I(\theta_0)\) without noting that they are evaluated at different values mixes two sources of difference.
Check the data dependence. Expected information requires averaging over the model distribution. Replacing random counts by observed counts without taking an expectation gives observed, not expected, information.
Check regularity. At parameter boundaries, derivatives or expectations may fail to behave as assumed. A parameter-dependent support can also invalidate the differentiation argument. In such cases, the information identity needs separate justification.
Check curvature in multiple dimensions. Expected information is positive semidefinite because it is an expected score outer product under regularity conditions. Observed information need not be positive semidefinite at every point; a negative eigenvalue can indicate that the log-likelihood is locally curved upward in some direction rather than having a local maximum there.
Check model adequacy. The expected information is an expectation under the specified model. If the data-generating process differs from that model, its expected value under the true process need not satisfy the model-based information identity.

A common mistake is to say, “The observed information is larger than expected, so the model is wrong.” That conclusion does not follow from one finite sample. Observed information is random and can fall above or below its expectation even under a correctly specified model. A discrepancy is a reason to check calculations, assumptions, and data behavior; it is not by itself a test of model correctness.

The practical distinction is therefore not “real” versus “theoretical” information. Both are defined through the likelihood. Expected information describes its average curvature under repeated sampling from the model, while observed information describes the curvature produced by the data actually obtained. Selecting one should follow the inferential or design question, and conclusions should remain conditional on the model and its regularity conditions.

Check Your Understanding

Answer each question, then compare with the solutions below.

In one sentence, distinguish observed information from expected Fisher information.
For \(n=8\) Bernoulli trials with \(x=3\) successes, calculate \(J(0.5;x)\) and \(I(0.5)\).
Why does the expected negative Hessian equal the expected score outer product only under regularity conditions?
In the normal model with known variance, what feature of the likelihood makes observed and expected information equal for all samples?
A colleague sees that observed information is below expected information in one small data set and concludes that the model is misspecified. What is the error in that reasoning?

Solutions. (1) Observed information is the negative Hessian for the realized data, whereas expected information averages that Hessian over data generated at a specified parameter value. (2) \(J(0.5;3)=3/0.5^2+5/0.5^2=12+20=32\); \(I(0.5)=8/[0.5(0.5)]=32\), both in inverse probability-squared units. (3) The derivation differentiates expectations and uses normalization of the model density; the interchange of differentiation and integration or summation, among other steps, requires suitable regularity conditions. (4) With known variance, the log-likelihood's second derivative with respect to the mean is the constant \(-n/\sigma^2\), independent of the observations, so averaging does not change the negative curvature. (5) Observed information varies randomly from sample to sample. A single value below its model expectation can occur under a correct model and does not by itself establish misspecification.