Observed Information And Expected Information

Why Information Measures Matter

A likelihood can be high at many parameter values if it is nearly flat, or concentrated near one value if it bends sharply. The likelihood’s local curvature therefore says something important about how precisely the data distinguish nearby parameter values. Observed information measures curvature in the realized sample; expected information measures variation in the score under the model. These are related summaries, but they are defined in different ways.

The distinction is useful in nonlinear likelihood models, where the score and Hessian may depend on several parameters and the data. Information matrices help describe the local shape of the likelihood and provide a basis for approximating uncertainty. They do not, by themselves, guarantee that parameters are identified or that an estimator has a particular finite-sample distribution.

Definitions

Let $y$ denote the observed sample and let $\ell_n(\theta;y)$ be its log likelihood, where $\theta$ is a $k$-dimensional parameter vector. The sample score is the gradient of the log likelihood, and the Hessian is the matrix of its second partial derivatives.

Definition: The observed information at $\theta$, for the realized sample $y$, is the negative Hessian of the sample log likelihood: $$ J_n(\theta;y)=-\frac{\partial^2 \ell_n(\theta;y)} {\partial\theta\,\partial\theta'}. $$ It is a $k\times k$ matrix. For a scalar parameter, it is the negative second derivative.

The minus sign makes positive curvature in the information matrix correspond to downward curvature in the log likelihood. Observed information is evaluated at a specified parameter value, often the maximum likelihood estimate, but it can also be evaluated at other values for diagnostics or comparison.

Definition: Write the sample score as $S_n(\theta)=\partial\ell_n(\theta;y)/\partial\theta$, viewed as a random vector when the sample is generated from a model indexed by $\theta$. The expected information from score variation is $$ I_n(\theta)=E_\theta\!\left[S_n(\theta)S_n(\theta)'\right]. $$ This expectation is over samples generated under the model at $\theta$, not over parameter values.

The expected outer product is well-defined when its entries have finite expectations. It is always a second-moment matrix and is positive semidefinite. It is the covariance matrix of the score only when the score has mean zero. In particular, a standard sufficient set of regularity conditions includes an interior parameter value, a common support that does not depend on $\theta$, a differentiable density, finite score second moments, and conditions allowing differentiation under the integral sign. Under those conditions,

E_\theta[S_n(\theta)] =\int \frac{\partial \log f(y;\theta)}{\partial\theta}f(y;\theta)\,dy =\int \frac{\partial f(y;\theta)}{\partial\theta}\,dy =\frac{\partial}{\partial\theta}\int f(y;\theta)\,dy =0.

The interchange of differentiation and integration in this argument requires suitable regularity; it does not follow merely from differentiability of the log likelihood. When the score has mean zero, $I_n(\theta)=\operatorname{Var}_\theta(S_n(\theta))$. If this condition is not established, the expected outer product remains defined, but calling it the score covariance is not justified.

Observed Information as Local Curvature

For a small change $h$ in the parameter vector, a twice-differentiable log likelihood has the local expansion

\ell_n(\theta+h;y) = \ell_n(\theta;y)+S_n(\theta)'h -\frac{1}{2}h'J_n(\theta;y)h +o(\|h\|^2).

The quadratic term shows how observed information summarizes curvature in different directions. If $J_n$ is positive definite at an interior maximum, the log likelihood bends downward in every nonzero direction there. A small eigenvalue indicates a direction with relatively little curvature; a zero eigenvalue indicates no second-order curvature in that direction. An indefinite matrix away from a maximum is possible and does not make the definition invalid.

Worked Example: Observed Information for a Normal Mean

Suppose three hypothetical observations are $8$, $10$, and $12$ units. Model them as independent draws from $N(\mu,4)$, so the variance is known to be $4$ square units and the unknown parameter is the mean $\mu$, measured in units.

Check of the model setup. The calculation assumes independent observations with the stated normal distribution and a known, parameter-independent variance. The normal density has common support on the real line and is differentiable in $\mu$.

Ignoring no parameter-dependent terms, the sample log likelihood is $$ \ell_3(\mu) =-\frac{3}{2}\log(8\pi)-\frac{1}{8}\sum_{i=1}^3(x_i-\mu)^2. $$

Its first derivative is $S_3(\mu)=\sum_{i=1}^3(x_i-\mu)/4$, and its second derivative is $-3/4$. Thus the observed information is $$ J_3(\mu)=-\frac{\partial^2\ell_3(\mu)}{\partial\mu^2} =\frac{3}{4}=0.7500\ \text{per square unit}. $$

The sample mean is $\hat{\mu}=(8+10+12)/3=10$ units, where the score is zero. At that value, the observed information remains $0.7500$ per square unit. The log likelihood therefore has the local quadratic term $-\tfrac12(0.7500)h^2$ for a small change $h$ in the mean, measured in units. The constant curvature here follows from the known-variance normal model; it need not be constant in other models.

Expected Information from Score Variation

Expected information concerns how much the score would vary across samples generated at the stated parameter value. For independent observations with individual scores $S_i(\theta)$, the sample score is their sum. If each score has mean zero and the observations are independent, cross-products between distinct observations have expectation zero, so the expected information is the sum of the individual expected score outer products. The independence and zero-mean conditions matter for this simplification.

Worked Example: Expected Information in a Bernoulli Model

Let $Y_1,\ldots,Y_5$ be independent Bernoulli outcomes, each equal to one with probability $p$. Use the scalar parameter $\theta=\log\{p/(1-p)\}$, so $p=e^\theta/(1+e^\theta)$. Evaluate expected information at the hypothetical value $p=0.4$, equivalently $\theta=\log(2/3)$.

Check of the model setup. Each outcome has support $\{0,1\}$, which does not depend on $\theta$. The Bernoulli probability is differentiable for finite $\theta$, and the score has finite second moment. The finite support permits differentiating the probability sum term by term, so the expected score is zero.

For one outcome, the log likelihood is $y\log p+(1-y)\log(1-p)$. Differentiating with respect to $\theta$ gives the score $y-p$. Therefore, at $p=0.4$, its two possible values are $0.6$ when $y=1$ and $-0.4$ when $y=0$. The expected score is $$ 0.4(0.6)+0.6(-0.4)=0. $$

The expected squared score for one observation is $$ E_\theta[(Y-p)^2] =0.4(0.6)^2+0.6(-0.4)^2 =0.144+0.096 =0.2400. $$

By independence and zero expected score, the expected outer product for the five-observation score is the sum of the five individual expectations: $$ I_5(\theta)=5(0.2400)=1.200. $$

Thus the expected score variation is $1.200$ in the squared-log-odds parameter scale. This is an expectation over possible samples at $p=0.4$; it is not the squared score of one particular observed sample.

Information Matrices in a Nonlinear Response Model

With multiple parameters, information describes curvature and score variation jointly. Off-diagonal entries represent cross-products between parameter score components, so they matter when the likelihood’s local shape is tilted rather than aligned with the coordinate axes.

Worked Example: Expected Information for a Logistic Response

Consider three hypothetical binary outcomes observed at fixed covariate values $x=(0,1,2)$, with outcomes $y=(1,0,1)$. Let $$ P(Y_i=1\mid x_i)=p_i,\qquad \operatorname{logit}(p_i)=\alpha+\beta x_i. $$ Evaluate information at $\alpha=0$ and $\beta=\log 2$. The parameter vector is $\theta=(\alpha,\beta)'$.

Check of the model setup. Treat the covariates as fixed and the outcomes as conditionally independent Bernoulli draws under the stated probabilities. Their support is $\{0,1\}$ at each covariate value, independent of the parameters, and each probability is differentiable at the evaluation point.

The linear predictors are $0,\log 2,\log 4$, giving probabilities $p_1=0.5$, $p_2=2/3$, and $p_3=0.8$. The individual score is $(y_i-p_i)z_i$, where $z_i=(1,x_i)'$. The expected outer product for an observation is $p_i(1-p_i)z_i z_i'$: this follows because $E(Y_i-p_i)=0$ and $E[(Y_i-p_i)^2]=p_i(1-p_i)$.

Summing over the three fixed covariate values gives $$ I_3(\theta) =\sum_{i=1}^3 p_i(1-p_i) \begin{pmatrix}1\\x_i\end{pmatrix} \begin{pmatrix}1&x_i\end{pmatrix} = \begin{pmatrix} 0.2500+0.2222+0.1600 & 0+0.2222+0.3200\\ 0+0.2222+0.3200 & 0+0.2222+0.6400 \end{pmatrix} = \begin{pmatrix} 0.6322&0.5422\\ 0.5422&0.8622 \end{pmatrix}. $$

The determinant is approximately $0.2511$, so this matrix is positive definite and invertible. Its inverse is approximately $$ I_3(\theta)^{-1} \approx \begin{pmatrix} 3.433&-2.159\\ -2.159&2.518 \end{pmatrix}. $$

This inverse describes a local uncertainty scale under the model at the specified parameter and covariates; it is not an exact finite-sample covariance statement. The negative off-diagonal entry indicates that the two parameter directions interact in the local approximation. In this particular logistic model, the negative Hessian at the stated parameter has the same weighted-design form as the displayed matrix, regardless of the realized outcomes. That is a feature of this calculation, not a general license to equate observed and expected information in every model.

Interpreting Information Carefully

When an information matrix is positive definite, greater curvature generally corresponds to less local uncertainty in the associated parameter direction. Under suitable conditions, an inverse information matrix is used as a local or large-sample approximation to estimator covariance. This interpretation depends on the model, regularity, and local identification; the matrix alone does not establish those conditions. Information is also parameterization-dependent: changing the scale or representation of a parameter changes the numerical curvature and its units.

Observed information is calculated from the realized log likelihood and can vary across samples. Expected information averages score variation over the model’s possible samples at a fixed parameter value. They may be close in a particular application, but their definitions do not make them interchangeable. Expected information can be singular when the score does not vary in some parameter direction, signaling a lack of local information in that direction.

Common Mistake

Wrong reasoning: “The log likelihood is differentiable, so the expected score is automatically zero; therefore the expected score outer product is the score covariance.” Differentiability alone does not justify moving a derivative inside an integral. If that interchange is invalid, the usual normalization argument for a zero expected score may fail.

Correction: State regularity conditions that justify differentiating the density’s integral, or separately assume the expected score is zero. Only then is $E_\theta[S_nS_n']$ equal to $\operatorname{Var}_\theta(S_n)$. Also keep the observed negative Hessian distinct from the expected score outer product unless a result establishing their equality is available under the relevant assumptions.

Key takeaway: Observed information is the negative sample log-likelihood Hessian; expected information here is the model expectation of the score outer product. Both summarize local information, but they arise from different calculations and require careful attention to the evaluation point and regularity conditions.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

For a scalar parameter, what sign and derivative define observed information?
Why is the expected score outer product not automatically the score covariance?
In the Bernoulli example, what is the one-observation score when $Y=0$ at $p=0.4$, and what is its probability?
What does a zero eigenvalue of an information matrix indicate about local curvature?
Why is an inverse expected information matrix not automatically an exact finite-sample covariance matrix?