Observed Information and Missing-Information Variance

Why Missing Values Change Information

A likelihood based on incomplete observations is obtained by integrating out the unobserved values. This integration changes more than the likelihood’s numerical value: it changes its curvature, and therefore the amount of information the data provide about the parameters. Even when an expectation-maximization (EM) algorithm uses complete-data calculations at each iteration, uncertainty about the missing values cannot be treated as if those values were observed.

The key distinction is between the curvature of the complete-data log-likelihood and the curvature of the observed-data log-likelihood. The difference is determined by how much the complete-data score varies after conditioning on what was actually observed. This gives a practical identity for calculating observed information and a precise account of how missing values affect uncertainty.

Definition: Let $Z=(Y,U)$ denote complete data, where $Y$ is observed and $U$ is unobserved. Let $g_\theta(y,u)$ be the complete-data density or mass function, and let $p_\theta(y)=\int g_\theta(y,u)\,du$ be the observed-data density, with a sum replacing the integral for discrete $U$. Write $S_c(\theta;Y,U)=\nabla_\theta\log g_\theta(Y,U)$ for the complete-data score and $H_c(\theta;Y,U)=\nabla_\theta^2\log g_\theta(Y,U)$ for its Hessian. The observed information at observed data $Y=y$ is $J_o(\theta;y)=-\nabla_\theta^2\log p_\theta(y)$.

These definitions assume that the likelihood contribution for the observed data is the appropriate one. Under an ignorable missingness mechanism, this is the marginal likelihood for the observed values. If the missingness mechanism is not ignorable, its contribution must also be included in the likelihood; the same calculations then apply to that full model.

Louis’ Identity

The conditional distribution of $U$ given $Y=y$ and parameter $\theta$ has density $g_\theta(y,u)/p_\theta(y)$. Denote conditional expectation and covariance under this distribution by $E_\theta[\cdot\mid Y=y]$ and $\operatorname{Cov}_\theta(\cdot\mid Y=y)$.

Theorem (Louis’ identity): Suppose $p_\theta(y)>0$, the complete-data density has a common support that does not depend on $\theta$, and the derivatives needed below can be passed under the integral or sum defining $p_\theta(y)$. Assume the relevant conditional expectations and covariances are finite. Then $$ \nabla_\theta\log p_\theta(y) = E_\theta[S_c\mid Y=y], \qquad J_o(\theta;y) = E_\theta[-H_c\mid Y=y] - \operatorname{Cov}_\theta(S_c\mid Y=y). $$

The first equation says that the observed score is the conditional expectation of the complete-data score. The second says that observed information is the conditional expectation of complete-data information minus the conditional covariance of the complete-data score. That covariance is called the missing information at the observed data and parameter value.

Differentiate the observed likelihood.
Express the derivative of the integral over missing values using the complete-data score.

Differentiate the observed score again.
The second derivative includes both the complete-data Hessian and a score-product term.

Separate the score covariance.
Subtracting the outer product of the observed score converts the score-product term into a conditional covariance.

Here is the full derivation. Since $\nabla_\theta g_\theta=g_\theta S_c$, differentiation under the integral gives $$ \nabla_\theta p_\theta(y) = \int g_\theta(y,u)S_c(\theta;y,u)\,du. $$ Dividing by $p_\theta(y)$ yields the first identity. For the second derivative, use $\nabla_\theta^2g_\theta=g_\theta(H_c+S_cS_c^{\mathsf T})$, where ${\mathsf T}$ denotes transpose. Thus $$ \nabla_\theta^2p_\theta(y) = \int g_\theta(y,u) \{H_c+S_cS_c^{\mathsf T}\}\,du. $$ The Hessian of the log of a positive function is its Hessian divided by the function, minus the outer product of its gradient divided by the function. Therefore $$ \nabla_\theta^2\log p_\theta(y) = E_\theta[H_c+S_cS_c^{\mathsf T}\mid Y=y] - E_\theta[S_c\mid Y=y]E_\theta[S_c\mid Y=y]^{\mathsf T}. $$ The two score-product terms combine to form $\operatorname{Cov}_\theta(S_c\mid Y=y)$. Negating this Hessian proves Louis’ identity. $\square$

Expected Information and the Missing-Information Principle

The observed information $J_o(\theta;Y)$ is a random matrix before the data are observed. Its expectation is the observed Fisher information, provided the usual score regularity conditions hold. The complete-data Fisher information is $I_{\mathrm{com}}(\theta)=E_\theta[-H_c]$. Taking expectations in Louis’ identity gives the information decomposition below.

Theorem (missing-information decomposition): Under the conditions for Louis’ identity and the usual regularity conditions that make the score mean zero and identify Fisher information with the expected negative Hessian, $$ I_{\mathrm{obs}}(\theta) = I_{\mathrm{com}}(\theta) - E_\theta\!\left[ \operatorname{Cov}_\theta(S_c\mid Y) \right]. $$ The expected conditional covariance is positive semidefinite. Consequently, observed data have no more Fisher information than the complete data in any parameter direction.

To verify the decomposition, take expectations of Louis’ identity over $Y$. The expectation of $E_\theta[-H_c\mid Y]$ is $E_\theta[-H_c]=I_{\mathrm{com}}(\theta)$, by iterated expectation. The expected observed information is $I_{\mathrm{obs}}(\theta)$, so rearrangement gives the displayed equation. For any vector $a$, $$ a^{\mathsf T}\operatorname{Cov}_\theta(S_c\mid Y)a = \operatorname{Var}_\theta(a^{\mathsf T}S_c\mid Y)\geq 0. $$ Taking expectations preserves nonnegativity, proving that the subtracted matrix is positive semidefinite. $\square$

This is a statement about information, not a claim that a particular realized observed-information matrix must be smaller in every entry than its complete-data counterpart. The matrix comparison is guaranteed for the expected information. For a regular model with an identifiable parameter and nonsingular observed information, the inverse observed information at a maximum-likelihood estimate is commonly used as an approximate covariance matrix: $\widehat{\operatorname{Var}}(\hat\theta)\approx J_o(\hat\theta;Y)^{-1}$. This is an approximation to sampling uncertainty, not an exact variance identity in general. If the information is singular, this inverse is unavailable, and some parameter directions may not be estimable from the observed data.

Worked Example: A Normal Mean With Some Values Missing

Suppose $Z_1,\ldots,Z_n$ are independent $N(\theta,\sigma^2)$ variables, where $\sigma^2$ is known. Exactly $r$ values are observed and $m=n-r$ are missing. Find the observed and missing information for $\theta$.

For one complete value, the score is $(Z_i-\theta)/\sigma^2$ and the negative second derivative is $1/\sigma^2$. Therefore the complete-data score for all $n$ values has conditional variance $m/\sigma^2$: the $r$ observed score terms are fixed given the data, while the $m$ independent missing score terms each have variance $1/\sigma^2$. The conditional expectation of the complete-data information is $n/\sigma^2$. Louis’ identity gives $$ J_o(\theta;Y) = \frac{n}{\sigma^2}-\frac{m}{\sigma^2} = \frac{r}{\sigma^2}. $$ This also follows directly from the observed likelihood, which is the product of the $r$ observed normal densities. Thus the missing values contribute $m/\sigma^2$ units of missing information, and the observed information comes only from the $r$ observed values.

Worked Gaussian Calculation With a Vector Parameter

With a vector parameter, the missing-information term is a matrix. Its off-diagonal entries can matter: uncertainty in a missing value can affect how well different components of the parameter are distinguished.

Worked Example: One Observed Coordinate of a Bivariate Normal

Let $X=(X_1,X_2)^{\mathsf T}\sim N_2(\mu,\Sigma)$, where $\mu=(\mu_1,\mu_2)^{\mathsf T}$ is unknown and $\Sigma=\begin{pmatrix}4&2\\2&3\end{pmatrix}$ is known. Observe $X_1=x_1$, but not $X_2$. Calculate the observed information for $\mu$ using Louis’ identity.

First, $$ \Sigma^{-1} = \frac{1}{8} \begin{pmatrix}3&-2\\-2&4\end{pmatrix} = \begin{pmatrix}3/8&-1/4\\-1/4&1/2\end{pmatrix}. $$ The complete-data score for $\mu$ is $S_c=\Sigma^{-1}(X-\mu)$, and the complete-data information is $\Sigma^{-1}$. Since $X_2$ is missing, the conditional variance of $X$ given $X_1$ is zero in its first coordinate and $2$ in its second. Indeed, the conditional variance of $X_2$ is $3-2^2/4=2$. Therefore $$ \operatorname{Cov}(S_c\mid X_1=x_1) = 2 \begin{pmatrix}-1/4\\1/2\end{pmatrix} \begin{pmatrix}-1/4&1/2\end{pmatrix} = \begin{pmatrix}1/8&-1/4\\-1/4&1/2\end{pmatrix}. $$ Subtracting this from the complete-data information gives $$ J_o(\mu;x_1) = \begin{pmatrix}3/8&-1/4\\-1/4&1/2\end{pmatrix} - \begin{pmatrix}1/8&-1/4\\-1/4&1/2\end{pmatrix} = \begin{pmatrix}1/4&0\\0&0\end{pmatrix}. $$ The result agrees with the marginal model $X_1\sim N(\mu_1,4)$: one observation gives information $1/4$ about $\mu_1$ and none about $\mu_2$. In particular, the complete-data information about $\mu_2$ is entirely removed by the conditional score uncertainty.

From Information to an Uncertainty Estimate

A useful numerical calculation makes clear why the missing-information correction matters. Consider five independent $N(\theta,4)$ measurements with three observed values $1,2,4$ and two missing values. Treat the pattern of observed and missing entries as fixed and the variance $4$ as known.

Worked Example: Observed Curvature and the Standard Error

The observed-data log-likelihood, apart from no omitted parameter-dependent terms, is $$ \ell_o(\theta) = -\frac{3}{2}\log(8\pi) -\frac{(1-\theta)^2+(2-\theta)^2+(4-\theta)^2}{8}. $$ Its score is $\{(1-\theta)+(2-\theta)+(4-\theta)\}/4$. Setting this equal to zero gives $$ \hat\theta=\frac{1+2+4}{3}=\frac{7}{3}. $$ The second derivative is $-3/4$, so the observed information at the estimate is $J_o(\hat\theta)=3/4$. Its inverse gives the variance estimate $4/3$, and the corresponding standard error is $\sqrt{4/3}\approx1.155$.

The same information follows from Louis’ identity. The complete-data information for five measurements is $5/4$. Given the three observed values, the two missing score terms are independent, each with variance $1/4$, so the missing-information term is $2(1/4)=1/2$. Hence $5/4-1/2=3/4$, as required. In this model, the number of observed measurements is fixed, and the estimator is their mean; its sampling variance is exactly $4/3$. Treating both missing values as if they were observed would instead use information $5/4$ and variance $4/5$, understating uncertainty.

Common Pitfall: Counting Imputed Values as Observations

A common mistake is to fill in missing values with conditional means and then calculate information as if those filled-in values were known. The reasoning is that the conditional means are the best predictions of the missing values, so they can stand in for actual observations. The correction is that a prediction is not an observation: it does not remove the conditional variability of the complete-data score. Louis’ identity explicitly subtracts this variability. A plug-in calculation that omits the subtraction generally overstates information and understates uncertainty.

The same distinction matters in EM calculations. The E-step evaluates conditional expectations given the observed data and current parameter values; those expectations support the likelihood maximization in the M-step. They do not make the missing values certain. For inference, retain the conditional score covariance when computing observed information. Also distinguish the realized observed information $J_o(\hat\theta;Y)$ from expected Fisher information: they are different quantities, even though both describe curvature or information and can be used in uncertainty calculations under appropriate conditions.

Key takeaway: Observed information equals the conditional expected complete-data information minus the conditional covariance of the complete-data score. The covariance term measures information lost because the missing values remain uncertain; omitting it can materially understate parameter uncertainty.

Check Your Understanding

Use the identities and conditions developed above to answer these questions.

In a scalar model, what does the conditional variance of the complete-data score represent in Louis’ identity?
Why must the complete-data score covariance be subtracted, rather than added, when calculating observed information?
For $n$ independent $N(\theta,\sigma^2)$ values with $r$ observed and known $\sigma^2$, what are the observed and missing information?
In the bivariate example, why is the observed information for $\mu_2$ zero even though the complete-data information matrix has a positive $(2,2)$ entry?
Why does replacing missing values by conditional means and treating them as observed generally understate uncertainty?