Observed Information via Louis' Identity

Why Observed Information Can Be Computed Through Latent Data

In a finite mixture model, the likelihood is built from component densities, but the component that generated each observation is unobserved. Directly differentiating the observed-data likelihood is always one route to curvature, but it can become cumbersome when the parameter is a vector or the component model is elaborate. A useful alternative is to calculate derivatives as if the latent component labels were known, then average those derivatives over their conditional distribution given the data.

Louis’ identity gives the exact correction needed for this calculation. The conditional mean of the complete-data score gives the observed-data score. For information, the conditional mean of the complete-data information is not enough: one must subtract the conditional covariance of the complete-data score. In a mixture, the posterior responsibilities supply the probabilities used in both calculations.

Definition: Let $Y$ be observed data and $Z$ latent data, with joint density or mass function $f_c(y,z;\theta)$ and observed-data density $g(y;\theta)=\sum_z f_c(y,z;\theta)$ for a discrete latent variable. The complete-data score and Hessian are $S_c(\theta;y,z)=\nabla_\theta\log f_c(y,z;\theta)$ and $H_c(\theta;y,z)=\nabla_\theta^2\log f_c(y,z;\theta)$. The observed-data information for $y$ is $J_o(\theta;y)=-\nabla_\theta^2\log g(y;\theta)$.

For continuous latent data, replace the sum by an integral with respect to a dominating measure. All derivatives are with respect to the same parameter coordinates. In particular, if a constraint is handled by a transformation, the derivatives and information are in the transformed, unconstrained coordinates.

Louis’ Identity

The identity requires that the observed-data density be differentiable twice in a neighborhood of the parameter value and that differentiation can be passed through the sum or integral defining it. A sufficient condition for an integral is a common support and, in that neighborhood, integrable bounds on the relevant first and second derivatives of the joint density. Finite Gaussian mixtures with positive weights and positive-definite component covariance matrices satisfy the finite-sum part of this condition at interior parameter values.

Theorem (Louis’ identity): Under the stated differentiation conditions, and where $g(y;\theta)>0$, $$ \nabla_\theta\log g(y;\theta) = E_\theta[S_c(\theta;Y,Z)\mid Y=y], $$ and $$ J_o(\theta;y) = E_\theta[-H_c(\theta;Y,Z)\mid Y=y] - \operatorname{Cov}_\theta(S_c(\theta;Y,Z)\mid Y=y). $$

Here the conditional expectation and covariance use the latent-data distribution given the observed value $y$, at the parameter value $\theta$. The covariance is a matrix: its $(a,b)$ entry is the conditional covariance of the $a$th and $b$th score components.

Differentiate the observed density.
Since $\nabla_\theta f_c=f_c S_c$, differentiation under the sum or integral gives $\nabla_\theta g=\sum_z f_c S_c$.

Condition on the observed data.
Divide by $g(y;\theta)$. The ratios $f_c(y,z;\theta)/g(y;\theta)$ are the conditional probabilities of the latent values, yielding the conditional mean score.

Differentiate a second time.
Use $\nabla_\theta^2 f_c=f_c(H_c+S_cS_c^\mathsf{T})$, then subtract the outer product of the observed score from the Hessian of the log density.

For completeness, the second-derivative step gives

\nabla_\theta^2 \log g = \frac{\nabla_\theta^2 g}{g} - \frac{(\nabla_\theta g)(\nabla_\theta g)^\mathsf{T}}{g^2} = E[H_c+S_cS_c^\mathsf{T}\mid y] - E[S_c\mid y]E[S_c\mid y]^\mathsf{T}.

The last two score terms combine to $\operatorname{Cov}(S_c\mid y)$. Negating the Hessian therefore gives Louis’ identity. This also proves the score identity: from $\nabla_\theta g=\sum_z f_cS_c$, division by $g$ is exactly the conditional average of $S_c$. Thus both formulas follow directly from differentiating the observed density, rather than from an approximation. $\square$

Responsibilities in a Finite Mixture

For independent observations $Y_1,\ldots,Y_n$, let $Z_i\in\{1,\ldots,K\}$ denote the component label, let $\pi_k$ be the component weights, and let $f_k(y;\theta)$ be the component densities. The responsibility for observation $i$ and component $k$ is

\tau_{ik} = P_\theta(Z_i=k\mid Y_i=y_i) = \frac{\pi_k f_k(y_i;\theta)} {\sum_{j=1}^K\pi_j f_j(y_i;\theta)}.

Consequently, conditional expectations of complete-data quantities are weighted sums over components. For example, $E[S_{c,i}\mid y_i]=\sum_k\tau_{ik}S_{c,i}(k)$. The conditional covariance is the weighted covariance of those component-specific score vectors, not zero merely because the observed data are fixed. The full observed information is the sum of the individual contributions because the observed likelihood factors over independent observations:

J_o(\theta;y_1,\ldots,y_n) = \sum_{i=1}^n \left\{ \sum_{k=1}^K\tau_{ik}[-H_{c,i}(k)] - \operatorname{Cov}(S_{c,i}\mid y_i) \right\}.

Worked Example: Information for a Mixing Weight

Suppose $Z$ is Bernoulli with $P(Z=1)=\pi$, and the observed $X$ has component distributions $P(X=1\mid Z=1)=0.8$ and $P(X=1\mid Z=0)=0.2$. Treat the component probabilities as known and calculate the observed information about $\pi$ for one observation $x=1$ at $\pi=0.4$.

The complete-data log likelihood, omitting terms that do not depend on $\pi$, is $z\log\pi+(1-z)\log(1-\pi)$. Thus the complete-data score and negative Hessian are

S_c=\frac{z}{\pi}-\frac{1-z}{1-\pi}, \qquad -H_c=\frac{z}{\pi^2}+\frac{1-z}{(1-\pi)^2}.

For $x=1$, the component-weighted probabilities are $0.4(0.8)=0.32$ and $0.6(0.2)=0.12$, so $P(X=1)=0.44$ and $\tau=P(Z=1\mid X=1)=0.32/0.44=8/11$. The conditional mean negative Hessian is

E[-H_c\mid x=1] = \frac{8}{11}\frac{1}{0.4^2} + \frac{3}{11}\frac{1}{0.6^2} = \frac{175}{33}.

The two possible scores are $2.5$ and $-5/3$. A two-point distribution with probabilities $8/11$ and $3/11$ has variance $(8/11)(3/11)(2.5+5/3)^2=1250/363$. Therefore,

J_o(0.4;1) = \frac{175}{33}-\frac{1250}{363} = \frac{225}{121} \approx 1.860.

As a direct check, the observed probability of $X=1$ is $q(\pi)=0.2+0.6\pi$. Its log likelihood is $\log q(\pi)$ for this observation, so its negative second derivative is $0.6^2/q^2$, which at $q=0.44$ is $225/121$, matching Louis’ identity.

Gaussian-Mixture Calculations

For a Gaussian mixture, the latent label selects the component contribution to the complete-data log likelihood. The same identity applies whether the parameter is a component mean, a covariance parameter, a weight, or a vector containing several of these. The complete-data derivatives are often simpler because, conditional on a label, only one component density contributes.

Worked Example: Information for One Component Mean

Consider a two-component normal mixture with equal known weights, unit known variances, and means $\mu_1=0$ and $\mu_2=2$. Treat only $\mu_1$ as unknown. For one observation $x=1$, compute the observed information for $\mu_1$.

If $Z=1$, the complete-data score for $\mu_1$ is $x-\mu_1$, and the negative Hessian is $1$. If $Z=2$, both derivatives are zero. The two component densities at $x=1$ are equal, so the responsibility for component 1 is $\tau=1/2$. Hence $E[-H_c\mid x]=1/2$. The score takes values $1$ and $0$ with equal conditional probabilities, giving conditional variance $1/4$. Louis’ identity yields

J_o(\mu_1;1)=\frac{1}{2}-\frac{1}{4}=\frac{1}{4}.

The subtraction matters: uncertainty about which component generated $x$ reduces the observed-data curvature relative to the conditional mean complete-data curvature. Direct differentiation confirms the result. The mixture density at $x$ is $\tfrac12\phi(x-\mu_1)+\tfrac12\phi(x-\mu_2)$, where $\phi$ is the standard normal density. At the stated values, its first derivative with respect to $\mu_1$ is $\phi(1)/2$, its second derivative is $0$, and its density is $\phi(1)$. Thus the second derivative of its log is $0-(1/2)^2=-1/4$, so the observed information is $1/4$.

Worked Example: Summing Information for a Standard Error

Use the two-component Bernoulli model from the first example, with known component probabilities $0.8$ and $0.2$, and observations $x_1=1,x_2=0$. The observed success probability is $q(\pi)=0.2+0.6\pi$. The likelihood is $q(1-q)$, which is maximized at $q=1/2$; since $q=0.2+0.6\pi$, the maximum occurs at $\hat\pi=0.5$.

At $\hat\pi=0.5$, each observation has component-1 responsibility $0.8$ if $x=1$ and $0.2$ if $x=0$. The complete-data score values are $2$ and $-2$, and the complete-data negative Hessian is $4$ for either label. For either observed value, the conditional mean negative Hessian is therefore $4$. The conditional score variance is $0.8(0.2)(2-(-2))^2=2.56$, so each observation contributes $4-2.56=1.44$ units of observed information. The total is

J_o(\hat\pi;x_1,x_2)=1.44+1.44=2.88.

The local observed-information standard error is the square root of the inverse information:

\widehat{\operatorname{SE}}(\hat\pi) = \sqrt{\frac{1}{2.88}} \approx 0.589.

As a check, the observed log likelihood is $\log q+\log(1-q)$. Its negative second derivative with respect to $\pi$ at $q=1/2$ is $2(0.6)^2/(0.5)^2=2.88$, as obtained from the responsibilities.

Using the Information Matrix Carefully

In a regular likelihood problem, the inverse observed information evaluated at an interior maximum-likelihood estimate estimates the local covariance matrix of the estimator. For a vector parameter, the diagonal entries of $J_o(\hat\theta)^{-1}$ give estimated marginal variances in the chosen parameter coordinates; their square roots are estimated standard errors. The off-diagonal entries retain the estimated dependence among parameter estimates. For independent observations, calculate each Louis contribution using that observation’s responsibilities and sum the matrices before inversion.

This is a local curvature calculation, not a guarantee that a normal approximation is accurate. It presumes an interior, locally identifiable solution and a nonsingular information matrix. Mixture likelihoods can have label symmetry or other irregularities, and solutions near a boundary or a singular configuration require particular caution. In those situations, a computed inverse may be unstable or fail to describe the estimator’s sampling behavior; the identity itself remains an algebraic derivative identity wherever its conditions hold.

Common Mistake

A tempting but incorrect calculation is to average the complete-data information and treat it as the observed information. In the first worked example that would give $175/33\approx5.303$, rather than the correct $225/121\approx1.860$. The error is omitting the conditional score covariance. Complete-data information presumes the latent label is known; when it is not, variation in the complete-data score across plausible labels must be subtracted. Responsibilities determine both the conditional mean information and this covariance correction.

Key takeaway: Louis’ identity expresses observed information as conditional mean complete-data information minus conditional score covariance. In finite mixtures, posterior responsibilities provide the weights for both terms; sum observation-level contributions, then invert the resulting matrix only when a regular local standard-error interpretation is appropriate.

Check Your Understanding

Use the definitions and derivations above to answer the following.

Why is the conditional covariance of the complete-data score subtracted, rather than added, in Louis’ identity?
In a finite mixture, what probability weights are used to average component-specific complete-data scores for an observation?
For the Bernoulli mixture in the first example, what is the responsibility for component 1 when $x=0$ and $\pi=0.4$? Show the two weighted component probabilities.
What information would be reported incorrectly if the conditional score covariance were omitted from the two-observation standard-error example?
Name two conditions under which inverting observed information may not provide a reliable standard-error approximation in a mixture model.