E- and M-Steps for Gaussian Mean and Covariance

From Conditional Moments to Parameter Updates

For Gaussian data with missing entries, an EM iteration does not require choosing one imputed value for each missing entry. It uses the conditional first and second moments of the complete data given what was observed. The first moments update the mean; the second moments also retain uncertainty about missing values, which is essential for updating the covariance.

Assume \(X_1,\ldots,X_n\) are independent \(p\)-dimensional observations from \(N(\mu,\Sigma)\), with unrestricted \(\mu\) and positive-definite \(\Sigma\). Assume the missingness mechanism is ignorable for likelihood inference. At iteration \(t\), the current parameter values are \(\mu^{(t)}\) and \(\Sigma^{(t)}\). For each observation, let \(O_i\) denote its observed coordinates and \(M_i\) its missing coordinates.

For a nonempty observed set and a nonempty missing set, the conditional Gaussian formula gives \(m_{i,M_i}^{(t)}=\mu_{M_i}^{(t)}+\Sigma_{M_iO_i}^{(t)}(\Sigma_{O_iO_i}^{(t)})^{-1}(x_{i,O_i}-\mu_{O_i}^{(t)})\), and the missing-coordinate block of the conditional covariance is \(\Sigma_{M_iM_i}^{(t)}-\Sigma_{M_iO_i}^{(t)}(\Sigma_{O_iO_i}^{(t)})^{-1}\Sigma_{O_iM_i}^{(t)}\). The other blocks of \(V_i^{(t)}\) are zero. If all coordinates are observed, \(m_i^{(t)}=x_i\) and \(V_i^{(t)}=0\). If none are observed, \(m_i^{(t)}=\mu^{(t)}\) and \(V_i^{(t)}=\Sigma^{(t)}\).

The expected complete-data second-moment matrix for observation \(i\) is \(E[X_iX_i'\mid X_{i,O_i}]=V_i^{(t)}+m_i^{(t)}m_i^{(t)\prime}\). Thus the E-step supplies two pooled quantities:

The M-step maximizes the expected complete-data log-likelihood using these moments. All observations contribute, including observations with different patterns; the divisor in the unrestricted Gaussian covariance update is \(n\), not \(n-1\).

Deriving the Unrestricted Gaussian M-Step

Write \(m_i=m_i^{(t)}\) and \(V_i=V_i^{(t)}\) temporarily. Apart from a constant independent of \(\mu,\Sigma\), the expected complete-data log-likelihood is

This expression follows by taking the conditional expectation of the complete-data quadratic form. In particular, the identity \(E[(X_i-\mu)(X_i-\mu)'\mid X_{i,O_i}]=V_i+(m_i-\mu)(m_i-\mu)'\) accounts for uncertainty in the missing entries.

Proof. Set \(A=n^{-1}\sum_i m_i\). For every \(\mu\), expanding around \(A\) gives

The cross terms vanish because \(\sum_i(m_i-A)=0\). At fixed \(\Sigma\), the extra term in \(Q\) when \(\mu\ne A\) is \(-\frac n2(A-\mu)'\Sigma^{-1}(A-\mu)\), which is nonpositive and is zero only at \(\mu=A\). Thus \(A\) uniquely maximizes \(Q\) over the mean. At \(\mu=A\), maximizing \(Q\) over \(\Sigma\) is equivalent to minimizing \(\log|\Sigma|+\operatorname{tr}(\Sigma^{-1}S)\). Let \(B=S^{-1/2}\Sigma S^{-1/2}\), which is positive definite, with eigenvalues \(\lambda_1,\ldots,\lambda_p>0\). The quantity to minimize, minus the constant \(\log|S|\), is \(\sum_j(\log\lambda_j+1/\lambda_j)\). The derivative of each summand is \((\lambda_j-1)/\lambda_j^2\); it is negative for \(0<\lambda_j<1\), zero at \(1\), and positive for \(\lambda_j>1\). Each summand therefore has its unique minimum at \(1\), so \(B=I\) and \(\Sigma=S\). This proves both updates. \(\square\)

If \(S\) is singular, it is not a positive-definite covariance matrix, and the stated interior maximizer does not exist. In that case the unrestricted M-step does not yield a positive-definite covariance maximum. The updates below presume that \(S\) is positive definite at the iteration being computed.

Equivalently, the updates can be written directly in terms of the pooled moments:

The second form follows by expanding \(\sum_i(m_i-\bar m)(m_i-\bar m)'\). It is useful computationally, while the centered form for \(S\) often makes its positive-semidefinite contributions easier to inspect.

Pattern-Specific Contributions

Suppose there are \(R\) distinct observation patterns. Let \(I_r\) be the observations with pattern \(r\), and compute \(m_i^{(t)}\) and \(V_i^{(t)}\) using that pattern’s observed coordinates. The pooled quantities can be assembled pattern by pattern:

For a pattern with many observations, its conditional covariance matrix is the same for every observation in that pattern at a fixed iteration. It still enters the second-moment sum once for each observation in the pattern. The observed values differ across observations, so their conditional means and outer products must still be accumulated individually.

Common Pitfall: Treating Conditional Means as Imputed Data

A tempting shortcut is to replace each missing value by its conditional mean and then calculate an ordinary sample covariance from those completed vectors. That shortcut omits \(V_i\). Conditional means represent the expected values of missing entries, but they do not make those entries known; their remaining conditional uncertainty contributes to the expected squared deviations.

In the two-observation example, if the conditional variances \(7/4\) are omitted, the updated second-coordinate variance would be \(1/16\), rather than \(29/16\). The difference is \(28/16=7/4\), exactly the average conditional variance contributed by the two records. Mean-only imputation therefore understates the covariance in this example.