Likelihood Covariance Estimation Across Missingness Patterns

Why Covariance Estimation Must Follow the Patterns

When different subjects have different subsets of variables observed, there is no single complete data matrix whose ordinary sample covariance can be used without qualification. A likelihood analysis instead asks what each subject’s observed values imply about the joint model. In a Gaussian model, each distinct observed subset contributes the density of the corresponding marginal Gaussian distribution. The covariance parameters are shared across these contributions, so the resulting score equations generally combine information across patterns rather than reducing to one sample covariance formula.

Assume independent subjects \(X_i\sim N_p(\mu,\Sigma)\), where \(\mu\in\mathbb{R}^p\) and \(\Sigma\) is positive definite. We use the observed-data likelihood under an ignorable missingness mechanism: the missingness mechanism can be omitted from the likelihood for \((\mu,\Sigma)\) under the usual MAR and parameter-distinctness conditions. The focus here is the Gaussian contribution from the observed values.

The likelihood contribution for a subject with pattern \(r\) is the marginal density of \(X_{i,O_r}\), not the density of a completed vector with guessed values inserted. Define \(q_r=|O_r|\) and \(\Sigma_r=\Sigma_{O_rO_r}\). The log-likelihood, up to any omitted missingness-mechanism term, is

Each pattern has its own determinant, inverse, and residual sum of squares. Nevertheless, all the \(\Sigma_r\) are submatrices of one common \(\Sigma\). That shared parameterization is what couples the pattern contributions.

Covariance and Mean Score Equations

For each pattern define the residual sum matrix \(S_r=\sum_{i\in I_r}(x_{i,O_r}-\mu_{O_r})(x_{i,O_r}-\mu_{O_r})^\mathsf{T}\). The covariance contribution depends on the data through this matrix. The following score formula makes the role of each pattern explicit.

To see the equation in full coordinates, let \(E_r\) be the \(p\times q_r\) matrix that selects coordinates in \(O_r\), so \(\Sigma_r=E_r^\mathsf{T}\Sigma E_r\). Put \(A_r=\Sigma_r^{-1}S_r\Sigma_r^{-1}-n_r\Sigma_r^{-1}\). Since \(d\Sigma_r=E_r^\mathsf{T}(d\Sigma)E_r\), the full-coordinate covariance score equation is

This is an equation in the free entries of a symmetric \(\Sigma\). Equivalently, the diagonal score for entry \(\sigma_{jj}\) is half the \(jj\) entry of the displayed sum, and the score for an off-diagonal parameter \(\sigma_{jk}\), \(j<k\), is the \(jk\) entry. The different factors follow from counting both symmetric positions in a trace.

For completeness, applying those identities to the pattern-\(r\) log-likelihood gives \[ d\ell_r=-\frac12 n_r\operatorname{tr}(\Sigma_r^{-1}d\Sigma_r) +\frac12\sum_{i\in I_r}\operatorname{tr}\!\left[ \Sigma_r^{-1}z_i z_i^\mathsf{T}\Sigma_r^{-1}d\Sigma_r\right], \] where \(z_i=x_{i,O_r}-\mu_{O_r}\). Summing over \(i\) yields the stated formula. Substituting \(d\Sigma_r=E_r^\mathsf{T}(d\Sigma)E_r\) and using \(\operatorname{tr}(AB)=\operatorname{tr}(BA)\) gives the embedded full-coordinate equation. This proves the covariance score result. \(\square\)

The mean score follows by differentiating the residual quadratics with respect to \(\mu\):

In general, the mean and covariance equations must be solved jointly. Substituting a pattern-specific sample mean into each pattern’s covariance calculation does not enforce one common \(\mu\), just as calculating separate covariances does not enforce one common \(\Sigma\).

A Solvable Case: Diagonal Covariance

General covariance score equations are nonlinear because each pattern contains a matrix inverse and determinant. A useful check on the patternwise construction is a model where the covariance is constrained to be diagonal. Suppose \(\mu\) is known, and \(\Sigma=\operatorname{diag}(v_1,\ldots,v_p)\) with every \(v_j>0\). Then the observed-data log-likelihood separates by variable, even when patterns contain different subsets.

Indeed, the terms involving \(v_j\) are \[ \ell_j(v_j)=-\frac12\left[m_j\log(2\pi)+m_j\log v_j+\frac{Q_j}{v_j}\right]. \] Differentiating gives \(\ell_j'(v_j)=-m_j/(2v_j)+Q_j/(2v_j^2)\), which vanishes exactly at \(v_j=Q_j/m_j\). For \(v_j<Q_j/m_j\) the derivative is positive, and for \(v_j>Q_j/m_j\) it is negative. Therefore this root is the unique maximum on \(v_j>0\), proving the result. \(\square\)

Numerical Estimation and Common Pitfalls

For an unrestricted covariance matrix, the score equations usually require numerical solution. A practical implementation evaluates every pattern’s marginal covariance submatrix, log determinant, inverse, and residual sum matrix, then adds the embedded score contributions. To keep candidate matrices positive definite, an optimizer can parameterize \(\Sigma=LL^\mathsf{T}\), with \(L\) lower triangular and positive diagonal entries. At a fitted interior solution, the free-parameter score should be near zero, and the observed-data log-likelihood should be checked directly.

A frequent mistake is to estimate each covariance entry using only subjects who have both corresponding variables, then assemble those pairwise estimates into one matrix. Such calculations do not generally solve the likelihood equations: observations of a single variable affect its variance, and that variance appears in inverses and determinants for every pattern containing it. Even pairwise values assembled from sensible samples need not form a positive-definite matrix. The likelihood approach instead estimates one shared positive-definite \(\Sigma\) using all observed marginal contributions.

Another mistake is to treat missing entries as zero or to use a completed data matrix without accounting for how the entries were filled. Neither operation is part of the observed-data likelihood. The pattern-specific density is defined by the actually observed coordinates. If a pattern contains no information about a particular cross-product directly, that does not license inventing a cross-product; the shared covariance structure and the other patterns determine how its parameters enter the likelihood.