Conditional Gaussian Moments for Missing Entries

Why Conditional Gaussian Moments Matter

When some components of a multivariate observation are missing, the observed components still constrain what the unobserved components could plausibly have been. Under a multivariate Gaussian model, that constraint has a precise form: the missing components have a conditional Gaussian distribution whose mean adjusts linearly with the observed values and whose covariance is a Schur complement of the original covariance matrix.

The formulas are useful whenever a complete-data Gaussian model must be related to incomplete observations. The calculation is carried out separately for each response pattern, because different patterns have different observed and missing blocks. The focus here is the conditional distribution under the Gaussian model, not an algorithm for fitting its parameters.

Block Notation and the Conditional Distribution

Let $\mathbf X\sim N_p(\boldsymbol\mu,\boldsymbol\Sigma)$, where $\boldsymbol\Sigma$ is positive definite. For a fixed observation pattern, let $O$ be the set of observed component indices and $M$ the set of missing component indices. Write the components in each set in a fixed order, and partition the vector and covariance matrix accordingly:

\mathbf X= \begin{pmatrix}\mathbf X_O\\ \mathbf X_M\end{pmatrix}, \qquad \boldsymbol\mu= \begin{pmatrix}\boldsymbol\mu_O\\ \boldsymbol\mu_M\end{pmatrix}, \qquad \boldsymbol\Sigma= \begin{pmatrix} \boldsymbol\Sigma_{OO}&\boldsymbol\Sigma_{OM}\\ \boldsymbol\Sigma_{MO}&\boldsymbol\Sigma_{MM} \end{pmatrix}.

The covariance blocks satisfy $\boldsymbol\Sigma_{OM}=\boldsymbol\Sigma_{MO}^{\mathsf T}$. Positive definiteness of $\boldsymbol\Sigma$ implies that $\boldsymbol\Sigma_{OO}$ is invertible. Define the Schur complement of $\boldsymbol\Sigma_{OO}$ in $\boldsymbol\Sigma$ by

\boldsymbol S_{M\mid O} = \boldsymbol\Sigma_{MM} -\boldsymbol\Sigma_{MO}\boldsymbol\Sigma_{OO}^{-1} \boldsymbol\Sigma_{OM}.

Theorem: For any fixed observed value $\mathbf X_O=\mathbf x_O$, the conditional distribution of the missing components is $$ \mathbf X_M\mid \mathbf X_O=\mathbf x_O \sim N\!\left( \boldsymbol\mu_M+ \boldsymbol\Sigma_{MO}\boldsymbol\Sigma_{OO}^{-1} (\mathbf x_O-\boldsymbol\mu_O), \ \boldsymbol S_{M\mid O} \right). $$ Thus the conditional mean depends on the observed value, while the conditional covariance depends on the model covariance and the response pattern, not on the particular observed value.

The expression $\boldsymbol\Sigma_{MO}\boldsymbol\Sigma_{OO}^{-1}$ describes how deviations of the observed components from their means shift the expected missing components. The shift is zero when $\mathbf x_O=\boldsymbol\mu_O$. The conditional covariance subtracts from the marginal covariance of the missing block the part explained linearly by the observed block.

Derivation by Completing the Square

Here is a direct derivation. Set $\mathbf z=\mathbf x_O-\boldsymbol\mu_O$, $\mathbf y=\mathbf x_M-\boldsymbol\mu_M$, and abbreviate $\boldsymbol A=\boldsymbol\Sigma_{OO}$, $\boldsymbol B=\boldsymbol\Sigma_{MO}$, and $\boldsymbol C=\boldsymbol\Sigma_{MM}$. Then $\boldsymbol S=\boldsymbol C-\boldsymbol B\boldsymbol A^{-1}\boldsymbol B^{\mathsf T}$. Both $\boldsymbol A$ and $\boldsymbol S$ are positive definite. The block inverse identity gives the following quadratic-form decomposition:

\begin{pmatrix}\mathbf z\\\mathbf y\end{pmatrix}^{\mathsf T} \boldsymbol\Sigma^{-1} \begin{pmatrix}\mathbf z\\\mathbf y\end{pmatrix} = \mathbf z^{\mathsf T}\boldsymbol A^{-1}\mathbf z+ \left(\mathbf y-\boldsymbol B\boldsymbol A^{-1}\mathbf z\right)^{\mathsf T} \boldsymbol S^{-1} \left(\mathbf y-\boldsymbol B\boldsymbol A^{-1}\mathbf z\right).

This identity follows by multiplying out the right side, or by block elimination in $\boldsymbol\Sigma$. Also, the block determinant identity gives $|\boldsymbol\Sigma|=|\boldsymbol A||\boldsymbol S|$. Substituting both identities into the joint Gaussian density separates it into a factor depending only on $\mathbf z$ and a factor depending on $\mathbf y$ given $\mathbf z$:

f(\mathbf x_O,\mathbf x_M) = f_O(\mathbf x_O)\, \frac{1}{(2\pi)^{|M|/2}|\boldsymbol S|^{1/2}} \exp\!\left[ -\frac12 (\mathbf y-\boldsymbol B\boldsymbol A^{-1}\mathbf z)^{\mathsf T} \boldsymbol S^{-1} (\mathbf y-\boldsymbol B\boldsymbol A^{-1}\mathbf z) \right].

For each fixed $\mathbf x_O$, the second factor is a normalized Gaussian density in $\mathbf x_M$. Its center is $\boldsymbol\mu_M+\boldsymbol B\boldsymbol A^{-1}\mathbf z$, and its covariance is $\boldsymbol S$. This proves the conditional distribution formula. The density argument applies when both blocks are nonempty; the cases with an empty block are described below. $\square$

A second useful fact is that the Schur complement is positive definite, not merely a formal covariance expression. For any nonzero vector $\mathbf v$ of the same dimension as $\mathbf X_M$, form the vector $\mathbf w=(-\boldsymbol A^{-1}\boldsymbol B^{\mathsf T}\mathbf v,\mathbf v)^{\mathsf T}$. It is nonzero, so positive definiteness of $\boldsymbol\Sigma$ gives $\mathbf w^{\mathsf T}\boldsymbol\Sigma\mathbf w>0$. Expanding the left side yields

\mathbf w^{\mathsf T}\boldsymbol\Sigma\mathbf w = \mathbf v^{\mathsf T} \left(\boldsymbol C-\boldsymbol B\boldsymbol A^{-1} \boldsymbol B^{\mathsf T}\right)\mathbf v = \mathbf v^{\mathsf T}\boldsymbol S\mathbf v>0.

Therefore $\boldsymbol S$ is positive definite. In particular, the conditional Gaussian distribution is nondegenerate whenever the full covariance matrix is positive definite and the missing block is nonempty. $\square$

Worked Calculations

Worked Example: One Missing Component Given Two Observed Components

Suppose $\mathbf X\sim N_3(\boldsymbol\mu,\boldsymbol\Sigma)$, with

\boldsymbol\mu= \begin{pmatrix}10\\20\\30\end{pmatrix}, \qquad \boldsymbol\Sigma= \begin{pmatrix} 4&2&1\\ 2&9&3\\ 1&3&4 \end{pmatrix}.

Components 1 and 3 are observed at $x_1=12$ and $x_3=28$, while component 2 is missing. In the order $O=(1,3)$, $M=(2)$, the observed covariance block and its inverse are

\boldsymbol\Sigma_{OO}= \begin{pmatrix}4&1\\1&4\end{pmatrix}, \qquad \boldsymbol\Sigma_{OO}^{-1} = \frac1{15}\begin{pmatrix}4&-1\\-1&4\end{pmatrix}.

The row of covariances from the missing component to the observed components is $\boldsymbol\Sigma_{MO}=(2,3)$. Thus

\boldsymbol\Sigma_{MO}\boldsymbol\Sigma_{OO}^{-1} = \frac1{15}(2,3) \begin{pmatrix}4&-1\\-1&4\end{pmatrix} = \left(\frac13,\frac23\right).

The observed deviations from their means are $(12-10,28-30)=(2,-2)$, so the conditional mean of $X_2$ is

20+\left(\frac13,\frac23\right) \begin{pmatrix}2\\-2\end{pmatrix} =20+\frac23-\frac43 =\frac{58}{3}.

The conditional variance is

9-(2,3)\frac1{15} \begin{pmatrix}4&-1\\-1&4\end{pmatrix} \begin{pmatrix}2\\3\end{pmatrix} = 9-(2,3)\begin{pmatrix}1/3\\2/3\end{pmatrix} =9-\frac83=\frac{19}{3}.

Therefore $X_2\mid X_1=12,X_3=28\sim N(58/3,19/3)$. The conditional variance is positive, as required.

Worked Example: Two Missing Components Given One Observed Component

Let $\boldsymbol\mu=(0,1,2)^{\mathsf T}$, and suppose the covariance matrix is

\boldsymbol\Sigma= \begin{pmatrix} 4&1&2\\ 1&4&1\\ 2&1&5 \end{pmatrix}.

Only component 2 is observed, at $x_2=6$; components 1 and 3 are missing. With $M=(1,3)$, the missing-to-observed covariance block is $\boldsymbol\Sigma_{MO}=(1,1)^{\mathsf T}$, and $\boldsymbol\Sigma_{OO}=4$. The conditional mean is

\begin{pmatrix}0\\2\end{pmatrix} + \begin{pmatrix}1\\1\end{pmatrix} \frac14(6-1) = \begin{pmatrix}5/4\\13/4\end{pmatrix}.

The marginal covariance of $(X_1,X_3)^{\mathsf T}$ and its adjustment are

\boldsymbol\Sigma_{MM} = \begin{pmatrix}4&2\\2&5\end{pmatrix}, \qquad \boldsymbol\Sigma_{MO}\boldsymbol\Sigma_{OO}^{-1}\boldsymbol\Sigma_{OM} = \begin{pmatrix}1\\1\end{pmatrix}\frac14(1,1) = \begin{pmatrix}1/4&1/4\\1/4&1/4\end{pmatrix}.

Subtracting gives

\boldsymbol S_{M\mid O} = \begin{pmatrix}15/4&7/4\\7/4&19/4\end{pmatrix}.

Hence the conditional distribution of $(X_1,X_3)^{\mathsf T}$ has mean $(5/4,13/4)^{\mathsf T}$ and this $2\times2$ covariance matrix. The positive diagonal entries alone do not establish positive definiteness; here its determinant is $(15\cdot19-7^2)/16=236/16>0$, consistent with the Schur-complement result.

Applying the Formulas Across Response Patterns

A response pattern specifies which components are observed for a case. To use the block formulas, first order the observed indices and missing indices, then extract the corresponding blocks from the same $\boldsymbol\mu$ and $\boldsymbol\Sigma$. The two patterns below illustrate why a single covariance matrix can produce different conditional calculations.

Worked Example: Calculating Moments for Two Response Patterns

Take $\boldsymbol\mu=\mathbf 0$ and

\boldsymbol\Sigma= \begin{pmatrix} 4&2&1\\ 2&5&2\\ 1&2&3 \end{pmatrix}.

For a case with $O=(1)$, $M=(2,3)$, and $x_1=2$, the conditional mean is

\begin{pmatrix}2\\1\end{pmatrix}\frac14(2) = \begin{pmatrix}1\\1/2\end{pmatrix},

and the conditional covariance is

\begin{pmatrix}5&2\\2&3\end{pmatrix} - \begin{pmatrix}2\\1\end{pmatrix}\frac14(2,1) = \begin{pmatrix}4&3/2\\3/2&11/4\end{pmatrix}.

For another case with $O=(1,3)$, $M=(2)$, and $(x_1,x_3)=(2,1)$, the observed covariance and its inverse are

\boldsymbol\Sigma_{OO}= \begin{pmatrix}4&1\\1&3\end{pmatrix}, \qquad \boldsymbol\Sigma_{OO}^{-1} =\frac1{11}\begin{pmatrix}3&-1\\-1&4\end{pmatrix}.

Here $\boldsymbol\Sigma_{MO}=(2,2)$, so the conditional mean is

(2,2)\frac1{11} \begin{pmatrix}3&-1\\-1&4\end{pmatrix} \begin{pmatrix}2\\1\end{pmatrix} = \left(\frac4{11},\frac6{11}\right) \begin{pmatrix}2\\1\end{pmatrix} =\frac{14}{11}.

Its conditional variance is $5-(4/11,6/11)(2,2)^{\mathsf T}=5-20/11=35/11$. These are pattern-specific calculations: the observed block, the missing block, and therefore the conditional covariance all change with the pattern.

Interpreting the Moments Carefully

For a fixed pattern, the conditional mean is the Gaussian model's best prediction of the missing vector given the observed values, in the sense of conditional expectation. The conditional covariance describes the remaining uncertainty after those values are known. It is not generally zero: even highly informative observed components need not determine the missing ones exactly.

There are two edge cases. If all components are observed, there is no missing block to predict; the conditional vector is the observed value itself and its conditional covariance is zero. If none are observed, conditioning supplies no information, so the missing vector retains its unconditional distribution $N_p(\boldsymbol\mu,\boldsymbol\Sigma)$. These conventions agree with the underlying interpretation, even though the displayed block formula uses an inverse for a nonempty observed block.

The calculation conditions on the values of the observed components under the specified Gaussian model. It does not, by itself, account for information carried by the event that a particular response pattern occurred. If missingness is informative beyond the observed values, the distribution conditional on both the pattern and observed values may require a model for the missingness process as well.

Common Mistake

Common pitfall: Replacing the conditional covariance by $\boldsymbol\Sigma_{MM}$ because “the covariance of the missing components is unchanged.” That is their marginal covariance, before using the observed values. The conditional covariance subtracts $\boldsymbol\Sigma_{MO}\boldsymbol\Sigma_{OO}^{-1}\boldsymbol\Sigma_{OM}$, the covariance contribution explained by the observed block. Conversely, the conditional mean is not generally just $\boldsymbol\mu_M$; it shifts according to the observed deviations.

A reliable check is to verify dimensions before multiplying: $\boldsymbol\Sigma_{MO}$ has one row per missing component and one column per observed component, so the conditional mean shift has one entry per missing component. Also check that the resulting conditional covariance is symmetric and positive definite for a nonempty missing block under the assumed positive-definite model.

Check Your Understanding

For each question, identify the relevant blocks before calculating.

In the conditional mean formula, what role does $\mathbf x_O-\boldsymbol\mu_O$ play?
Why is $\boldsymbol\Sigma_{OO}$ invertible when the full covariance matrix is positive definite?
For $O=(1)$, $M=(2)$, and $\boldsymbol\Sigma=\begin{pmatrix}4&2\\2&5\end{pmatrix}$, calculate the conditional variance of $X_2$ given $X_1$.
Does the conditional covariance depend on the particular observed values? Does it depend on the response pattern?
Why might conditioning on the observed values alone fail to describe the distribution of missing values when the missingness mechanism is informative?