Conditional Multivariate Normal Distributions

Why Conditioning Changes a Gaussian Distribution

For jointly Gaussian measurements, observing one block updates what we expect about another block whenever the two are correlated. The update is linear in the observed value, while the remaining uncertainty is summarized by a covariance matrix that does not depend on that value. Deriving both pieces carefully matters: the conditional covariance is generally not the original marginal covariance, and its formula involves a matrix Schur complement.

Let $X_1$ be a $p$-dimensional random vector and $X_2$ a $q$-dimensional random vector, jointly Gaussian with mean vectors $\mu_1,\mu_2$. Partition their covariance matrix as follows:

\begin{pmatrix}X_1\\X_2\end{pmatrix} \sim N\left( \begin{pmatrix}\mu_1\\\mu_2\end{pmatrix}, \begin{pmatrix} \Sigma_{11} & \Sigma_{12}\\ \Sigma_{21} & \Sigma_{22} \end{pmatrix} \right), \qquad \Sigma_{21}=\Sigma_{12}^{\mathsf T}.

Assume the full covariance matrix is positive definite. In particular, $\Sigma_{22}$ is invertible. The block $X_2$ is the one being observed, and $X_1$ is the block whose conditional distribution we seek.

Theorem: Under these assumptions, for every $x_2\in\mathbb R^q$, a conditional distribution of $X_1$ given $X_2=x_2$ is multivariate normal with mean and covariance $$ E[X_1\mid X_2=x_2] =\mu_1+\Sigma_{12}\Sigma_{22}^{-1}(x_2-\mu_2), \qquad \operatorname{Cov}(X_1\mid X_2=x_2) =\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}. $$ The covariance on the right is called the Schur complement of $\Sigma_{22}$ in the joint covariance matrix. It is positive definite and does not depend on $x_2$.

Deriving the Conditional Distribution

The key is to separate the part of $X_1$ linearly associated with $X_2$ from a residual. Define

B=\Sigma_{12}\Sigma_{22}^{-1}, \qquad R=(X_1-\mu_1)-B(X_2-\mu_2).

Because $R$ and $X_2$ are linear transformations of a jointly Gaussian vector, they are jointly Gaussian. Their cross-covariance is

\begin{aligned} \operatorname{Cov}(R,X_2) &=\operatorname{Cov}(X_1,X_2) -B\operatorname{Cov}(X_2,X_2)\\ &=\Sigma_{12}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{22}\\ &=0. \end{aligned}

For jointly Gaussian vectors, zero cross-covariance implies independence. In this setting, that fact follows from the joint Gaussian characteristic function: when the cross-covariance is zero, its quadratic exponent separates into a term involving $R$ and a term involving $X_2$, so the joint characteristic function factors into the product of their marginal characteristic functions. Thus $R$ and $X_2$ are independent.

The residual has mean zero. Expanding its covariance and using symmetry of the covariance blocks gives

\begin{aligned} \operatorname{Cov}(R) &=\Sigma_{11}-B\Sigma_{21}-\Sigma_{12}B^{\mathsf T} +B\Sigma_{22}B^{\mathsf T}\\ &=\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}. \end{aligned}

The last equality holds because $B\Sigma_{21}=\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$, $\Sigma_{12}B^{\mathsf T}$ has the same value, and $B\Sigma_{22}B^{\mathsf T}$ equals that value as well. Since

X_1=\mu_1+B(X_2-\mu_2)+R

and $R$ is independent of $X_2$, conditioning on $X_2=x_2$ leaves the distribution of $R$ unchanged. Adding the fixed vector $\mu_1+B(x_2-\mu_2)$ shifts its mean and leaves its covariance unchanged. This proves the stated conditional Gaussian law.

It remains to verify that the conditional covariance is positive definite, rather than merely nonnegative definite. For any nonzero $a\in\mathbb R^p$, set $b=-\Sigma_{22}^{-1}\Sigma_{21}a$. The vector $(a^{\mathsf T},b^{\mathsf T})^{\mathsf T}$ is nonzero, so positive definiteness of the full covariance matrix implies

0< \begin{pmatrix}a\\b\end{pmatrix}^{\mathsf T} \begin{pmatrix}\Sigma_{11}&\Sigma_{12}\\ \Sigma_{21}&\Sigma_{22}\end{pmatrix} \begin{pmatrix}a\\b\end{pmatrix} =a^{\mathsf T} \left(\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}\right)a.

The equality follows by substituting the definition of $b$ and expanding the quadratic form. Since this holds for every nonzero $a$, the Schur complement is positive definite. This completes the derivation. $\square$

Interpreting the Formula

The conditional mean has the form of a multivariate linear prediction. The matrix $B=\Sigma_{12}\Sigma_{22}^{-1}$ maps a deviation $x_2-\mu_2$ into an adjustment to $\mu_1$. If $X_1$ and $X_2$ are uncorrelated, then $\Sigma_{12}=0$, so observing $X_2$ does not change the conditional mean or covariance. For jointly Gaussian blocks, uncorrelatedness also means independence.

The conditional covariance subtracts the part of $\Sigma_{11}$ associated with the observed block. It is therefore no larger than the marginal covariance in the positive-semidefinite ordering: for every $a$, the variance of $a^{\mathsf T}X_1$ conditional on $X_2$ is no greater than its marginal variance. In particular, the conditional covariance describes residual uncertainty after using $X_2$, not the variability of $X_1$ before observing anything.

Worked Example: A Scalar Conditioning Block

Suppose $(X,Y)^{\mathsf T}$ is jointly normal with mean $(10,20)^{\mathsf T}$ and covariance matrix

\begin{pmatrix}9&6\\6&16\end{pmatrix}.

Find the distribution of $X$ given $Y=24$. The conditioning block is scalar, so $\Sigma_{12}=6$, $\Sigma_{22}=16$, and $\Sigma_{11}=9$. The conditional mean is

10+\frac{6}{16}(24-20) =10+\frac{3}{8}\cdot4 =11.5.

The conditional variance is

9-\frac{6\cdot6}{16} =9-\frac{36}{16} =9-2.25 =6.75.

Thus $X\mid Y=24\sim N(11.5,6.75)$. The joint covariance is valid and nonsingular: its leading diagonal entry is $9>0$, and its determinant is $9\cdot16-6^2=108>0$. The observed value is four units above $E[Y]$; the positive covariance shifts the conditional mean of $X$ upward by $1.5$.

Worked Example: A Vector Conditioning Block

Let $X_1\in\mathbb R^2$ and $X_2\in\mathbb R^2$ have mean vectors $\mu_1=(1,-2)^{\mathsf T}$ and $\mu_2=(0,1)^{\mathsf T}$, with covariance blocks

\Sigma_{11}=\begin{pmatrix}6&0\\0&4\end{pmatrix}, \qquad \Sigma_{12}=\begin{pmatrix}2&3\\1&-3\end{pmatrix}, \qquad \Sigma_{22}=\begin{pmatrix}4&0\\0&9\end{pmatrix}.

First compute the regression matrix and conditional covariance:

\begin{aligned} B&=\Sigma_{12}\Sigma_{22}^{-1} =\begin{pmatrix}2&3\\1&-3\end{pmatrix} \begin{pmatrix}1/4&0\\0&1/9\end{pmatrix} =\begin{pmatrix}1/2&1/3\\1/4&-1/3\end{pmatrix},\\ \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21} &=B\Sigma_{12}^{\mathsf T} =\begin{pmatrix}2&-1/2\\-1/2&5/4\end{pmatrix},\\ S&=\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21} =\begin{pmatrix}4&1/2\\1/2&11/4\end{pmatrix}. \end{aligned}

The conditional covariance is positive definite: its first diagonal entry is $4>0$, and its determinant is $4(11/4)-(1/2)^2=11-1/4=43/4>0$. This also verifies that the displayed blocks define a positive-definite joint covariance, since $\Sigma_{22}$ is positive definite and its Schur complement $S$ is positive definite.

For the observed value $x_2=(2,-2)^{\mathsf T}$, the deviation from its mean is $(2,-3)^{\mathsf T}$. Therefore

B(x_2-\mu_2) =\begin{pmatrix}1/2&1/3\\1/4&-1/3\end{pmatrix} \begin{pmatrix}2\\-3\end{pmatrix} =\begin{pmatrix}1-1\\1/2+1\end{pmatrix} =\begin{pmatrix}0\\3/2\end{pmatrix}.

Hence $E[X_1\mid X_2=x_2]=(1,-2)^{\mathsf T}+(0,3/2)^{\mathsf T}=(1,-1/2)^{\mathsf T}$, and the conditional covariance is $S$ above. The resulting conditional law is $N((1,-1/2)^{\mathsf T},S)$.

Worked Example: A Vector Target Given One Measurement

Suppose $X_1=(U,V)^{\mathsf T}$ and $Y$ are jointly normal, with $\mu_1=(0,1)^{\mathsf T}$, $\mu_Y=2$, and covariance blocks

\Sigma_{11}=\begin{pmatrix}4&1\\1&3\end{pmatrix}, \qquad \Sigma_{12}=\begin{pmatrix}1\\2\end{pmatrix}, \qquad \Sigma_{22}=4.

For $Y=4$, the deviation is $4-2=2$. The conditional mean is

\begin{pmatrix}0\\1\end{pmatrix} +\begin{pmatrix}1\\2\end{pmatrix}\frac{1}{4}(2) =\begin{pmatrix}1/2\\2\end{pmatrix}.

The conditional covariance is

\begin{pmatrix}4&1\\1&3\end{pmatrix} -\begin{pmatrix}1\\2\end{pmatrix}\frac{1}{4} \begin{pmatrix}1&2\end{pmatrix} =\begin{pmatrix}4&1\\1&3\end{pmatrix} -\begin{pmatrix}1/4&1/2\\1/2&1\end{pmatrix} =\begin{pmatrix}15/4&1/2\\1/2&2\end{pmatrix}.

Its determinant is $(15/4)2-(1/2)^2=15/2-1/4=29/4>0$, and its first diagonal entry is positive, so it is positive definite. The full covariance is valid by the same Schur-complement check, because $\Sigma_{22}=4>0$. Thus the conditional law is $N((1/2,2)^{\mathsf T},\begin{pmatrix}15/4&1/2\\1/2&2\end{pmatrix})$.

Common Mistake: Keeping the Marginal Covariance

A frequent error is to use the conditional mean formula but leave the covariance equal to $\Sigma_{11}$, on the reasoning that observing $X_2$ changes the center but not the spread. That is generally wrong: when $\Sigma_{12}\ne0$, the observed block supplies information about $X_1$, reducing its uncertainty. The scalar example makes this visible: the marginal variance of $X$ is $9$, while its conditional variance given $Y$ is $6.75$. The correct covariance is always the Schur complement $\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$ under the stated nonsingularity assumptions.

Another point of care is the order of the blocks. If the conditioning block is $X_2$, the inverse in the formula is $\Sigma_{22}^{-1}$, and the cross-covariance on the left is $\Sigma_{12}$. Matrix multiplication order matters; replacing the expression with a scalar-style division or reversing the blocks can produce an undefined or incorrectly dimensioned result.

Key takeaway: For a jointly Gaussian vector with positive-definite block covariance, conditioning $X_1$ on $X_2=x_2$ gives a Gaussian law. Its mean shifts by $\Sigma_{12}\Sigma_{22}^{-1}(x_2-\mu_2)$, and its covariance is the positive-definite Schur complement $\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$.

Check Your Understanding

Use the block formula and show the matrix calculations in each case.

If $\Sigma_{12}=0$, what are the conditional mean and covariance of $X_1$ given $X_2=x_2$? What does this imply about the Gaussian blocks?
For the scalar example, calculate the conditional mean of $X$ given $Y=16$, and explain the direction of its shift from $E[X]$.
In the vector-conditioning example, explain why the conditional covariance is the same for every possible observed $x_2$.
Why does positive definiteness of the full covariance matrix imply positive definiteness of the Schur-complement covariance?
Identify the error in the claim that the conditional covariance is always $\Sigma_{11}$, regardless of $\Sigma_{12}$.

Conditional Multivariate Normal Distributions

What You'll Learn