Conditional Matrix-Normal Laws From Block Covariance

Why Block Conditioning Matters

In a multivariate Gaussian model, observing one group of coordinates updates the distribution of another group. The update is determined by two features of the joint covariance: the covariance linking the blocks, which controls the conditional mean, and the uncertainty left after that link is accounted for, which controls the conditional covariance. For a matrix-normal variable, the same principle applies, but the row and column covariance factors must be kept in their proper positions.

We will first condition on a block of columns. This makes the role of the column covariance especially clear. The result follows from ordinary Gaussian block conditioning after vectorizing the matrices; we derive that Gaussian result before applying it.

Matrix-Normal Setup and Notation

Definition: A $p\times q$ random matrix $X$ has a matrix-normal distribution with mean $M$, row covariance $U$, and column covariance $V$, written $X\sim MN_{p,q}(M,U,V)$, if $$ \operatorname{vec}(X)\sim N\!\left(\operatorname{vec}(M),\,V\otimes U\right). $$ Here $U$ is a $p\times p$ symmetric positive-definite matrix, $V$ is a $q\times q$ symmetric positive-definite matrix, and $\operatorname{vec}(X)$ stacks the columns of $X$. Thus the covariance between entries $X_{ij}$ and $X_{k\ell}$ is $U_{ik}V_{j\ell}$.

Partition the columns into groups of sizes $q_1$ and $q_2$, where $q=q_1+q_2$. Write $X=[X_1\ X_2]$ and $M=[M_1\ M_2]$, so $X_1,M_1$ are $p\times q_1$ and $X_2,M_2$ are $p\times q_2$. Partition the column covariance conformably:

V= \begin{pmatrix} V_{11}&V_{12}\\ V_{21}&V_{22} \end{pmatrix}, \qquad V_{21}=V_{12}^{T}.

The dimensions are $V_{11}:q_1\times q_1$, $V_{12}:q_1\times q_2$, $V_{21}:q_2\times q_1$, and $V_{22}:q_2\times q_2$. Since $V$ is positive definite, $V_{22}$ is positive definite and invertible.

The Gaussian Block-Conditioning Result

The matrix-normal calculation uses a general fact about jointly Gaussian vectors. Its proof also explains why the conditional covariance is a Schur complement.

Theorem: Suppose $A$ and $B$ are jointly Gaussian vectors with means $\mu_A,\mu_B$ and positive-definite joint covariance $$ \begin{pmatrix} \Sigma_{11}&\Sigma_{12}\\ \Sigma_{21}&\Sigma_{22} \end{pmatrix}. $$ Then $\Sigma_{22}$ is invertible, and $$ A\mid B=b\sim N\!\left( \mu_A+\Sigma_{12}\Sigma_{22}^{-1}(b-\mu_B),\ \Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21} \right). $$

To prove the result, define the residual vector

R=A-\mu_A-\Sigma_{12}\Sigma_{22}^{-1}(B-\mu_B).

It is a linear transformation of jointly Gaussian vectors, so $(R,B)$ is jointly Gaussian. Directly using the covariance blocks gives

\operatorname{Cov}(R,B) =\Sigma_{12}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{22}=0,

and

\operatorname{Var}(R) =\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}.

For jointly Gaussian vectors, zero cross-covariance implies independence: their joint characteristic function has exponent determined by the covariance matrix, and when the cross-covariance blocks are zero, that characteristic function factors into the product of the two marginal characteristic functions. Hence $R$ and $B$ are independent. The identity

A=\mu_A+\Sigma_{12}\Sigma_{22}^{-1}(B-\mu_B)+R

then gives the conditional distribution: once $B=b$ is fixed, the first two terms are fixed and $R$ retains its unchanged Gaussian distribution. This proves the stated conditional mean and covariance. $\square$

Conditional Law for a Column Block

For the matrix-normal variable, column-stacking the partition gives $\operatorname{vec}(X)=[\operatorname{vec}(X_1)^T,\operatorname{vec}(X_2)^T]^T$. The covariance blocks of these two Gaussian vectors are $V_{11}\otimes U$, $V_{12}\otimes U$, $V_{21}\otimes U$, and $V_{22}\otimes U$. Applying the theorem yields the conditional mean and covariance. In matrix form, the result is:

Theorem: If $X\sim MN_{p,q}(M,U,V)$ with $U,V$ positive definite, then for any $X_2=x_2$, $$ X_1\mid X_2=x_2\sim MN_{p,q_1}\!\left( M_1+(x_2-M_2)V_{22}^{-1}V_{21},\ U,\ V_{11}-V_{12}V_{22}^{-1}V_{21} \right). $$

The order in the conditional mean is important. The coefficient $V_{22}^{-1}V_{21}$ has dimensions $q_2\times q_1$, so multiplying the $p\times q_2$ residual $x_2-M_2$ on the right produces a $p\times q_1$ matrix, as required. The conditional row covariance remains $U$; conditioning changes the column covariance to the Schur complement

S=V_{11}-V_{12}V_{22}^{-1}V_{21}.

For completeness, this covariance can also be seen from the independent residual in the Gaussian proof. Set $C=V_{22}^{-1}V_{21}$ and define

$$ R_X=X_1-M_1-(X_2-M_2)C. $$

Then $\operatorname{vec}(R_X)$ is Gaussian and has zero covariance with $\operatorname{vec}(X_2)$. Using $\operatorname{vec}(AB)=(B^T\otimes I)\operatorname{vec}(A)$, its covariance is $S\otimes U$. Thus it is independent of $X_2$, and $X_1=M_1+(X_2-M_2)C+R_X$ gives the stated conditional matrix-normal law.

Why the Schur Complement Is a Valid Covariance

The conditional column covariance $S$ is positive definite, not merely symmetric. To verify this, take any nonzero $z\in\mathbb{R}^{q_1}$ and form $w=-V_{22}^{-1}V_{21}z$. Since $V$ is positive definite, every nonzero vector has a positive quadratic form. The vector $(z^T,w^T)^T$ is nonzero because $z\ne 0$, and direct multiplication gives

\begin{pmatrix}z\\w\end{pmatrix}^{T} V \begin{pmatrix}z\\w\end{pmatrix} =z^T\!\left(V_{11}-V_{12}V_{22}^{-1}V_{21}\right)z =z^T S z>0.

Therefore $S$ is positive definite. This check matters because a proposed conditional covariance must itself be a valid covariance matrix. The argument depends on the assumed positive definiteness of the full covariance $V$.

Worked Example: Conditioning One Scalar on Another

Let $p=1$, $M=(0,0)$, $U=(2)$, and $$ V=\begin{pmatrix}4&1.2\\1.2&1\end{pmatrix}. $$ The matrix $V$ is positive definite: its leading diagonal entry is $4>0$, and its determinant is $4-1.2^2=2.56>0$. Condition the first entry $X_1$ on $X_2=3$.

Here $V_{22}=1$, $V_{21}=1.2$, and $V_{12}=1.2$. The conditional mean is

0+(3-0)(1)^{-1}(1.2)=3.6.

The conditional column variance is $4-(1.2)(1)^{-1}(1.2)=2.56$. Because the row covariance is $U=2$, the scalar conditional variance is $2(2.56)=5.12$. Thus $X_1\mid X_2=3\sim N(3.6,5.12)$, where the second argument is the variance.

Worked Example: Conditioning a Two-Column Block

Let $p=2$, $q_1=2$, $q_2=1$, and take $$ U=\begin{pmatrix}1&0.3\\0.3&2\end{pmatrix},\quad V_{11}=\begin{pmatrix}3&0.5\\0.5&2\end{pmatrix},\quad V_{12}=\begin{pmatrix}1\\0.5\end{pmatrix},\quad V_{22}=(2). $$ Set $V_{21}=V_{12}^T$, $M_1=\begin{pmatrix}0&1\\2&-1\end{pmatrix}$, $M_2=\begin{pmatrix}0\\1\end{pmatrix}$, and observe $x_2=\begin{pmatrix}2\\0\end{pmatrix}$.

First, the conditional-mean coefficient has dimensions $1\times2$:

V_{22}^{-1}V_{21} =\frac12\begin{pmatrix}1&0.5\end{pmatrix} =\begin{pmatrix}0.5&0.25\end{pmatrix}.

Since $x_2-M_2=(2,-1)^T$, the adjustment is $$ \begin{pmatrix}2\\-1\end{pmatrix} \begin{pmatrix}0.5&0.25\end{pmatrix} = \begin{pmatrix}1&0.5\\-0.5&-0.25\end{pmatrix}. $$ Adding this to $M_1$, the conditional mean is $$ \begin{pmatrix}1&1.5\\1.5&-1.25\end{pmatrix}. $$ The Schur complement is $$ S=\begin{pmatrix}3&0.5\\0.5&2\end{pmatrix} -\begin{pmatrix}1\\0.5\end{pmatrix}\frac12\begin{pmatrix}1&0.5\end{pmatrix} =\begin{pmatrix}2.5&0.25\\0.25&1.875\end{pmatrix}. $$ Its first diagonal entry is positive and its determinant is $2.5(1.875)-0.25^2=4.625>0$, so $S$ is positive definite. The conditional law is therefore matrix-normal with the displayed mean, row covariance $U$, and column covariance $S$.

Conditioning on a Row Block

The same Gaussian result also gives the corresponding formula when the partition is by rows. Write $X=\begin{pmatrix}X_1\\X_2\end{pmatrix}$, partition $M$ in the same way, and partition the row covariance as $U=(U_{ij})_{i,j=1,2}$. If the two row-block sizes are $p_1,p_2$, then $U_{12}$ is $p_1\times p_2$, and conditioning on $X_2=x_2$ gives

X_1\mid X_2=x_2\sim MN_{p_1,q}\!\left( M_1+U_{12}U_{22}^{-1}(x_2-M_2),\ U_{11}-U_{12}U_{22}^{-1}U_{21},\ V \right).

Here the row covariance changes by a Schur complement, while the column covariance $V$ is unchanged. This is the row-column counterpart of conditioning on column blocks.

Worked Example: Conditioning the First Row

Let $X\sim MN_{2,2}(M,U,V)$, where $$ M=\begin{pmatrix}1&0\\0&2\end{pmatrix},\quad U=\begin{pmatrix}3&1\\1&2\end{pmatrix},\quad V=\begin{pmatrix}1&0.2\\0.2&2\end{pmatrix}. $$ Both covariance matrices are positive definite: $\det(U)=5>0$ and $\det(V)=1.96>0$, with positive diagonal entries. Condition on the second row being $x_2=(2,1)$.

The row-block coefficient is $U_{12}U_{22}^{-1}=1/2$, and the observed residual is $x_2-M_2=(2,-1)$. The conditional mean of the first row is $$ (1,0)+\frac12(2,-1)=(2,-0.5). $$ The conditional row variance is $U_{11}-U_{12}U_{22}^{-1}U_{21}=3-\frac12=2.5$; the column covariance remains $V$. Thus the conditional law is $MN_{1,2}((2,-0.5),(2.5),V)$.

Common Mistakes and Practical Checks

A frequent error is to put the column-block coefficient in the wrong order. The expression $V_{21}V_{22}^{-1}$ is generally not conformable: $V_{21}$ is $q_2\times q_1$, while $V_{22}^{-1}$ is $q_2\times q_2$, so their inner dimensions need not match. Even when $q_1=q_2$ and the product exists, that does not make it the correct coefficient on the right of $X_2-M_2$. The required coefficient is $V_{22}^{-1}V_{21}$, which is $q_2\times q_1$ and makes $(X_2-M_2)V_{22}^{-1}V_{21}$ a $p\times q_1$ matrix.

Another error is to replace the conditional column covariance with $V_{11}$, as though learning $X_2$ had no effect on uncertainty in $X_1$. The subtracted term $V_{12}V_{22}^{-1}V_{21}$ accounts for the part of the first block's covariance explained by its association with the second block. Conversely, do not change the row covariance $U$ when conditioning on column blocks: the Schur complement is applied to the covariance factor for the block being partitioned.

Key takeaway: For column blocks, the conditional mean adjustment is $(x_2-M_2)V_{22}^{-1}V_{21}$, and the conditional column covariance is $V_{11}-V_{12}V_{22}^{-1}V_{21}$. Check the dimensions of each product before using either formula.

Check Your Understanding

Use the dimensions and block-conditioning argument to answer these questions.

For column blocks of widths $q_1$ and $q_2$, what are the dimensions of $V_{22}^{-1}V_{21}$ and of the resulting conditional-mean adjustment?
Why does the conditional column covariance involve a Schur complement rather than $V_{11}$ alone?
State the condition on $V$ that ensures the Schur complement is positive definite.
When conditioning on row blocks instead of column blocks, which covariance factor changes?
Explain why $V_{21}V_{22}^{-1}$ is generally not a valid coefficient for the column-block conditional mean.