Partitioned Gaussian Vectors and Block Covariance Notation

Why Partition a Gaussian Vector?

Many multivariate models describe two groups of random quantities: a response vector and another vector that will serve as conditioning information. Before a conditional distribution is calculated, the joint vector must be organized carefully. In particular, the order of its coordinates determines where each covariance submatrix belongs and what its dimensions must be.

This tutorial focuses on that bookkeeping. A block label such as $\Sigma_{YX}$ records both which variables are involved and the orientation of the matrix. It is not interchangeable with $\Sigma_{XY}$, even though the two blocks are related. A partition by itself does not assert any conditional distribution or independence.

Definition: Let $Y$ be a $p$-dimensional random vector and $X$ a $q$-dimensional random vector. Write the ordered joint vector, mean, and covariance matrix as $$ Z=\begin{pmatrix}Y\\X\end{pmatrix},\qquad E[Z]=\begin{pmatrix}\mu_Y\\\mu_X\end{pmatrix},\qquad \operatorname{Cov}(Z)= \begin{pmatrix} \Sigma_{YY} & \Sigma_{YX}\\ \Sigma_{XY} & \Sigma_{XX} \end{pmatrix}. $$ Here $\mu_Y=E[Y]$ is $p\times1$, $\mu_X=E[X]$ is $q\times1$, $\Sigma_{YY}=\operatorname{Cov}(Y)$ is $p\times p$, $\Sigma_{XX}=\operatorname{Cov}(X)$ is $q\times q$, $\Sigma_{YX}=\operatorname{Cov}(Y,X)$ is $p\times q$, and $\Sigma_{XY}=\operatorname{Cov}(X,Y)$ is $q\times p$. If $Z$ is jointly Gaussian, then $(Y^T,X^T)^T$ is its partition into response and conditioning blocks.

For vectors $A$ and $B$ with finite second moments, the cross-covariance convention in this definition is $\operatorname{Cov}(A,B)=E[(A-E[A])(B-E[B])^T]$. The first argument determines the row dimension and the second determines the column dimension. This order is essential: $\operatorname{Cov}(Y,X)$ has $p$ rows and $q$ columns.

Reading the Block Matrix

The diagonal blocks describe variation within each vector: $\Sigma_{YY}$ contains the variances and covariances among response coordinates, while $\Sigma_{XX}$ contains those among conditioning coordinates. The off-diagonal blocks describe cross-covariances between the two groups. The complete matrix has $p+q$ rows and $p+q$ columns, consistent with the length of $Z$.

Since a covariance matrix is symmetric, the cross-covariance blocks have a fixed relationship. The block above the diagonal is not generally equal to the block below it entry by entry; it is its transpose.

Theorem: For finite-second-moment random vectors $Y\in\mathbb{R}^p$ and $X\in\mathbb{R}^q$, $$ \Sigma_{XY}=\Sigma_{YX}^T. $$ The diagonal covariance blocks are symmetric: $\Sigma_{YY}=\Sigma_{YY}^T$ and $\Sigma_{XX}=\Sigma_{XX}^T$.

Proof. By the cross-covariance definition, $$ \Sigma_{YX}=E[(Y-\mu_Y)(X-\mu_X)^T]. $$ Transposing gives $$ \Sigma_{YX}^T =\left(E[(Y-\mu_Y)(X-\mu_X)^T]\right)^T =E[(X-\mu_X)(Y-\mu_Y)^T] =\Sigma_{XY}. $$ The transpose can be taken inside the expectation because each entry is an expectation of an integrable product. Setting $X=Y$, or $Y=X$, in the same calculation shows that each diagonal covariance block equals its transpose. $\square$

There is also a basic validity check: every covariance matrix is positive semidefinite. A principal block, such as $\Sigma_{YY}$, must therefore be positive semidefinite too. A principal block is formed by retaining the same selected coordinates for its rows and columns; the diagonal blocks in a partition are principal blocks.

Theorem: If $\operatorname{Cov}(Z)$ is positive semidefinite, then each diagonal block $\Sigma_{YY}$ and $\Sigma_{XX}$ is positive semidefinite.

Proof. A symmetric matrix $M$ is positive semidefinite when $a^TMa\geq 0$ for every compatible vector $a$. Take any $u\in\mathbb{R}^p$ and form the $(p+q)$-vector $(u^T,0^T)^T$. Then $$ \begin{pmatrix}u\\0\end{pmatrix}^{T} \begin{pmatrix}\Sigma_{YY}&\Sigma_{YX}\\\Sigma_{XY}&\Sigma_{XX}\end{pmatrix} \begin{pmatrix}u\\0\end{pmatrix} =u^T\Sigma_{YY}u\geq 0. $$ Thus $\Sigma_{YY}$ is positive semidefinite. For any $v\in\mathbb{R}^q$, using $(0^T,v^T)^T$ instead gives $$ \begin{pmatrix}0\\v\end{pmatrix}^{T} \begin{pmatrix}\Sigma_{YY}&\Sigma_{YX}\\\Sigma_{XY}&\Sigma_{XX}\end{pmatrix} \begin{pmatrix}0\\v\end{pmatrix} =v^T\Sigma_{XX}v\geq 0. $$ Therefore $\Sigma_{XX}$ is positive semidefinite as well. $\square$

Worked Block Calculations

Worked Example: Scalar Response and Scalar Conditioning Variable

Let $A$ and $B$ be independent $N(0,1)$ random variables. Define the scalar response $Y=2A$ and the scalar conditioning variable $X=\frac{3}{2}A+\frac{\sqrt{7}}{2}B$. Find the joint mean and covariance in the order $(Y,X)^T$, and identify each block.

Both variables have mean zero. Their variances are $$ \operatorname{Var}(Y)=4\operatorname{Var}(A)=4, $$ and, using independence, $$ \operatorname{Var}(X) =\frac{9}{4}\operatorname{Var}(A)+\frac{7}{4}\operatorname{Var}(B) +2\left(\frac{3}{2}\right)\left(\frac{\sqrt 7}{2}\right)\operatorname{Cov}(A,B) =\frac{9}{4}+\frac{7}{4}=4. $$ Their covariance is $$ \operatorname{Cov}(Y,X) =2\left(\frac{3}{2}\right)\operatorname{Var}(A) +2\left(\frac{\sqrt 7}{2}\right)\operatorname{Cov}(A,B) =3. $$ Thus $p=q=1$, and $$ E\begin{bmatrix}Y\\X\end{bmatrix} =\begin{bmatrix}0\\0\end{bmatrix},\qquad \operatorname{Cov}\begin{pmatrix}Y\\X\end{pmatrix} =\begin{pmatrix}4&3\\3&4\end{pmatrix}. $$ Here $\Sigma_{YY}=[4]$, $\Sigma_{YX}=[3]$, $\Sigma_{XY}=[3]$, and $\Sigma_{XX}=[4]$. The matrix is symmetric, and its determinant is $16-9=7>0$, consistent with a valid nondegenerate covariance matrix.

Worked Example: A Two-Dimensional Response Block

Let $U_1,U_2,U_3$ be independent $N(0,1)$ variables, and define $Y_1=U_1$, $Y_2=2U_1+U_2$, and $X=-U_1+U_3$. Take $Y=(Y_1,Y_2)^T$ as the response block and $X$ as the scalar conditioning block. Calculate and check the block covariance matrix.

The within-response entries are $\operatorname{Var}(Y_1)=1$, $\operatorname{Cov}(Y_1,Y_2)=2$, and $\operatorname{Var}(Y_2)=4+1=5$. Hence $\Sigma_{YY}=\begin{pmatrix}1&2\\2&5\end{pmatrix}$. For the cross-covariances, $$ \operatorname{Cov}(Y_1,X)=\operatorname{Cov}(U_1,-U_1+U_3)=-1, $$ $$ \operatorname{Cov}(Y_2,X)=\operatorname{Cov}(2U_1+U_2,-U_1+U_3)=-2. $$ Also, $\operatorname{Var}(X)=1+1=2$. Therefore $$ \Sigma_{YX}=\begin{pmatrix}-1\\-2\end{pmatrix},\quad \Sigma_{XY}=\begin{pmatrix}-1&-2\end{pmatrix},\quad \Sigma_{XX}=\begin{pmatrix}2\end{pmatrix}, $$ and the full matrix is $$ \begin{pmatrix} 1&2&-1\\ 2&5&-2\\ -1&-2&2 \end{pmatrix}. $$ The dimensions are $2\times2$, $2\times1$, $1\times2$, and $1\times1$, respectively. The cross-blocks are transposes, and the full matrix is symmetric. Its validity also follows directly from the construction using independent normal variables.

Worked Example: Partitioning Nonconsecutive Coordinates

Let $A,B,C,D$ be independent $N(0,1)$ variables and define an ordered four-vector $W=(W_1,W_2,W_3,W_4)^T$ by $W_1=A$, $W_2=A+C$, $W_3=2A+B$, and $W_4=B+D$. Choose $Y=(W_1,W_3)^T$ and $X=(W_2,W_4)^T$. Find the block covariance matrix in the order $(Y^T,X^T)^T$.

The chosen order is $(W_1,W_3,W_2,W_4)^T$, not the original order of $W$. Within $Y$, the variances are $1$ and $4+1=5$, and the covariance is $2$. Within $X$, both variances are $2$, while $\operatorname{Cov}(W_2,W_4)=0$. Across the blocks, $$ \operatorname{Cov}(W_1,W_2)=1,\quad \operatorname{Cov}(W_1,W_4)=0,\quad \operatorname{Cov}(W_3,W_2)=2,\quad \operatorname{Cov}(W_3,W_4)=1. $$ Consequently, $$ \Sigma_{YY}=\begin{pmatrix}1&2\\2&5\end{pmatrix},\quad \Sigma_{YX}=\begin{pmatrix}1&0\\2&1\end{pmatrix},\quad \Sigma_{XY}=\begin{pmatrix}1&2\\0&1\end{pmatrix},\quad \Sigma_{XX}=\begin{pmatrix}2&0\\0&2\end{pmatrix}. $$ Each block is $2\times2$; the cross-blocks satisfy $\Sigma_{XY}=\Sigma_{YX}^T$. The full covariance in the selected order is $$ \begin{pmatrix} 1&2&1&0\\ 2&5&2&1\\ 1&2&2&0\\ 0&1&0&2 \end{pmatrix}. $$ It is symmetric. Its covariance validity is guaranteed by the explicit construction from $A,B,C,D$. This example shows why the partition order must be written down before extracting submatrices.

Common Mistake: Reversing the Cross-Block

A frequent error is to put $\Sigma_{XY}$ in the upper-right position while keeping the vector ordered as $(Y^T,X^T)^T$. That placement can even be dimensionally impossible: if $p=2$ and $q=1$, then $\Sigma_{XY}$ is $1\times2$, but the upper-right block must have $2$ rows and $1$ column. The correction is to keep the argument order aligned with the row and column groups: the upper-right block is $\operatorname{Cov}(Y,X)=\Sigma_{YX}$, and the lower-left block is $\operatorname{Cov}(X,Y)=\Sigma_{XY}=\Sigma_{YX}^T$.

Another mistake is to assume that a covariance block must be positive definite. Covariance matrices and their principal blocks are guaranteed to be positive semidefinite, not necessarily positive definite. A zero-variance coordinate, or a linear dependence among coordinates, can make a block singular without making it invalid.

Key takeaway: Write the ordered vector first. Then label each block by its row and column variables, check its dimensions, and verify that the full covariance is symmetric with positive semidefinite diagonal blocks.

Check Your Understanding

Use the block order shown in each question, and justify your dimension checks.

If $Y$ has dimension $3$ and $X$ has dimension $2$, what are the dimensions of $\Sigma_{YY}$, $\Sigma_{YX}$, $\Sigma_{XY}$, and $\Sigma_{XX}$?
For the order $(Y^T,X^T)^T$, which covariance block occupies the upper-right position? Why must the lower-left block be its transpose?
Explain why each diagonal block of a valid joint covariance matrix must be positive semidefinite.
Suppose a proposed $2\times1$ upper-right block is labelled $\Sigma_{XY}$, where $Y$ is two-dimensional and $X$ is one-dimensional. Identify the error.
Can a valid diagonal covariance block be singular? Give a reason for your answer.

Partitioned Gaussian Vectors and Block Covariance Notation

What You'll Learn