Block Covariance Structure of Joint Gaussian Vectors

Why Organize a Gaussian Vector into Blocks?

A multivariate measurement often contains two related collections of variables: for example, measurements taken at two visits, two groups of clinical outcomes, or a response together with several predictors. If the collections are represented by vectors $X$ and $Y$, their joint distribution is described not only by the variation within each collection but also by how the collections vary together. A block representation keeps those pieces visible.

The goal here is to read and construct this representation accurately. The diagonal blocks describe variation within each subvector; the off-diagonal blocks describe covariance across the two subvectors. Their dimensions and consistency matter: not every collection of four matrices can be a valid covariance matrix.

Definition: Let $X$ be a random vector in $\mathbb{R}^p$ and $Y$ a random vector in $\mathbb{R}^q$, with finite second moments. Write their means as $\mu_X=E[X]$ and $\mu_Y=E[Y]$. Their cross-covariance matrix is $$ \operatorname{Cov}(X,Y)=E[(X-\mu_X)(Y-\mu_Y)^T], $$ a $p$-by-$q$ matrix. The joint covariance matrix of the concatenated vector $(X^T,Y^T)^T$ has diagonal blocks $\operatorname{Cov}(X)$ and $\operatorname{Cov}(Y)$, and off-diagonal blocks $\operatorname{Cov}(X,Y)$ and its transpose.

If the concatenated vector is jointly Gaussian, we can summarize its distribution by its joint mean and covariance. In the notation used here, its mean has $p+q$ entries, and its covariance has $p+q$ rows and columns. The coordinate marginals are Gaussian, as in the earlier material on marginal Gaussian laws; the block form adds an explicit account of how the two subvectors covary.

The Partitioned Mean and Covariance

Set $Z=(X^T,Y^T)^T$. Its mean and covariance can be written

E[Z]= \begin{pmatrix} \mu_X\\ \mu_Y \end{pmatrix}, \qquad \operatorname{Cov}(Z)= \begin{pmatrix} \Sigma_{XX} & \Sigma_{XY}\\ \Sigma_{YX} & \Sigma_{YY} \end{pmatrix},

where $\Sigma_{XX}=\operatorname{Cov}(X)$ is $p$-by-$p$, $\Sigma_{YY}=\operatorname{Cov}(Y)$ is $q$-by-$q$, and $\Sigma_{XY}=\operatorname{Cov}(X,Y)$ is $p$-by-$q$. The remaining block is $\Sigma_{YX}=\operatorname{Cov}(Y,X)$, which is $q$-by-$p$. Because covariance is symmetric after the two subvectors are concatenated, $\Sigma_{YX}=\Sigma_{XY}^T$.

The entries of $\Sigma_{XY}$ are pairwise covariances: its entry in row $i$, column $j$ is $\operatorname{Cov}(X_i,Y_j)$. A positive entry means that the two centered coordinates tend to vary in the same direction in the covariance sense; a negative entry means they tend to vary in opposite directions. An entry of zero states that the corresponding covariance is zero. The full block is not generally symmetric by itself, since it may be rectangular; symmetry applies to the whole joint covariance matrix.

A covariance matrix must also be positive semidefinite. In block form this requires, for every $a\in\mathbb{R}^p$ and $b\in\mathbb{R}^q$,

\begin{pmatrix}a^T & b^T\end{pmatrix} \begin{pmatrix} \Sigma_{XX} & \Sigma_{XY}\\ \Sigma_{YX} & \Sigma_{YY} \end{pmatrix} \begin{pmatrix}a\\b\end{pmatrix} \geq 0.

This condition says that every linear combination of the coordinates has nonnegative variance. It is a necessary consistency check on all four blocks together, not just on the two diagonal blocks individually. A singular covariance is allowed: positive semidefinite does not mean positive definite.

Theorem: For finite-second-moment random vectors $X$ and $Y$, the covariance of their concatenation is $$ \operatorname{Cov}\!\begin{pmatrix}X\\Y\end{pmatrix} = \begin{pmatrix} \operatorname{Cov}(X) & \operatorname{Cov}(X,Y)\\ \operatorname{Cov}(Y,X) & \operatorname{Cov}(Y) \end{pmatrix}. $$ It is symmetric and positive semidefinite. Moreover, for every $a\in\mathbb{R}^p$ and $b\in\mathbb{R}^q$, $$ \left|a^T\operatorname{Cov}(X,Y)b\right| \leq \sqrt{a^T\operatorname{Cov}(X)a}\, \sqrt{b^T\operatorname{Cov}(Y)b}. $$

Proof. Let $\mu_X=E[X]$, $\mu_Y=E[Y]$, and $Z=(X^T,Y^T)^T$. Then $Z-E[Z]$ is the vector formed by stacking $X-\mu_X$ and $Y-\mu_Y$. Multiplying this vector by its transpose and taking expectations gives four blocks. The upper-right block is $E[(X-\mu_X)(Y-\mu_Y)^T]=\operatorname{Cov}(X,Y)$; the other three blocks follow from the same definition. Also,

\operatorname{Cov}(Y,X) =E[(Y-\mu_Y)(X-\mu_X)^T] =\operatorname{Cov}(X,Y)^T.

For any vector $c\in\mathbb{R}^{p+q}$, the quadratic form of the joint covariance is

c^T\operatorname{Cov}(Z)c =E\!\left[\{c^T(Z-E[Z])\}^2\right]\geq 0.

This proves positive semidefiniteness. Finally, set $U=a^T(X-\mu_X)$ and $V=b^T(Y-\mu_Y)$. Then $E[UV]=a^T\operatorname{Cov}(X,Y)b$, $E[U^2]=a^T\operatorname{Cov}(X)a$, and $E[V^2]=b^T\operatorname{Cov}(Y)b$. The Cauchy–Schwarz inequality for square-integrable random variables gives $|E[UV]|\leq\sqrt{E[U^2]E[V^2]}$, which is the stated bound. $\square$

Worked Block Models

Worked Example: Calculating Blocks from a Gaussian Construction

Let $Z_1,Z_2,Z_3$ be independent $N(0,1)$ variables, and define $X=Z_1+Z_2$ and $Y=2Z_1-Z_2+Z_3$. Find the joint mean and covariance in block form.

Both variables have mean zero, so the joint mean is $(0,0)^T$. For the first diagonal block, $$ \operatorname{Var}(X)=\operatorname{Var}(Z_1)+\operatorname{Var}(Z_2)=1+1=2. $$ For the second, $$ \operatorname{Var}(Y)=4\operatorname{Var}(Z_1)+\operatorname{Var}(Z_2)+\operatorname{Var}(Z_3)=4+1+1=6. $$ Independence makes the covariances between distinct $Z_i$'s zero. Thus the cross-covariance is $$ \operatorname{Cov}(X,Y) =\operatorname{Cov}(Z_1+Z_2,2Z_1-Z_2+Z_3) =2\operatorname{Var}(Z_1)-\operatorname{Var}(Z_2)=2-1=1. $$ The joint mean and covariance are therefore $$ E\!\begin{pmatrix}X\\Y\end{pmatrix} =\begin{pmatrix}0\\0\end{pmatrix}, \qquad \operatorname{Cov}\!\begin{pmatrix}X\\Y\end{pmatrix} =\begin{pmatrix}2&1\\1&6\end{pmatrix}. $$ The off-diagonal entry is positive: in this model the shared $Z_1$ component contributes positively to both variables, while the $Z_2$ contribution has opposite signs and partly offsets it.

Worked Example: Two Measurements and a Shared Component

Let $U,E_1,E_2,V$ be independent $N(0,1)$ variables. Define the two-dimensional subvector $X=(U+E_1,U+E_2)^T$ and the scalar subvector $Y=2U+V$. Compute all covariance blocks.

The means are zero. Each coordinate of $X$ has variance $1+1=2$, and their covariance is $\operatorname{Cov}(U+E_1,U+E_2)=\operatorname{Var}(U)=1$. Hence $$ \Sigma_{XX}=\begin{pmatrix}2&1\\1&2\end{pmatrix}. $$ For $Y$, $\Sigma_{YY}=\operatorname{Var}(2U+V)=4+1=5$. Each coordinate of $X$ has covariance $2$ with $Y$, because its only shared component with $Y$ is $U$: $$ \operatorname{Cov}(U+E_1,2U+V)=2,\qquad \operatorname{Cov}(U+E_2,2U+V)=2. $$ Thus the complete block covariance is $$ \operatorname{Cov}\!\begin{pmatrix}X\\Y\end{pmatrix} = \begin{pmatrix} 2&1&2\\ 1&2&2\\ 2&2&5 \end{pmatrix}, $$ with $\Sigma_{XY}=(2,2)^T$ and $\Sigma_{YX}=(2,2)$. The two diagonal blocks describe variability within the measurements and within $Y$; the cross-block entries describe how each measurement covaries with $Y$.

Worked Example: Checking Whether a Cross-Covariance Is Possible

Suppose two scalar Gaussian variables are proposed to have variances $4$ and $9$, with covariance $7$. Could these numbers form a covariance matrix?

For scalar blocks, the theorem's bound with $a=b=1$ requires $$ |\operatorname{Cov}(X,Y)| \leq \sqrt{\operatorname{Var}(X)\operatorname{Var}(Y)} =\sqrt{4\cdot 9}=6. $$ The proposed covariance has absolute value $7$, which exceeds $6$. The proposed matrix $$ \begin{pmatrix}4&7\\7&9\end{pmatrix} $$ cannot be a covariance matrix, Gaussian or otherwise.

For comparison, covariance $3$ is attainable with the same variances. Let $Z_1,Z_2$ be independent $N(0,1)$ variables, and set $X=2Z_1$, $Y=\frac{3}{2}Z_1+\frac{3\sqrt{3}}{2}Z_2$. Then $\operatorname{Var}(X)=4$, $\operatorname{Cov}(X,Y)=3$, and $$ \operatorname{Var}(Y)=\frac{9}{4}+\frac{27}{4}=9. $$ So the resulting covariance matrix is $\begin{pmatrix}4&3\\3&9\end{pmatrix}$. This construction verifies attainability rather than relying only on a numerical matrix check.

Interpreting and Using the Blocks

Block notation is useful because it separates questions about within-subvector variation from questions about cross-subvector association. In a repeated-measurement model, for instance, $\Sigma_{XX}$ could describe covariance among outcomes at one visit, while $\Sigma_{YY}$ describes covariance among outcomes at another. The cross-block $\Sigma_{XY}$ records which coordinates from the first collection covary with which coordinates from the second.

A block may be rectangular, and its entries need not be variances or correlations. They are covariances in the units of the paired variables. If variables have different scales, a large covariance need not indicate a stronger standardized association than a smaller covariance elsewhere. The matrix dimensions are also a practical diagnostic: if $X$ has $p$ coordinates and $Y$ has $q$, then $\Sigma_{XY}$ must have $p$ rows and $q$ columns.

Two checks are especially useful when specifying a joint Gaussian model. First, the whole matrix must be symmetric, so the lower-left block must be the transpose of the upper-right block. Second, the whole matrix must be positive semidefinite, so every linear combination of the coordinates has nonnegative variance. The covariance bound provides a quick necessary check for any pair of linear contrasts, even when the blocks are large.

Common Mistake

A tempting but incorrect shortcut is to choose any two valid diagonal covariance blocks and then fill in the cross-block entries independently. For example, scalar variances $4$ and $9$ are each valid on their own, but covariance $7$ is impossible: it violates the bound $|\operatorname{Cov}(X,Y)|\leq 6$. Valid diagonal blocks do not guarantee a valid joint covariance.

The correction is to treat the partitioned matrix as one covariance matrix. Check dimensions, transpose symmetry, and positive semidefiniteness of the full matrix. When a proposed cross-block entry is questionable, the covariance bound can reject it immediately; when constructing a model, calculate all blocks from a coherent joint model and verify the resulting matrix.

Key takeaway: For jointly Gaussian subvectors, the joint mean stacks the two means, and the covariance matrix places within-subvector covariances on the diagonal and cross-covariances in the off-diagonal blocks. The cross-block dimensions, transpose relationship, and compatibility with positive semidefiniteness are essential parts of the representation.

Check Your Understanding

Use block dimensions and covariance identities to answer the following.

If $X\in\mathbb{R}^3$ and $Y\in\mathbb{R}^2$, what are the dimensions of $\Sigma_{XX}$, $\Sigma_{XY}$, and $\Sigma_{YY}$?
Why must $\Sigma_{YX}$ equal $\Sigma_{XY}^T$, even when $\Sigma_{XY}$ is not square?
For scalar variables with variances $16$ and $25$, what is the largest possible absolute covariance?
In a model where two coordinates share a component with the same coefficient sign, how can that shared component contribute to their cross-covariance?
Why does checking that each diagonal block is positive semidefinite not by itself establish that the full block covariance is valid?