Block Covariance and Cross-Covariance

Why Partition a Covariance Matrix?

A random vector often groups variables of different roles: measurements on the same subject, predictors and responses, or two sets of features. Its covariance matrix records all pairwise covariances at once. Partitioning that matrix makes the structure easier to read: the diagonal blocks describe variation within each group, while the off-diagonal blocks describe how the groups vary together.

The central bookkeeping point is that a cross-covariance block is generally rectangular. Its rows index coordinates of one subvector and its columns index coordinates of the other. Its transpose appears in the opposite position of the full covariance matrix. Once those dimensions are clear, variances and covariances of sums or other linear combinations follow by matrix multiplication.

Definition and Block Structure

Let \(X\) be a \(p\)-dimensional random vector and \(Y\) a \(q\)-dimensional random vector, both defined on the same probability space. Assume every coordinate of both vectors has a finite second moment. Write their means as \(\mu_X=E[X]\) and \(\mu_Y=E[Y]\).

Definition: The covariance matrix of \(X\) is \(\Sigma_X=E[(X-\mu_X)(X-\mu_X)^T]\), and the covariance matrix of \(Y\) is \(\Sigma_Y=E[(Y-\mu_Y)(Y-\mu_Y)^T]\). The cross-covariance matrix of \(X\) and \(Y\) is \(\Sigma_{XY}=E[(X-\mu_X)(Y-\mu_Y)^T]\). It has \(p\) rows and \(q\) columns.

The entry in row \(i\), column \(j\) of \(\Sigma_{XY}\) is \(\operatorname{Cov}(X_i,Y_j)\). Reversing the order gives \(\Sigma_{YX}=E[(Y-\mu_Y)(X-\mu_X)^T]\), a \(q\)-by-\(p\) matrix. Since transposing the outer product reverses its factors, \(\Sigma_{YX}=\Sigma_{XY}^T\).

Stack the subvectors into \(Z=(X^T,Y^T)^T\), a vector of length \(p+q\). Its covariance matrix is partitioned as follows.

\operatorname{Var}(Z) = \begin{pmatrix} \Sigma_X & \Sigma_{XY}\\ \Sigma_{XY}^T & \Sigma_Y \end{pmatrix}.

The diagonal blocks are the covariances within \(X\) and within \(Y\). The upper-right block contains covariances between coordinates of \(X\) and coordinates of \(Y\); the lower-left block contains the same information in the reverse order. The full matrix is symmetric, but \(\Sigma_{XY}\) need not be square or symmetric.

Worked Example: Reading a Scalar Block Matrix

Let \(A\) and \(B\) be independent random variables with mean zero and variance one. Define \(X=A+B\) and \(Y=2A-B\). Find the covariance matrix of the vector \((X,Y)^T\).

First, independence and centering give \(\operatorname{Cov}(A,B)=0\). Therefore \[ \operatorname{Var}(X)=\operatorname{Var}(A)+\operatorname{Var}(B)=1+1=2. \] Similarly, \[ \operatorname{Var}(Y)=4\operatorname{Var}(A)+\operatorname{Var}(B)=4+1=5. \] For the cross-covariance, expand both linear combinations: \[ \operatorname{Cov}(X,Y) =\operatorname{Cov}(A+B,2A-B) =2\operatorname{Var}(A)-\operatorname{Cov}(A,B) +2\operatorname{Cov}(B,A)-\operatorname{Var}(B) =2-1=1. \] Thus the two diagonal blocks are the scalar variances \(2\) and \(5\), and the cross-covariance block is the scalar \(1\). The full covariance matrix is \[ \begin{pmatrix}2&1\\1&5\end{pmatrix}. \] The lower-left entry is the transpose of the upper-right entry; for scalars, transposing changes nothing.

Covariances of Linear Combinations

The block representation is useful because it lets us compute the covariance of new vectors without returning to every pair of coordinates. Let \(A\) be an \(r\)-by-\(p\) matrix and \(B\) an \(r\)-by-\(q\) matrix, and form \(U=AX+BY\). Each coordinate of \(U\) combines coordinates from both subvectors.

Theorem: If \(X\) and \(Y\) have finite second moments, then \[ \operatorname{Var}(AX+BY) =A\Sigma_XA^T+A\Sigma_{XY}B^T +B\Sigma_{XY}^TA^T+B\Sigma_YB^T. \] More generally, if \(V=DX+EY\), where \(D\) is \(s\)-by-\(p\) and \(E\) is \(s\)-by-\(q\), then \[ \operatorname{Cov}(AX+BY,DX+EY) =A\Sigma_XD^T+A\Sigma_{XY}E^T +B\Sigma_{XY}^TD^T+B\Sigma_YE^T. \]

Proof. Center each vector: \(X_c=X-\mu_X\) and \(Y_c=Y-\mu_Y\). The centered version of \(AX+BY\) is \(AX_c+BY_c\). Using the definition of covariance and distributing the matrix products gives \[ \begin{aligned} \operatorname{Var}(AX+BY) &=E[(AX_c+BY_c)(AX_c+BY_c)^T]\\ &=E[AX_cX_c^TA^T]+E[AX_cY_c^TB^T]\\ &\quad+E[BY_cX_c^TA^T]+E[BY_cY_c^TB^T]\\ &=A\Sigma_XA^T+A\Sigma_{XY}B^T +B\Sigma_{XY}^TA^T+B\Sigma_YB^T. \end{aligned} \] Each expectation is finite because the coordinates have finite second moments. For the more general covariance formula, multiply the centered vectors \(AX_c+BY_c\) and \(DX_c+EY_c\) in that order, then take expectations. The four resulting terms are \(A\Sigma_XD^T\), \(A\Sigma_{XY}E^T\), \(B\Sigma_{XY}^TD^T\), and \(B\Sigma_YE^T\), respectively. \(\square\)

Equivalently, define the combined matrix \(M=(A\ \ B)\), formed by placing \(A\) and \(B\) side by side. Then \(\operatorname{Var}(AX+BY)=M\operatorname{Var}(Z)M^T\). This compact form and the expanded formula express the same calculation; the expanded form shows which covariance block contributes to each term.

Worked Example: Variance and Covariance from Two Vector Blocks

Let \(A,B,C\) be independent, mean-zero random variables with variance one, and define \[ X=\begin{pmatrix}A\\A+B\end{pmatrix}, \qquad Y=\begin{pmatrix}A+C\\B\end{pmatrix}. \] Find \(\Sigma_X\), \(\Sigma_Y\), and \(\Sigma_{XY}\). Then calculate the variance of \(S=X_1+2X_2\), the variance of \(T=Y_1-Y_2\), and \(\operatorname{Cov}(S,T)\).

For \(X\), the coordinate variances are \(\operatorname{Var}(A)=1\) and \(\operatorname{Var}(A+B)=2\), while \(\operatorname{Cov}(A,A+B)=1\). Thus \[ \Sigma_X=\begin{pmatrix}1&1\\1&2\end{pmatrix}. \] For \(Y\), \(\operatorname{Var}(A+C)=2\), \(\operatorname{Var}(B)=1\), and \(\operatorname{Cov}(A+C,B)=0\), so \[ \Sigma_Y=\begin{pmatrix}2&0\\0&1\end{pmatrix}. \] The \((i,j)\) entry of \(\Sigma_{XY}\) is \(\operatorname{Cov}(X_i,Y_j)\). The four entries are \[ \operatorname{Cov}(A,A+C)=1,\quad \operatorname{Cov}(A,B)=0,\quad \operatorname{Cov}(A+B,A+C)=1,\quad \operatorname{Cov}(A+B,B)=1. \] Consequently, \[ \Sigma_{XY}=\begin{pmatrix}1&0\\1&1\end{pmatrix}. \]

Write \(a=(1,2)\) and \(b=(1,-1)\), so \(S=aX\) and \(T=bY\). The block formula gives \[ \operatorname{Var}(S)=a\Sigma_Xa^T =(1,2)\begin{pmatrix}3\\5\end{pmatrix}=13. \] For \(T\), \[ \operatorname{Var}(T)=b\Sigma_Yb^T =(1,-1)\begin{pmatrix}2\\-1\end{pmatrix}=3. \] The cross-covariance is \[ \operatorname{Cov}(S,T)=a\Sigma_{XY}b^T =(1,2)\begin{pmatrix}1\\0\end{pmatrix}=1. \] As a check, \(S=3A+2B\) and \(T=A-B+C\), so their variances are \(9+4=13\) and \(1+1+1=3\); their covariance is \(3-2=1\). The direct calculations agree with the block calculation.

Covariance Between Transformed Subvectors

The general covariance formula is particularly useful when two outputs are different. For instance, one output might be a scalar summary of \(X\), while the other retains several coordinates of \(Y\). Their covariance is then a vector whose entries are the covariances of the scalar summary with each coordinate of the second output.

Worked Example: A Scalar Summary and a Vector Output

Use the preceding model and its covariance blocks. Let \(R=X_1+X_2\) and let \(Q=Y\). Find \(\operatorname{Var}(R)\) and the cross-covariance between \(R\) and \(Q\).

Here \(R=aX\) with \(a=(1,1)\), while \(Q=Y\) corresponds to \(D\) equal to the zero \(2\)-by-\(2\) matrix and \(E=I_2\), the \(2\)-by-\(2\) identity matrix. Its variance is \[ \operatorname{Var}(R)=a\Sigma_Xa^T =(1,1)\begin{pmatrix}2\\3\end{pmatrix}=5. \] For the cross-covariance, the general formula reduces to \[ \operatorname{Cov}(R,Q)=a\Sigma_{XY} =(1,1)\begin{pmatrix}1&0\\1&1\end{pmatrix} =(2,1). \] This is a one-by-two row matrix: its first entry is \(\operatorname{Cov}(R,Y_1)=2\), and its second is \(\operatorname{Cov}(R,Y_2)=1\). Indeed, \(R=2A+B\), so \(\operatorname{Cov}(R,A+C)=2\) and \(\operatorname{Cov}(R,B)=1\).

Reading the Blocks Correctly

A few distinctions prevent common errors when working with block covariance matrices:

Within-block versus between-block covariance. \(\Sigma_X\) describes covariances among coordinates of \(X\); \(\Sigma_Y\) describes covariances among coordinates of \(Y\); \(\Sigma_{XY}\) describes covariances between the two groups.
Transpose versus equality. The opposite off-diagonal block is \(\Sigma_{XY}^T\), not generally \(\Sigma_{XY}\). The two blocks have different dimensions whenever \(p\ne q\).
Covariance versus independence. A zero cross-covariance matrix means every coordinate pair has covariance zero. It does not, in general, establish that \(X\) and \(Y\) are independent. Independence does imply zero cross-covariance when the required second moments are finite, but the converse needs additional assumptions.
All four terms matter for a sum. The variance of \(AX+BY\) includes two cross terms. Omitting them is valid only when their contribution is zero, for example when \(\Sigma_{XY}=0\).

Key takeaway: In a partitioned covariance matrix, the diagonal blocks describe variation within each subvector and the off-diagonal blocks describe cross-covariation. To calculate covariance after linear combinations, preserve the block order and use the transpose that matches each matrix dimension.

Check Your Understanding

Try these questions using the definitions and block formulas above.

If \(X\) has dimension 3 and \(Y\) has dimension 2, what are the dimensions of \(\Sigma_{XY}\) and \(\Sigma_{YX}\)?
Why must the lower-left block of \(\operatorname{Var}((X^T,Y^T)^T)\) be the transpose of the upper-right block?
For scalar random variables \(X\) and \(Y\), write \(\operatorname{Var}(2X-Y)\) using \(\operatorname{Var}(X)\), \(\operatorname{Var}(Y)\), and \(\operatorname{Cov}(X,Y)\).
Suppose \(\Sigma_{XY}=0\). What does this say about coordinate covariances, and what does it not generally say about independence?
For \(U=AX+BY\), which two terms in its variance formula represent cross-covariation between \(X\) and \(Y\)?