Cross-Covariance Blocks and Dependence

Why Partition a Covariance Matrix?

A multivariate observation often contains two natural groups of measurements: for example, a vector of baseline variables and a vector of outcomes. The covariance matrix of the combined vector records variation within each group as well as how the groups vary together. Separating these parts makes the joint structure easier to read.

The key interpretive caution is that covariance describes linear co-variation. A zero cross-covariance block says that every coordinate in one group has zero covariance with every coordinate in the other. It does not, without additional assumptions, establish that the groups are independent.

Definition: Let $X$ be a random vector in $\mathbb{R}^p$ and $Y$ a random vector in $\mathbb{R}^q$, with finite second moments. Write their means as $\mu_X=E[X]$ and $\mu_Y=E[Y]$. Their cross-covariance matrix is $$ \Sigma_{XY}=\operatorname{Cov}(X,Y)=E\left[(X-\mu_X)(Y-\mu_Y)^T\right], $$ a $p\times q$ matrix. The covariance matrix of the concatenated vector $Z=(X^T,Y^T)^T$ is partitioned as $$ \operatorname{Cov}(Z)= \begin{pmatrix} \Sigma_{XX} & \Sigma_{XY}\\ \Sigma_{YX} & \Sigma_{YY} \end{pmatrix}, $$ where $\Sigma_{XX}=\operatorname{Cov}(X)$, $\Sigma_{YY}=\operatorname{Cov}(Y)$, and $\Sigma_{YX}=\Sigma_{XY}^T$.

The $(i,j)$ entry of $\Sigma_{XY}$ is $\operatorname{Cov}(X_i,Y_j)$. Thus each entry links one coordinate from the first group to one from the second. The block need not be square: its number of rows is the number of coordinates in $X$, and its number of columns is the number in $Y$. Reversing the order of the groups transposes the cross-covariance block.

What the Blocks Must Satisfy

The diagonal blocks describe variation within each group; the off-diagonal blocks describe cross-group covariance. The full matrix is symmetric, but a cross-covariance block by itself generally is not. A basic constraint on the full matrix follows from the fact that every scalar linear combination of a random vector has nonnegative variance.

Theorem: If $X$ and $Y$ have finite second moments, their block covariance matrix is positive semidefinite. In particular, for all $a\in\mathbb{R}^p$ and $b\in\mathbb{R}^q$, $$ \begin{pmatrix}a^T&b^T\end{pmatrix} \begin{pmatrix} \Sigma_{XX} & \Sigma_{XY}\\ \Sigma_{YX} & \Sigma_{YY} \end{pmatrix} \begin{pmatrix}a\\b\end{pmatrix} = \operatorname{Var}(a^T X+b^T Y)\geq 0. $$

Proof. By the definition of covariance, the covariance matrix of $Z=(X^T,Y^T)^T$ is $E[(Z-E[Z])(Z-E[Z])^T]$. For any vector $c=(a^T,b^T)^T$, multiplication on the left and right gives $$ c^T\operatorname{Cov}(Z)c =E\left[\{c^T(Z-E[Z])\}^2\right] =\operatorname{Var}(a^T X+b^T Y)\geq 0. $$ The expectation exists because $X$ and $Y$ have finite second moments. Since the quadratic form is nonnegative for every $c$, the block covariance matrix is positive semidefinite. $\square$

This result is useful when checking a proposed covariance matrix: its blocks cannot be chosen independently without regard to the whole matrix. The diagonal blocks must be valid covariance matrices, and the cross-blocks must fit with them so that every combined linear contrast has nonnegative variance.

Worked Example: A Shared Random Component Creates Cross-Covariance

Let $A$, $B$, and $C$ be mutually independent, each with mean zero and variance one. Define $X=(A+B,A-B)^T$ and $Y=A+C$. Find the block covariance matrix and interpret its cross-block.

All three variables have mean zero, so the vector means are also zero. For the first within-group block, $$ \operatorname{Var}(A+B)=1+1=2,\qquad \operatorname{Var}(A-B)=1+1=2, $$ and $$ \operatorname{Cov}(A+B,A-B) =\operatorname{Var}(A)-\operatorname{Var}(B)=0. $$ Therefore $\Sigma_{XX}=\begin{pmatrix}2&0\\0&2\end{pmatrix}$. Also, $\Sigma_{YY}=\operatorname{Var}(A+C)=2$.

The cross-covariances are $$ \operatorname{Cov}(A+B,A+C)=\operatorname{Var}(A)=1,\qquad \operatorname{Cov}(A-B,A+C)=\operatorname{Var}(A)=1. $$ All terms involving distinct independent, mean-zero variables vanish. Hence $$ \operatorname{Cov}\begin{pmatrix}X\\Y\end{pmatrix} = \begin{pmatrix} 2&0&1\\ 0&2&1\\ 1&1&2 \end{pmatrix}. $$ The cross-block is $\begin{pmatrix}1\\1\end{pmatrix}$: both coordinates of $X$ have positive covariance with $Y$. In this construction, the shared component $A$ contributes to each of those covariances.

Independence Implies a Zero Cross-Block

Independence is a statement about the joint distribution, not merely a collection of moment calculations. It does, however, have a direct covariance consequence when the relevant second moments exist.

Theorem: If the random vectors $X$ and $Y$ are independent and have finite second moments, then $\Sigma_{XY}=0$.

Proof. Independence of the vectors implies independence of $X_i$ and $Y_j$ for every pair of coordinates. Finite second moments ensure that these coordinates have finite first moments and that $E[|X_iY_j|]$ is finite; for example, this follows from the Cauchy–Schwarz inequality. Independence therefore gives $$ E[X_iY_j]=E[X_i]E[Y_j], $$ so $$ \operatorname{Cov}(X_i,Y_j) =E[X_iY_j]-E[X_i]E[Y_j]=0. $$ This holds for every $i=1,\ldots,p$ and $j=1,\ldots,q$, so every entry of $\Sigma_{XY}$ is zero. $\square$

The reverse implication is not a general theorem: a zero block only summarizes coordinatewise covariances. It says nothing by itself about other features of the joint distribution. In particular, it does not assert that the conditional distribution of one group is unchanged when the other group is observed.

Worked Example: Independent Groups Give a Zero Cross-Block

Let $A,E,B,F$ be mutually independent random variables, each with mean zero and variance one. Set $X=(A,A+E)^T$ and $Y=(B,B+F)^T$. Compute the within-group and cross-group blocks.

Within $X$, $\operatorname{Var}(A)=1$, $\operatorname{Cov}(A,A+E)=1$, and $\operatorname{Var}(A+E)=2$. Thus $$ \Sigma_{XX}= \begin{pmatrix}1&1\\1&2\end{pmatrix}. $$ The same calculation gives $\Sigma_{YY}=\begin{pmatrix}1&1\\1&2\end{pmatrix}$.

Each coordinate of $X$ is a function of $A,E$, while each coordinate of $Y$ is a function of $B,F$. The two underlying groups are independent, so every cross-covariance is zero. The complete block matrix is $$ \operatorname{Cov}\begin{pmatrix}X\\Y\end{pmatrix} = \begin{pmatrix} 1&1&0&0\\ 1&2&0&0\\ 0&0&1&1\\ 0&0&1&2 \end{pmatrix}. $$ Here the zero cross-block is consistent with independence, while the nonzero entries within each diagonal block describe variation within a group.

A Zero Cross-Block Does Not Establish Independence

A scalar case already shows the limitation. When $p=q=1$, the cross-covariance block contains just one number. That number can vanish because positive and negative contributions cancel, even when one variable is determined by the other.

Worked Example: Zero Covariance Despite Dependence

Let $X$ take the four values $-2,-1,1,2$, each with probability $1/4$, and set $Y=X^2$. Calculate the cross-covariance and check whether $X$ and $Y$ are independent.

By symmetry, $E[X]=0$, and $$ E[Y]=E[X^2]=\frac{4+1+1+4}{4}=\frac{5}{2}. $$ Also, $$ E[XY]=E[X^3]=\frac{-8-1+1+8}{4}=0. $$ It follows that $$ \operatorname{Cov}(X,Y)=E[XY]-E[X]E[Y]=0. $$ Nevertheless, $Y$ is determined by $X$. More explicitly, $P(Y=4)=1/2$, whereas $P(Y=4\mid X=2)=1$. These probabilities differ, so $X$ and $Y$ are not independent. The zero covariance records the cancellation in $E[X^3]$; it does not remove the dependence.

A useful algebraic way to read the block is that, for any fixed vectors $a\in\mathbb{R}^p$ and $b\in\mathbb{R}^q$, $$ \operatorname{Cov}(a^T X,b^T Y)=a^T\Sigma_{XY}b. $$ Consequently, $\Sigma_{XY}=0$ means that every linear combination of the coordinates of $X$ has zero covariance with every linear combination of the coordinates of $Y$. This is a precise statement, but it remains a statement about covariance, not a general characterization of the joint law.

Common Mistake and Practical Interpretation

Mistake: treating a zero cross-block as proof of independence. The reasoning “all cross-covariances are zero, therefore the groups are independent” is invalid in general. The correction is to report zero cross-covariance as absence of coordinatewise linear covariance, and to use additional distributional assumptions or other evidence if independence is the question.
Mistake: treating $\Sigma_{XY}$ as a symmetric matrix. It is usually rectangular. The correctly dimensioned identity is $\Sigma_{YX}=\Sigma_{XY}^T$.
Mistake: ignoring units and scale. Each entry has the product of the units of its two variables. A large numerical value cannot be interpreted without considering the measurement scales; covariance is not a scale-free measure.
Mistake: checking diagonal blocks but not the full matrix. Valid within-group covariance blocks do not alone guarantee that an arbitrarily chosen cross-block yields a valid joint covariance matrix. The full matrix must be positive semidefinite.

Cross-covariance blocks are useful for describing how two collections of variables co-vary and for organizing multivariate calculations. Their interpretation is deliberately narrower than a complete description of dependence. Keep the distinction clear: independence with finite second moments guarantees a zero cross-block, but a zero cross-block alone does not guarantee independence.

Check Your Understanding

Answer each question by referring to the block structure and its interpretation.

If $X$ has three coordinates and $Y$ has two, what are the dimensions of $\Sigma_{XY}$ and $\Sigma_{YX}$, and how are the two blocks related?
Under what moment condition does independence of $X$ and $Y$ imply $\Sigma_{XY}=0$?
What does a zero cross-covariance block imply about covariances between linear combinations of the two subvectors?
Why does a zero cross-covariance block not, in general, establish independence?
What property must the complete block covariance matrix satisfy, and what variance gives its quadratic form for vectors $a$ and $b$?