Covariance Partitioning for Two Variable Sets

Why Partition the Covariance Matrix?

Canonical correlation analysis studies association between two sets of variables. For example, one set might contain several clinical measurements at baseline, while the other contains several measures of later health. The joint covariance matrix separates the variability within each set from the covariation between sets. That separation is not just convenient notation: its positive-definiteness properties determine whether the usual classical analysis is well-defined and whether any association can be perfect.

Let $X$ be a $p$-dimensional random vector and $Y$ a $q$-dimensional random vector, with finite second moments. Write their means as $\mu_X=E[X]$ and $\mu_Y=E[Y]$. The cross-covariance block has rows indexed by components of $X$ and columns indexed by components of $Y$.

Definition: The joint covariance matrix of $X$ and $Y$, formed by stacking $X$ above $Y$, is $$ \operatorname{Cov}\!\begin{pmatrix}X\\Y\end{pmatrix} = \begin{pmatrix} \Sigma_{XX} & \Sigma_{XY}\\ \Sigma_{YX} & \Sigma_{YY} \end{pmatrix}, $$ where $\Sigma_{XX}=\operatorname{Cov}(X,X)$ is $p\times p$, $\Sigma_{YY}=\operatorname{Cov}(Y,Y)$ is $q\times q$, and $\Sigma_{XY}=\operatorname{Cov}(X,Y)$ is $p\times q$. The remaining block is $\Sigma_{YX}=\Sigma_{XY}^{T}$.

The diagonal blocks describe variation and association within each set. Their diagonal entries are component variances, and their off-diagonal entries are covariances between components in the same set. The off-diagonal blocks describe cross-set covariances: entry $(i,j)$ of $\Sigma_{XY}$ is the covariance between $X_i$ and $Y_j$. These blocks are transposes, not generally equal, because their row and column dimensions are reversed.

For compactness, write $A=\Sigma_{XX}$, $B=\Sigma_{YY}$, and $C=\Sigma_{XY}$. Then the joint covariance is $M=\begin{pmatrix}A&C\\C^T&B\end{pmatrix}$. The blocks cannot be chosen independently: a proposed joint covariance must be positive semidefinite. Classical canonical correlation analysis additionally requires each within-set covariance block to be positive definite. If the entire joint covariance is positive definite, the two sets have no exact nonzero linear relation and the usual canonical correlations are strictly less than one.

Worked Example: Reading the Blocks in a Clinical Covariance Matrix

Suppose $X=(X_1,X_2)^T$ contains two baseline measurements and $Y=(Y_1,Y_2)^T$ contains two follow-up measurements. Their joint covariance matrix is $$ M= \begin{pmatrix} 4&1&2&-1\\ 1&3&0&2\\ 2&0&5&1\\ -1&2&1&6 \end{pmatrix}. $$ Partition it using the first two components for $X$ and the last two for $Y$.

The blocks are $$ A=\begin{pmatrix}4&1\\1&3\end{pmatrix},\qquad B=\begin{pmatrix}5&1\\1&6\end{pmatrix},\qquad C=\begin{pmatrix}2&-1\\0&2\end{pmatrix}. $$ Thus $\operatorname{Cov}(X_1,Y_1)=2$, while $\operatorname{Cov}(X_2,Y_1)=0$. The lower-left block is $C^T=\begin{pmatrix}2&0\\-1&2\end{pmatrix}$; it records the same pairwise covariances in the reversed row-column order.

This partition identifies the roles and dimensions of the blocks, but it does not by itself establish that the displayed matrix is positive definite. That requires checking how the cross-block $C$ interacts with both within-set blocks.

Covariance Matrices Must Be Positive Semidefinite

A symmetric matrix $M$ is positive semidefinite if $z^TMz\geq 0$ for every vector $z$, and positive definite if $z^TMz>0$ for every nonzero $z$. A covariance matrix is always positive semidefinite: for any fixed vector $z$, its quadratic form is the variance of a linear combination of the random variables.

Result: If $X$ and $Y$ have finite second moments, their joint covariance matrix $M$ is positive semidefinite. In particular, for all $u\in\mathbb{R}^p$ and $v\in\mathbb{R}^q$, $$ u^TAu+2u^TCv+v^TBv = \operatorname{Var}\!\bigl(u^T X+v^T Y\bigr)\geq 0. $$

To verify the identity, center each random vector. The variance on the right is $E[(u^T(X-\mu_X)+v^T(Y-\mu_Y))^2]$. Expanding the square gives $u^TAu$, $v^TBv$, and two cross terms. Each cross term is a scalar, and together they equal $2u^TCv$, since $\Sigma_{YX}=C^T$. A variance is nonnegative, proving the claim.

This condition also implies that each within-set block is positive semidefinite: take $v=0$ to obtain $u^TAu\geq0$, and take $u=0$ to obtain $v^TBv\geq0$. But positive semidefiniteness of $A$ and $B$ alone is not enough to guarantee that a chosen cross-block $C$ produces a valid joint covariance.

Worked Example: A Valid Covariance with Perfect Association

Let $X$ and $Y$ be scalar random variables with covariance matrix $$ M=\begin{pmatrix}1&1\\1&1\end{pmatrix}. $$ The within-set variances are both 1, so each individual block is positive definite. But the determinant of $M$ is $1\cdot1-1\cdot1=0$, and $M$ is not positive definite.

The matrix is still a valid covariance matrix: for any real $u,v$, $$ \begin{pmatrix}u&v\end{pmatrix}M\begin{pmatrix}u\\v\end{pmatrix} =u^2+2uv+v^2=(u+v)^2\geq0. $$ For instance, it arises if $Y=X$ and $\operatorname{Var}(X)=1$. In that case $X-Y=0$ has variance zero. The joint covariance is singular because a nonzero linear combination of the stacked variables is constant. This is the boundary case of perfect linear association, not a positive-definite joint covariance.

The Schur Complement Gives the Positive-Definiteness Test

When $A$ is positive definite, its inverse exists. The matrix $B-C^TA^{-1}C$ is called the Schur complement of $A$ in $M$. It measures what remains of the $Y$-block after accounting, at the covariance-matrix level, for its connection to the $X$-block. The following equivalence is the central block-matrix test.

Theorem: Let $A$ be symmetric positive definite, $B$ symmetric, and $C$ a $p\times q$ matrix. For $M=\begin{pmatrix}A&C\\C^T&B\end{pmatrix}$, $$ M\text{ is positive definite} \quad\Longleftrightarrow\quad B-C^TA^{-1}C\text{ is positive definite}. $$ The same equivalence holds with “positive semidefinite” in place of “positive definite,” provided $A$ remains positive definite.

Proof. For arbitrary $x\in\mathbb{R}^p$ and $y\in\mathbb{R}^q$, expand and complete the square: $$ \begin{aligned} \begin{pmatrix}x^T&y^T\end{pmatrix} M \begin{pmatrix}x\\y\end{pmatrix} &=x^TAx+2x^TCy+y^TBy\\ &=(x+A^{-1}Cy)^TA(x+A^{-1}Cy) +y^T(B-C^TA^{-1}C)y. \end{aligned} $$ The equality follows by expanding the first term and using the symmetry of $A$. If the Schur complement is positive definite, the right side is positive for every nonzero pair $(x,y)$: if $y\ne0$, its second term is positive; if $y=0$ and $x\ne0$, its first term is positive. Thus $M$ is positive definite.

Conversely, suppose $M$ is positive definite. For any nonzero $y$, choose $x=-A^{-1}Cy$. The first term in the completed-square expression is then zero, so $$ y^T(B-C^TA^{-1}C)y = \begin{pmatrix}-A^{-1}Cy\\y\end{pmatrix}^{T} M \begin{pmatrix}-A^{-1}Cy\\y\end{pmatrix}>0. $$ The stacked vector is nonzero because $y\ne0$. This proves that the Schur complement is positive definite. Replacing strict inequalities by non-strict ones in both parts of the argument proves the positive-semidefinite version. $\square$

The same test can be applied with $B$ as the first block when $B$ is positive definite. In that order, the corresponding Schur complement is $A-CB^{-1}C^T$. These are equivalent tests for a matrix whose two diagonal blocks are positive definite.

Worked Example: Checking a Two-By-One Joint Covariance

Consider $$ M=\begin{pmatrix} 4&1&2\\ 1&2&3\\ 2&3&9 \end{pmatrix} = \begin{pmatrix}A&C\\C^T&B\end{pmatrix}, \quad A=\begin{pmatrix}4&1\\1&2\end{pmatrix},\quad C=\begin{pmatrix}2\\3\end{pmatrix},\quad B=(9). $$ First, $A$ is positive definite: its leading diagonal entry is $4>0$, and its determinant is $4\cdot2-1\cdot1=7>0$.

Its inverse and the Schur complement are $$ A^{-1}=\frac{1}{7}\begin{pmatrix}2&-1\\-1&4\end{pmatrix},\qquad C^TA^{-1}C =\begin{pmatrix}2&3\end{pmatrix}\frac{1}{7}\begin{pmatrix}1\\10\end{pmatrix} =\frac{32}{7}, $$ so $$ B-C^TA^{-1}C=9-\frac{32}{7}=\frac{31}{7}>0. $$ The Schur-complement theorem therefore shows that $M$ is positive definite. As a determinant check, $\det(M)=\det(A)(31/7)=7(31/7)=31>0$, consistent with the result.

Standardizing Both Sets Clarifies the CCA Condition

For classical canonical correlation analysis, both within-set covariance matrices must be positive definite: $A\succ0$ and $B\succ0$. This ensures that neither set contains a nonzero linear combination with zero variance and that the inverse square roots used to standardize the sets exist. Define the standardized cross-covariance matrix $K=A^{-1/2}CB^{-1/2}$, where the inverse square roots are the symmetric positive-definite inverse square roots.

Theorem: If $A\succ0$ and $B\succ0$, then $$ \begin{pmatrix}A&C\\C^T&B\end{pmatrix}\succ0 \quad\Longleftrightarrow\quad \|K\|_2<1, $$ where $\|K\|_2=\sup_{\|v\|=1}\|Kv\|$ is the operator norm. The joint matrix is positive semidefinite if and only if $\|K\|_2\leq1$.

Proof. Apply the invertible block change of coordinates with diagonal blocks $A^{-1/2}$ and $B^{-1/2}$. The resulting matrix is $$ \begin{pmatrix}I_p&K\\K^T&I_q\end{pmatrix}. $$ An invertible change of coordinates preserves positive definiteness and positive semidefiniteness, since its quadratic forms are the original quadratic forms evaluated at an invertibly transformed vector. By the Schur-complement theorem, the displayed matrix is positive definite exactly when $I_q-K^TK\succ0$; it is positive semidefinite exactly when $I_q-K^TK\succeq0$. For any $v$, $$ v^T(I_q-K^TK)v=\|v\|^2-\|Kv\|^2. $$ Thus the strict condition for every nonzero $v$ is $\|Kv\|<\|v\|$, equivalently $\|K\|_2<1$; the non-strict condition is equivalently $\|K\|_2\leq1$. $\square$

This condition expresses the compatibility of cross-set association with within-set variability. A valid joint covariance with positive-definite within-set blocks cannot have standardized cross-association exceeding one. If the joint covariance is positive definite, it cannot attain one either. The operator norm condition is a matrix-level statement; it does not require that every individual cross-covariance be small in the original measurement units.

Worked Example: Verifying a Standardized Cross-Block

Let $A=B=I_2$ and $$ C=\begin{pmatrix}0.8&0\\0&0.3\end{pmatrix}. $$ Since both within-set blocks are identity matrices, $K=A^{-1/2}CB^{-1/2}=C$. The Schur complement of $A$ is $$ B-C^TA^{-1}C =I_2-C^TC =\begin{pmatrix}1-0.8^2&0\\0&1-0.3^2\end{pmatrix} =\begin{pmatrix}0.36&0\\0&0.91\end{pmatrix}. $$ Both diagonal entries are strictly positive, so the Schur complement and the full joint covariance are positive definite. Equivalently, the operator norm of this diagonal $K$ is $0.8<1$.

If the first diagonal entry of $C$ were changed to $1.1$, the resulting block matrix would not be a valid covariance matrix. In the direction $x=(1,0)^T$, $y=(-1,0)^T$, its quadratic form would be $$ x^TAx+2x^TCy+y^TBy=1-2.2+1=-0.2<0. $$ A covariance quadratic form cannot be negative, so this proposed cross-block is incompatible with the two identity within-set blocks.

Common Pitfall and Practical Meaning

A common mistake is to check only that $A$ and $B$ are positive definite, then assume any cross-block $C$ gives a valid joint covariance. The $1.1$ calculation above disproves that reasoning: the within-set blocks are both positive definite, yet the full matrix has a negative quadratic form. The correction is to check the joint matrix, for example with a Schur complement or, after standardizing, by checking that $\|K\|_2\leq1$ for validity and $\|K\|_2<1$ for strict positive definiteness.

A second pitfall is to treat positive semidefinite and positive definite as interchangeable. A singular covariance can be perfectly valid, as in the case $Y=X$, but it has a zero-variance linear combination. Classical CCA requires positive-definite within-set blocks; assuming a positive-definite joint covariance is a stronger condition that excludes perfect linear dependence across the two sets and ensures strict inequality in the standardized condition. In data analysis, exact or near singularity can make the classical calculations unavailable or unstable, even when the observed covariance estimates appear reasonable.

Key takeaway: The joint covariance partitions as $\begin{pmatrix}A&C\\C^T&B\end{pmatrix}$. The within-set blocks describe each variable set separately, while $C$ links them. For $A\succ0$, the full matrix is positive definite exactly when $B-C^TA^{-1}C\succ0$; when both within-set blocks are positive definite, the equivalent standardized condition is $\|A^{-1/2}CB^{-1/2}\|_2<1$.

Check Your Understanding

Use the block definitions and positive-definiteness conditions to answer these questions.

For $X\in\mathbb{R}^p$ and $Y\in\mathbb{R}^q$, what are the dimensions of $\Sigma_{XY}$ and $\Sigma_{YX}$, and how are the two blocks related?
Why must every joint covariance matrix be positive semidefinite, even when it is not positive definite?
State the Schur-complement condition for positive definiteness when $A\succ0$.
If $A=B=(1)$ and $C=(1)$, is the joint covariance positive definite? Explain using a nonzero vector with zero quadratic form.
When $A\succ0$ and $B\succ0$, what condition on $K=A^{-1/2}CB^{-1/2}$ makes the joint covariance positive definite?