CCA Through Whitening and Principal Angles

Why Whitening Gives CCA a Geometric Interpretation

Canonical correlation analysis (CCA) seeks linear combinations of two data blocks whose scores are highly correlated. The generalized eigenvalue formulation makes the optimization conditions explicit; whitening gives a complementary view: it first removes each block’s own covariance scale, then measures how closely the resulting directions align across the two blocks.

This geometric view is useful in multi-omics analysis because different blocks can contain different numbers of features and very different within-block scales. Whitening puts their feature directions on a common unit-variance footing. The resulting canonical correlations have a direct interpretation as angles between subspaces in the space of observations. That interpretation is about the observed sample geometry; by itself, it does not establish a biological mechanism or a population-level relationship.

Definition: Let \(X\) be an \(n\times p\) matrix and \(Y\) an \(n\times q\) matrix of measurements on the same \(n\) samples, with every column centered. Define \(S_{XX}=X^\mathsf{T}X/(n-1)\), \(S_{YY}=Y^\mathsf{T}Y/(n-1)\), and \(S_{XY}=X^\mathsf{T}Y/(n-1)\). If \(S_{XX}\) and \(S_{YY}\) are positive definite, their symmetric inverse square roots are \(S_{XX}^{-1/2}\) and \(S_{YY}^{-1/2}\). The whitened cross-block matrix is \(K=S_{XX}^{-1/2}S_{XY}S_{YY}^{-1/2}\).

Positive definiteness means that no nonzero linear combination of a block’s columns has zero sample variance. In particular, each block must have full column rank; for centered data, this requires \(p\leq n-1\) and \(q\leq n-1\). The positive-definite condition is not a technical detail that can be ignored: if it fails, the inverse square roots in this definition do not exist. Handling singular covariance matrices requires a different treatment.

From Whitened Blocks to Singular Values

Define the whitened data matrices \(Z_X=X S_{XX}^{-1/2}/\sqrt{n-1}\) and \(Z_Y=Y S_{YY}^{-1/2}/\sqrt{n-1}\). Their columns have unit sample variance and are mutually uncorrelated within each block. In matrix form, \(Z_X^\mathsf{T}Z_X=I_p\) and \(Z_Y^\mathsf{T}Z_Y=I_q\), where \(I_p\) and \(I_q\) are identity matrices. Moreover, \(Z_X^\mathsf{T}Z_Y=K\).

Any unit-length direction \(u\) in the whitened \(X\)-block and unit-length direction \(v\) in the whitened \(Y\)-block produce scores \(Z_Xu\) and \(Z_Yv\). Their sample variances are both one, and their sample correlation is their inner product: \((Z_Xu)^\mathsf{T}(Z_Yv)=u^\mathsf{T}Kv\). Thus the largest possible correlation is the largest singular value of \(K\). More generally, the ordered singular values of \(K\) are the canonical correlations, with at most \(\min(p,q)\) values.

\max_{\lVert u\rVert=\lVert v\rVert=1} u^\mathsf{T}Kv =\sigma_1(K), \qquad K=U\Sigma V^\mathsf{T}

Here \(\sigma_1(K)\) is the largest singular value, and \(U\) and \(V\) contain the corresponding left and right singular directions. The identity follows directly from the singular value decomposition: write \(a=U^\mathsf{T}u\) and \(b=V^\mathsf{T}v\), which preserve lengths. Then \(u^\mathsf{T}Kv=a^\mathsf{T}\Sigma b\), which is at most \(\sigma_1(K)\) by the Cauchy–Schwarz inequality. Choosing \(u\) and \(v\) as the first columns of \(U\) and \(V\) attains that bound. This also shows why whitening converts the constrained correlation problem into a singular-value problem.

The corresponding directions in the original feature coordinates are \(S_{XX}^{-1/2}u\) and \(S_{YY}^{-1/2}v\). The whitening view therefore agrees with the generalized eigenvalue approach: both express CCA’s variance-normalized optimization, but whitening makes the geometry of the solution visible.

Principal Angles Between Sample-Space Subspaces

The columns of \(Z_X\) are orthonormal vectors in \(\mathbb{R}^n\), as are the columns of \(Z_Y\). They span the same subspaces as the columns of \(X\) and \(Y\), respectively, because whitening only applies an invertible transformation within each block. Principal angles describe how these two subspaces are oriented relative to one another.

Definition: For two subspaces with orthonormal basis matrices \(Q_X\) and \(Q_Y\), the cosines of their principal angles are the singular values of \(Q_X^\mathsf{T}Q_Y\), ordered from largest to smallest. Angles lie between \(0\) and \(90\) degrees. In CCA, \(Z_X\) and \(Z_Y\) are such basis matrices, so each canonical correlation \(\rho_j\) equals \(\cos(\theta_j)\), where \(\theta_j\) is the corresponding principal angle.

A small principal angle means that some direction in one block’s sample-space span lies close to a direction in the other block’s span. A zero angle means that the subspaces share a direction and the corresponding canonical correlation is one. A right angle means the associated directions are orthogonal and the correlation is zero. These statements concern linear geometry in the observed samples, not causal relationships.

The correlations cannot exceed one in magnitude. For any unit vectors \(u\) and \(v\), the whitened score vectors \(Z_Xu\) and \(Z_Yv\) each have unit length. Cauchy–Schwarz gives \(\left|(Z_Xu)^\mathsf{T}(Z_Yv)\right|\leq 1\). Taking the maximum over such directions shows every singular value of \(K\) is at most one. For principal angles between subspaces, the singular values are nonnegative, so they lie in \([0,1]\).

Worked Example: One Direction in Each Block

Consider three hypothetical samples with centered measurements \(X=(1,-1,0)^\mathsf{T}\) and \(Y=(2,0,-2)^\mathsf{T}\). Both columns are centered. Compute the canonical correlation and the angle between their sample-space directions.

There is one feature in each block, so CCA reduces to the ordinary sample correlation. The sample covariances are \(S_{XX}=2/2=1\), \(S_{YY}=8/2=4\), and \(S_{XY}=2/2=1\). Hence \(K=1^{-1/2}(1)4^{-1/2}=1/2=0.5000\).

S_{XY}=\frac{1(2)+(-1)(0)+0(-2)}{3-1}=1,\qquad \rho_1=K=\frac{1}{\sqrt{1}\sqrt{4}}=0.5000

The angle is \(\theta_1=\arccos(0.5000)=60.0^\circ\). Equivalently, the centered sample vectors have dot product \(2\) and lengths \(\sqrt{2}\) and \(\sqrt{8}\), so their cosine is \(2/(\sqrt{2}\sqrt{8})=0.5000\). Whitening converts their lengths to one without changing the angle.

Worked Example: Two Angles Between Two Blocks

Suppose five hypothetical samples have centered observations whose four relevant orthonormal contrast directions are \(e_1,e_2,e_3,e_4\). These are unit-length vectors in the centered sample space and are mutually orthogonal. Let the whitened \(X\)-block have columns \(e_1,e_2\), and let the whitened \(Y\)-block have columns \(0.8e_1+0.6e_3\) and \(0.6e_2+0.8e_4\). Each column in either block has unit length, and the two columns within each block are orthogonal.

The cross-product of the whitened blocks has diagonal entries \(0.8\) and \(0.6\), with zero off-diagonal entries: the first \(X\) direction has inner product \(0.8\) with the first \(Y\) direction and zero with the second, while the second \(X\) direction has inner product \(0.6\) with the second \(Y\) direction and zero with the first. Thus \(K=\begin{pmatrix}0.8&0\\0&0.6\end{pmatrix}\). Its singular values are \(0.8\) and \(0.6\), so these are the two canonical correlations.

\theta_1=\arccos(0.8)=36.87^\circ,\qquad \theta_2=\arccos(0.6)=53.13^\circ

The first pair of directions is closer because its angle is smaller and its correlation is larger. This example also illustrates that canonical correlations summarize alignment across whole block subspaces, not just the association between any one originally named feature in \(X\) and any one in \(Y\).

Worked Geometry and What a Perfect Correlation Means

The angle interpretation also clarifies the limiting case of a canonical correlation equal to one. It means there is an exact shared direction in the two sample-space subspaces. Such a match can reflect genuine shared structure, but in a finite sample it can also arise because the blocks are high-dimensional relative to the number of samples, or because they contain duplicated or derived measurements. A perfect in-sample canonical correlation is therefore not, on its own, evidence of a stable association beyond the observed data.

Worked Example: An Exact Shared Direction

In a hypothetical centered sample space, suppose the two whitened blocks have columns \(Z_X=(e_1,e_2)\) and \(Z_Y=(e_1,e_3)\), where \(e_1,e_2,e_3\) are mutually orthonormal. Each block has full column rank, and its columns are orthonormal. Their cross-product is \(K=\begin{pmatrix}1&0\\0&0\end{pmatrix}\).

The singular values are \(1\) and \(0\), giving canonical correlations \(1\) and \(0\), with angles \(0^\circ\) and \(90^\circ\). The first subspaces share \(e_1\) exactly; the remaining directions \(e_2\) and \(e_3\) are orthogonal. The first canonical correlation is perfect in this sample geometry, but these calculations alone cannot distinguish a reproducible shared signal from a data-construction artifact or a chance alignment.

Assumptions, Diagnostics, and Alternatives

The whitening derivation requires that the rows of \(X\) and \(Y\) refer to the same samples in the same order, that columns are centered, and that both within-block sample covariance matrices are positive definite. In applications, these conditions should be checked before interpreting the singular values. The centered columns must have nonzero variance and no exact linear dependence within a block.

Several checks help identify problems:

Check rank and conditioning. A zero variance column or exact linear dependence makes the covariance matrix singular. Near-dependence can make its inverse square root highly sensitive to small data changes.
Check the range of singular values. Values outside \([0,1]\), beyond minor numerical rounding, suggest an implementation or scaling error. A value exactly or nearly one deserves scrutiny for shared columns, derived features, or a high-dimensional fit.
Check alignment and centering. Mismatched sample order or uncentered data changes the cross-block geometry and can produce results that do not answer the intended CCA question.
Check stability. Large changes in the leading directions or correlations under small sample perturbations indicate that the apparent alignment may be fragile.

The generalized eigenvalue formulation and the whitening formulation are equivalent ways to express classical CCA when the within-block covariance matrices are positive definite. The generalized eigenvalue form emphasizes the variance-normalized optimization equations; whitening emphasizes orthonormal directions and principal angles. A direct correlation calculation is simpler when each block has only one feature, but it does not replace the multivariate subspace view. When the full-rank assumptions fail, neither the ordinary inverse-square-root derivation nor the corresponding positive-definite generalized eigenvalue setup applies unchanged.

Whitening also has limits as an interpretation aid. It deliberately removes each block’s within-block covariance scale, so it answers a correlation-based question rather than a question about raw covariance magnitude. Its geometry is sample-dependent, and unregularized CCA can produce strong in-sample alignments when there are many features relative to samples. The angle interpretation explains what the fitted correlations mean; it does not resolve those estimation or generalization concerns.

Common Mistakes

Confusing feature-space and sample-space angles. The principal angles here are between the column spans of the blocks in the \(n\)-sample space, not between lists of feature names.
Calling whitening a change in the samples. Whitening rescales and rotates directions within each block. It creates orthonormal columns but does not change the block’s sample-space column span.
Interpreting a correlation of one as proof of a biological relationship. It certifies an exact shared direction in the fitted sample subspaces. It does not establish why that direction is shared or whether it will persist in new samples.
Using inverse square roots with singular covariance matrices. The stated derivation assumes positive definite within-block covariance matrices. If a block is rank-deficient, the ordinary inverse square root is undefined.

Key takeaway: With centered, full-column-rank blocks, whitening produces orthonormal bases for their sample-space spans. The singular values of their cross-product are the canonical correlations, and each is the cosine of a principal angle between those spans.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

If the largest canonical correlation is \(0.25\), what is its principal angle to the nearest degree?
Why does the whitening derivation require positive-definite within-block covariance matrices?
Two whitened one-dimensional blocks have a cross-product of \(0\). What are their canonical correlation and angle?
What geometric feature of the sample-space subspaces produces a canonical correlation of one?

Solutions. (1) The angle is \(\arccos(0.25)=75.52^\circ\), or approximately \(76^\circ\). (2) The derivation uses \(S_{XX}^{-1/2}\) and \(S_{YY}^{-1/2}\); these ordinary inverse square roots exist only when the matrices are positive definite. (3) The sole singular value is \(0\), so the correlation is \(0\) and the angle is \(\arccos(0)=90^\circ\). (4) The subspaces share an exact nonzero direction. Its normalized vectors have inner product \(1\), so the associated principal angle is \(0^\circ\) and its cosine is \(1\).