Whitening and the Singular Value Decomposition

Why Whitening Clarifies Canonical Correlation

Canonical correlation compares linear combinations of two variable sets. The within-set covariance matrices complicate the comparison: a coefficient vector with a large numerical size need not produce a score with large variance, and different directions within a set may have different scales. Whitening removes those within-set scales. After whitening, each set has identity covariance, and the association between the sets is represented by a single cross-covariance matrix.

The singular value decomposition (SVD) of that matrix gives a direct description of the canonical correlations. This is more than a computational convenience: it is the ordinary Euclidean geometry underlying the generalized eigenvalue equations.

Whitened Cross-Covariance

Let \(X\) be a \(p\)-dimensional random vector and \(Y\) a \(q\)-dimensional random vector, with finite second moments. Write their covariance matrices and cross-covariance matrix as \(\Sigma_{XX}=\operatorname{Cov}(X,X)\), \(\Sigma_{YY}=\operatorname{Cov}(Y,Y)\), and \(\Sigma_{XY}=\operatorname{Cov}(X,Y)\). Thus \(\Sigma_{YX}=\Sigma_{XY}'\). Assume throughout the main derivations that \(\Sigma_{XX}\) and \(\Sigma_{YY}\) are positive definite.

For a positive definite matrix \(\Sigma\), \(\Sigma^{1/2}\) denotes its symmetric positive definite square root, and \(\Sigma^{-1/2}\) its inverse. These are matrix operations, not element-by-element operations. Define the centered, whitened vectors and their cross-covariance matrix by

The covariance of \(X_w\) is \(I_p\), since \(\operatorname{Cov}(X_w,X_w)=\Sigma_{XX}^{-1/2}\Sigma_{XX}\Sigma_{XX}^{-1/2}=I_p\). Similarly, \(\operatorname{Cov}(Y_w,Y_w)=I_q\), while \(\operatorname{Cov}(X_w,Y_w)=C\).

From Canonical Correlation to a Singular Value

For coefficient vectors \(a\in\mathbb{R}^p\) and \(b\in\mathbb{R}^q\), the correlation between the scores \(a'(X-E[X])\) and \(b'(Y-E[Y])\) is \[ \frac{a'\Sigma_{XY}b} {\sqrt{(a'\Sigma_{XX}a)(b'\Sigma_{YY}b)}}. \] With positive definite within-set covariance matrices, both score variances are positive whenever the corresponding coefficient vector is nonzero. Set \(u=\Sigma_{XX}^{1/2}a\) and \(v=\Sigma_{YY}^{1/2}b\). Then the score variances are \(u'u\) and \(v'v\), and the score covariance is \(u'Cv\). Consequently, the canonical correlation optimization becomes the maximization of \(u'Cv\) over unit vectors \(u\) and \(v\). The sign of either coefficient vector is free, so maximizing the signed quantity gives the same maximum as maximizing its absolute value.

To see the optimization at the heart of the result, let \(\|u\|=\|v\|=1\). For fixed \(v\), the Cauchy–Schwarz inequality gives \(u'Cv\leq \|Cv\|\), with equality by taking \(u\) in the direction of \(Cv\) when \(Cv\neq 0\). Therefore \[ \max_{\|u\|=\|v\|=1}u'Cv =\max_{\|v\|=1}\|Cv\|. \] The right-hand side is the operator norm of \(C\), which equals its largest singular value. This establishes the largest-root statement directly.

For the full set of roots, take an SVD \(C=U D V'\), where \(U\) and \(V\) have orthonormal columns and \(D\) has the singular values on its diagonal. The equations \(Cv_j=\sigma_j u_j\) and \(C'u_j=\sigma_j v_j\) show that each positive singular value supplies a paired direction with score correlation \(\sigma_j\). Equivalently, \(CC'u_j=\sigma_j^2u_j\) and \(C'Cv_j=\sigma_j^2v_j\). The variational characterization of successive canonical correlations imposes orthogonality to the previously selected whitened directions; the SVD gives precisely these successive singular values. Zero singular values correspond to directions with zero cross-covariance. This proves the stated correspondence. \(\square\)

Connection to the Generalized Eigenvalue Equations

The SVD representation also recovers the generalized eigenvalue equations in a symmetric form. Let \(a\) and \(b\) be canonical coefficient vectors for a nonzero canonical correlation \(\rho\), and put \(u=\Sigma_{XX}^{1/2}a\), \(v=\Sigma_{YY}^{1/2}b\). The singular-vector equations \(Cv=\rho u\) and \(C'u=\rho v\), after substituting the definitions, yield the following generalized equations.

For example, the first generalized equation follows by starting with \(C v=\rho u\), using \(v=\Sigma_{YY}^{1/2}b\) and \(u=\Sigma_{XX}^{1/2}a\), and then using the paired equation \(C'u=\rho v\) to eliminate \(b\). In detail, multiplying \(C'C v=\rho^2 v\) by the appropriate covariance square roots, or directly substituting \(C\), gives \[ \Sigma_{XX}^{-1/2}\Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}\Sigma_{XX}^{-1/2}u=\rho^2u. \] Since \(u=\Sigma_{XX}^{1/2}a\), multiplication by \(\Sigma_{XX}^{1/2}\) gives the first equation. The second follows by the same calculation with \(X\) and \(Y\) exchanged. Conversely, a positive eigenvalue \(\rho^2\) of either whitened product gives a singular value \(\rho\) of \(C\), with paired left and right singular vectors. Thus the symmetric eigenvalue formulation and the SVD formulation describe the same nonzero roots. \(\square\)

Why the Singular Values Cannot Exceed One

For genuine covariance matrices, every singular value of \(C\) lies between zero and one. This bound is a consequence of the positive semidefiniteness of the joint covariance matrix, not an extra modeling assumption. The joint covariance of the whitened vectors is \[ \begin{pmatrix}I_p&C\\C'&I_q\end{pmatrix}. \] For any unit vectors \(u\in\mathbb{R}^p\) and \(v\in\mathbb{R}^q\), positive semidefiniteness implies that the variance of \(u'X_w+ t v'Y_w\) is nonnegative for every real \(t\). That variance is \(1+2t\,u'Cv+t^2\). A quadratic with leading coefficient one is nonnegative for all \(t\) only if its discriminant is nonpositive, so \((u'Cv)^2\leq 1\). Maximizing over unit \(u,v\) gives \(\sigma_1\leq1\), and all other singular values are no larger. Since singular values are nonnegative, each lies in \([0,1]\). \(\square\)

Common Mistakes and Practical Meaning

A common mistake is to read canonical correlations directly from the entries of \(\Sigma_{XY}\). Those entries depend on the measurement scales and the within-set covariance patterns. The relevant matrix is the whitened cross-covariance \(C\), and its singular values—not its individual entries—are the canonical correlations. In the same way, the eigenvalues of \(C\) itself are not generally the answer: \(C\) may be rectangular or nonsymmetric, and canonical correlations are nonnegative singular values.

Another mistake is to treat whitening as division by each variable's standard deviation in all cases. That operation is sufficient only when the within-set covariance matrix is diagonal. With correlated variables, the full inverse square root accounts for their covariance geometry. If a within-set covariance matrix is singular, the inverse square root used here does not exist; one must first address the reduced-rank structure or use an explicitly defined generalized-inverse approach rather than silently applying these formulas.

The SVD matters because it converts a covariance-adjusted comparison into familiar Euclidean geometry. It also offers a numerically direct route to the canonical roots and makes the squared-root eigenvalue equations transparent. In practice, the assumptions and numerical conditioning of the covariance estimates still matter: a nearly singular within-set covariance matrix can make its inverse square root unstable.