Tutorials › Multivariate Statistics (Graduate) › Covariance Geometry of Canonical Scores

Home  /  Tutorials  /  Canonical Correlation Analysis

Canonical Correlation Analysis · Tutorial 387 of 975

Covariance Geometry of Canonical Scores

See how canonical normalization gives identity covariance within each set of scores and a diagonal matrix of canonical correlations between the sets.

Intermediate 9 min read

What You'll Learn

  • Define canonical score vectors using normalized canonical weights
  • Derive their within-set identity covariance matrices from whitening
  • Show how the singular value decomposition gives diagonal cross-covariance
  • Connect score variances and covariances to their pairwise correlations
  • Check the identities with explicit covariance matrices and score weights

Why Canonical Scores Have a Simple Covariance Geometry

Canonical correlation analysis replaces two collections of measurements with paired linear combinations, called canonical scores. The weights are chosen so that the paired scores capture strong association across the two sets. Their covariance geometry is deliberately simple: each set of scores has identity covariance, while the cross-covariance matrix between the sets is diagonal. Understanding those matrices clarifies what normalization accomplishes and where the canonical correlations appear.

The statements below concern the population covariance matrices. Assume the two random vectors have finite second moments, and that each has a positive-definite covariance matrix. These conditions allow whitening: transforming each vector so that its covariance becomes the identity. The singular value decomposition of the resulting cross-covariance matrix supplies the canonical directions.

Definition: Let \(X\in\mathbb{R}^p\) and \(Y\in\mathbb{R}^q\) be random vectors with finite second moments. Write \(\Sigma_{XX}=\operatorname{Cov}(X,X)\), \(\Sigma_{YY}=\operatorname{Cov}(Y,Y)\), and \(\Sigma_{XY}=\operatorname{Cov}(X,Y)\). For weight vectors \(a_i\) and \(b_i\), the \(i\)th canonical scores are \(U_i=a_i'X\) and \(V_i=b_i'Y\). Their vector forms are \(U=A'X\) and \(V=B'Y\), where the columns of \(A\) and \(B\) are the score weights.

Scores can instead be formed from centered vectors \(X-E[X]\) and \(Y-E[Y]\). Subtracting these means does not change any covariance, so the formulas below apply either way. Let \(k=\min(p,q)\). Under the positive-definiteness assumption, the canonical weights can be normalized so that each score has variance one. Their paired signs are chosen so that the corresponding cross-covariances are nonnegative.

From Whitening to Score Covariance Matrices

The whitened cross-covariance matrix is \(K=\Sigma_{XX}^{-1/2}\Sigma_{XY}\Sigma_{YY}^{-1/2}\), where the inverse square roots are the symmetric positive-definite ones. Take a singular value decomposition \(K=P D Q'\), with the singular values ordered from largest to smallest. Use the first \(k\) left and right singular vectors to form \(P_k\) and \(Q_k\), and let \(D_k\) be the diagonal matrix of the corresponding singular values. These singular values are the canonical correlations.

Theorem: With \(A=\Sigma_{XX}^{-1/2}P_k\) and \(B=\Sigma_{YY}^{-1/2}Q_k\), the canonical score vectors satisfy $$ \operatorname{Cov}(U,U)=I_k,\qquad \operatorname{Cov}(V,V)=I_k,\qquad \operatorname{Cov}(U,V)=D_k. $$ Here \(I_k\) is the \(k\times k\) identity matrix. The diagonal entries of \(D_k\) are the ordered canonical correlations.

Proof. For any random vector \(Z\) and fixed matrices \(C\) and \(G\), \(\operatorname{Cov}(CZ,GZ)=C\operatorname{Cov}(Z,Z)G'\); the corresponding formula for two different vectors is \(\operatorname{Cov}(CX,GY)=C\operatorname{Cov}(X,Y)G'\). Applying these formulas to \(U=A'X\) and \(V=B'Y\) gives

$$ \begin{aligned} \operatorname{Cov}(U,U) &=A'\Sigma_{XX}A\\ &=P_k'\Sigma_{XX}^{-1/2}\Sigma_{XX}\Sigma_{XX}^{-1/2}P_k\\ &=P_k'P_k=I_k, \end{aligned} $$

because the columns of \(P_k\) are orthonormal. The same calculation with \(B\), \(\Sigma_{YY}\), and \(Q_k\) gives \(\operatorname{Cov}(V,V)=Q_k'Q_k=I_k\). For the cross-covariance, substitute the weights and use the SVD:

$$ \begin{aligned} \operatorname{Cov}(U,V) &=A'\Sigma_{XY}B\\ &=P_k'\Sigma_{XX}^{-1/2}\Sigma_{XY}\Sigma_{YY}^{-1/2}Q_k\\ &=P_k'KQ_k=D_k. \end{aligned} $$

This establishes all three identities. \(\square\)

The singular values are at most one. In fact, for any vectors \(s\) and \(t\), Cauchy–Schwarz applied to the two scalar linear combinations of \(X\) and \(Y\) gives \(|s'Kt|\leq \lVert s\rVert\lVert t\rVert\). Thus the largest singular value of \(K\), and hence every canonical correlation, cannot exceed one. A value of one is possible when a linear combination from each set agrees perfectly after normalization.

Variances, Covariances, and Correlations

The matrix identities have direct componentwise meanings. The diagonal entries of \(\operatorname{Cov}(U,U)=I_k\) say \(\operatorname{Var}(U_i)=1\) for every score in the first set. Its off-diagonal entries say \(\operatorname{Cov}(U_i,U_j)=0\) when \(i\ne j\). The corresponding statements hold for the \(V\) scores. Zero covariance does not, in general, imply independence; the covariance identities alone make no independence claim.

The cross-covariance identity says \(\operatorname{Cov}(U_i,V_j)=0\) for \(i\ne j\), and \(\operatorname{Cov}(U_i,V_i)=\rho_i\), where \(\rho_i\) is the \(i\)th canonical correlation. Since both scores in each pair have variance one, their correlation is

$$ \operatorname{Corr}(U_i,V_i) =\frac{\operatorname{Cov}(U_i,V_i)} {\sqrt{\operatorname{Var}(U_i)\operatorname{Var}(V_i)}} =\frac{\rho_i}{\sqrt{1\cdot 1}} =\rho_i. $$

More generally, every entry of \(\operatorname{Cov}(U,V)\) is also the correlation of the corresponding scores, because their individual variances are one. The diagonal form therefore records the matched canonical correlations and the zero covariances between unmatched score coordinates.

Worked Example: Scaling Two Variables in Each Set

Suppose the covariance matrices are \(\Sigma_{XX}=\begin{pmatrix}4&0\\0&1\end{pmatrix}\), \(\Sigma_{YY}=\begin{pmatrix}9&0\\0&4\end{pmatrix}\), and \(\Sigma_{XY}=\begin{pmatrix}4.8&0\\0&0.6\end{pmatrix}\). The whitened cross-covariance is

$$ K= \begin{pmatrix}1/2&0\\0&1\end{pmatrix} \begin{pmatrix}4.8&0\\0&0.6\end{pmatrix} \begin{pmatrix}1/3&0\\0&1/2\end{pmatrix} = \begin{pmatrix}0.8&0\\0&0.3\end{pmatrix}. $$

The singular values are \(0.8\) and \(0.3\), with identity left and right singular-vector matrices. Thus \(A=\operatorname{diag}(1/2,1)\) and \(B=\operatorname{diag}(1/3,1/2)\). The scores are \(U=(X_1/2,X_2)'\) and \(V=(Y_1/3,Y_2/2)'\). Directly,

$$ A'\Sigma_{XX}A=I_2,\qquad B'\Sigma_{YY}B=I_2,\qquad A'\Sigma_{XY}B= \begin{pmatrix}0.8&0\\0&0.3\end{pmatrix}. $$

So each score has variance one, and the correlations of the matched pairs are \(0.8\) and \(0.3\). The joint covariance is valid: after whitening, its block matrix has canonical correlations below one, so its eigenvalues are \(1+0.8,1-0.8,1+0.3,1-0.3\), all positive.

Worked Example: Raw Cross-Covariances Need Not Be Diagonal

Let \(\Sigma_{XX}=\Sigma_{YY}=I_2\), and let \(\Sigma_{XY}=\begin{pmatrix}0&0.6\\0.2&0\end{pmatrix}\). Since the within-set covariance matrices are already identity, this cross-covariance is also the whitened matrix. Its singular values are \(0.6\) and \(0.2\): the first maps the \(Y\)-direction \(e_2\) to the \(X\)-direction \(e_1\), and the second maps \(e_1\) to \(e_2\).

Choose the score weights \(A=(e_1,e_2)\) and \(B=(e_2,e_1)\), where the listed vectors are columns. Then \(U=(X_1,X_2)'\) and \(V=(Y_2,Y_1)'\). The within-set covariances remain \(I_2\), and

$$ A'\Sigma_{XY}B= \begin{pmatrix} e_1'\Sigma_{XY}e_2&e_1'\Sigma_{XY}e_1\\ e_2'\Sigma_{XY}e_2&e_2'\Sigma_{XY}e_1 \end{pmatrix} = \begin{pmatrix}0.6&0\\0&0.2\end{pmatrix}. $$

The raw variables have nonzero cross-covariances in off-diagonal positions, but pairing and ordering the canonical score directions puts the canonical correlations on the diagonal. The joint covariance is valid because the singular values are both less than one.

Worked Example: Rotated Canonical Directions

A diagonal cross-covariance is not required for the result. Set \(\Sigma_{XX}=\operatorname{diag}(4,1)\), \(\Sigma_{YY}=\operatorname{diag}(1,9)\), and define \(R=2^{-1/2}\begin{pmatrix}1&-1\\1&1\end{pmatrix}\). Take the whitened cross-covariance to be \(K=R\operatorname{diag}(0.8,0.3)R'\). Multiplication gives \(K=\begin{pmatrix}0.55&0.25\\0.25&0.55\end{pmatrix}\). The corresponding original cross-covariance is

$$ \Sigma_{XY} =\Sigma_{XX}^{1/2}K\Sigma_{YY}^{1/2} = \begin{pmatrix}2&0\\0&1\end{pmatrix} \begin{pmatrix}0.55&0.25\\0.25&0.55\end{pmatrix} \begin{pmatrix}1&0\\0&3\end{pmatrix} = \begin{pmatrix}1.1&1.5\\0.25&1.65\end{pmatrix}. $$

Here \(P=Q=R\). The weights are \(A=\Sigma_{XX}^{-1/2}R\) and \(B=\Sigma_{YY}^{-1/2}R\). Their score covariances follow from the matrix products:

$$ A'\Sigma_{XX}A=R'R=I_2,\qquad B'\Sigma_{YY}B=R'R=I_2,\qquad A'\Sigma_{XY}B=R'KR= \begin{pmatrix}0.8&0\\0&0.3\end{pmatrix}. $$

Both original covariance blocks involve nontrivial scaling, and the original cross-covariance is not diagonal. Whitening and rotating to the singular-vector directions nevertheless give unit score variances and matched score correlations \(0.8\) and \(0.3\). The joint covariance is valid because both singular values of \(K\) are less than one.

Common Pitfall: Confusing Covariance and Correlation

A tempting but incorrect argument is: “The canonical correlation is \(\rho_i\), so the covariance of the corresponding scores must be \(\rho_i\), whatever weights were used.” Correlation is covariance divided by the product of the score standard deviations. Without unit-variance normalization, the covariance need not equal the correlation. For example, replacing \(U_i\) by \(3U_i\) leaves its correlation with \(V_i\) unchanged, but multiplies their covariance by three. The equality \(\operatorname{Cov}(U_i,V_i)=\operatorname{Corr}(U_i,V_i)=\rho_i\) holds here because the canonical scores have both been normalized to variance one.

A second mistake is to expect the original cross-covariance matrix \(\Sigma_{XY}\) to be diagonal. It is the cross-covariance of the appropriately weighted canonical scores, \(A'\Sigma_{XY}B\), that is diagonal. The weights select and order the directions that reveal this structure.

Key takeaway: With unit-variance canonical normalization, each score set has identity covariance and the cross-covariance matrix is diagonal. Its diagonal entries are the canonical correlations; because each score variance is one, they are also the correlations of the matched scores.

Check Your Understanding

Use the score covariance identities to answer these questions.

  1. What does a diagonal entry of \(\operatorname{Cov}(U,U)=I_k\) tell you about the variance of a canonical score?
  2. If \(\operatorname{Cov}(U_i,V_i)=0.65\) and both scores have variance one, what is their correlation?
  3. Why can \(\Sigma_{XY}\) have nonzero off-diagonal entries even when the canonical score cross-covariance is diagonal?
  4. Does zero covariance between two canonical scores establish that they are independent? Explain.
  5. If one canonical score is multiplied by three, what happens to its variance and its covariance with its paired score? Does its correlation with that score change?

Previous

Next

Back to Tutorials
Clinical Biostats