From Covariance Blocks to Canonical Pairs
Canonical correlation analysis looks for linear summaries of two variable sets that are strongly associated. For two-variable sets, each summary is a weighted sum of two variables, but finding the best weights still requires respecting the variability and correlation within each set. A large cross-covariance alone does not identify a strong association if the variables themselves have very different scales.
Write the centered random vectors as \(X=(X_1,X_2)'\) and \(Y=(Y_1,Y_2)'\). Their joint covariance matrix has within-set blocks \(\Sigma_{XX}=\operatorname{Cov}(X,X)\), \(\Sigma_{YY}=\operatorname{Cov}(Y,Y)\), and cross-block \(\Sigma_{XY}=\operatorname{Cov}(X,Y)\). The aim is to find coefficient vectors \(a\) and \(b\) such that the scores \(U=a'X\) and \(V=b'Y\) have unit variance and their correlation is as large as possible. A second pair is chosen subject to the canonical orthogonality conditions for the first pair.
The positive-definiteness condition makes the inverse square roots well-defined. The singular values are nonnegative; in a valid joint covariance matrix they cannot exceed one. Coefficient signs are not unique: changing both \(a\) and \(b\) to their negatives leaves the pair's correlation unchanged. For two-variable sets there are two canonical correlations, conventionally ordered from largest to smallest.
Why the Whitening Calculation Works
The calculation reduces the covariance blocks to an ordinary singular value decomposition. Suppose \(K v=r u\) and \(K'u=r v\), where \(u\) and \(v\) are unit vectors. Define \(a=\Sigma_{XX}^{-1/2}u\) and \(b=\Sigma_{YY}^{-1/2}v\). Then \(a'\Sigma_{XX}a=u'u=1\) and \(b'\Sigma_{YY}b=v'v=1\). Also, \(a'\Sigma_{XY}b=u'Kv=r\). Thus the variances of the scores are one and their covariance, hence their correlation, is \(r\).
Proof. Set \(u=\Sigma_{XX}^{1/2}a\) and \(v=\Sigma_{YY}^{1/2}b\). The unit-variance restrictions are exactly \(u'u=v'v=1\), and the score covariance is \(a'\Sigma_{XY}b=u'Kv\). For any unit \(u,v\), the Cauchy–Schwarz inequality gives \(|u'Kv|\leq \|Kv\|\leq \|K\|_2\), where \(\|K\|_2\) is the largest singular value. Equality is attained by a leading left and right singular-vector pair, with signs chosen to make \(u'Kv\) nonnegative. Restricting to directions orthogonal to that pair gives the next singular value by the singular value decomposition. Therefore the stated maxima are the singular values in order. \(\square\)
This gives a convenient check on any hand calculation: the proposed coefficient vectors should have unit score variances, and their score covariance should equal the corresponding singular value. When the joint covariance matrix is valid, the same calculation cannot produce a canonical correlation above one. For example, after whitening, the covariance matrix is \(\begin{pmatrix}I&K\\K'&I\end{pmatrix}\); its positive semidefiniteness requires every singular value of \(K\) to be at most one.
Worked Calculations
Worked Example: Unequal Variances and Diagonal Cross-Covariance
Suppose the covariance blocks are \(\Sigma_{XX}=\begin{pmatrix}4&0\\0&1\end{pmatrix}\), \(\Sigma_{YY}=\begin{pmatrix}1&0\\0&9\end{pmatrix}\), and \(\Sigma_{XY}=\begin{pmatrix}1&0\\0&9/10\end{pmatrix}\). Compute both canonical pairs and check their score variances and correlations.
The inverse square roots are \(\Sigma_{XX}^{-1/2}=\operatorname{diag}(1/2,1)\) and \(\Sigma_{YY}^{-1/2}=\operatorname{diag}(1,1/3)\). Therefore \(K=\Sigma_{XX}^{-1/2}\Sigma_{XY}\Sigma_{YY}^{-1/2}=\operatorname{diag}(1/2,3/10)\). Its singular values are \(r_1=1/2\) and \(r_2=3/10\). The singular directions are the two coordinate vectors, so the coefficient vectors are \(a_1=(1/2,0)'\), \(b_1=(1,0)'\), \(a_2=(0,1)'\), and \(b_2=(0,1/3)'\).
For the first pair, \(\operatorname{Var}(a_1'X)=a_1'\Sigma_{XX}a_1=(1/2)^2(4)=1\) and \(\operatorname{Var}(b_1'Y)=b_1'\Sigma_{YY}b_1=1\). Its covariance is \(a_1'\Sigma_{XY}b_1=(1/2)(1)(1)=1/2\), so its correlation is \(1/2\). For the second pair, the variances are \(a_2'\Sigma_{XX}a_2=1\) and \(b_2'\Sigma_{YY}b_2=(1/3)^2(9)=1\). Its covariance is \(a_2'\Sigma_{XY}b_2=(1)(9/10)(1/3)=3/10\), so its correlation is \(3/10\). The within-set and cross-pair covariances are zero because these blocks are diagonal. The joint covariance is valid: the whitened cross-block has singular values below one.
Worked Example: Correlated Directions in Both Sets
Let \(\Sigma_{XX}=\Sigma_{YY}=I_2\) and \(\Sigma_{XY}=\begin{pmatrix}3/5&1/5\\1/5&3/5\end{pmatrix}\). Find and verify the canonical pairs.
Here \(K=\Sigma_{XY}\). Its unit eigenvectors are \(u_1=v_1=(1,1)'/\sqrt{2}\) and \(u_2=v_2=(1,-1)'/\sqrt{2}\). Multiplication gives \(Ku_1=(4/5)u_1\) and \(Ku_2=(2/5)u_2\). Since the matrix is symmetric with positive eigenvalues, these are also its singular values and singular-vector pairs. Thus \(r_1=4/5\), \(r_2=2/5\), and \(a_j=b_j=u_j\).
Both covariance blocks are identity matrices, so each score variance is the squared length of its coefficient vector: \(a_j'a_j=b_j'b_j=1\). The first cross-covariance is \(a_1'\Sigma_{XY}b_1=u_1'Ku_1=4/5\), and the second is \(a_2'\Sigma_{XY}b_2=u_2'Ku_2=2/5\). Since both score variances equal one, these are also the score correlations. The cross-covariance between the two \(X\) scores is \(u_1'u_2=0\), and likewise for the two \(Y\) scores. The largest singular value is \(4/5<1\), consistent with a valid joint covariance matrix.
Worked Example: Different Directions Across the Sets
Let \(\Sigma_{XX}=\Sigma_{YY}=I_2\) and \(\Sigma_{XY}=\begin{pmatrix}3/(5\sqrt{2})&1/(5\sqrt{2})\\3/(5\sqrt{2})&-1/(5\sqrt{2})\end{pmatrix}\). Compute the canonical directions and verify both pairs.
Write \(u_1=(1,1)'/\sqrt{2}\), \(u_2=(1,-1)'/\sqrt{2}\), \(e_1=(1,0)'\), and \(e_2=(0,1)'\). The first column of \(\Sigma_{XY}\) is \((3/5)u_1\), and the second is \((1/5)u_2\). Hence \(K e_1=(3/5)u_1\) and \(K e_2=(1/5)u_2\). The vectors \(u_1,u_2\) are orthonormal, so these equations exhibit a singular value decomposition directly: \(r_1=3/5\), \(r_2=1/5\), with left vectors \(u_1,u_2\) and right vectors \(e_1,e_2\).
The coefficient vectors are \(a_1=u_1,b_1=e_1\) and \(a_2=u_2,b_2=e_2\). Each has squared length one, so all four score variances equal one. Directly, \(a_1'\Sigma_{XY}b_1=u_1'((3/5)u_1)=3/5\), and \(a_2'\Sigma_{XY}b_2=u_2'((1/5)u_2)=1/5\). Thus the score correlations match the computed roots. The cross-covariance between the two \(X\) scores is \(u_1'u_2=0\); between the two \(Y\) scores it is \(e_1'e_2=0\). The largest singular value \(3/5\) is less than one, so the block covariance matrix is positive semidefinite.
Checks, Interpretation, and a Common Pitfall
A reliable two-by-two calculation is not complete when the roots have been found. For each pair, verify the score variances using the original within-set blocks and the score covariance using the original cross-block. These direct checks catch mistakes in matrix order, coefficient scaling, and the interpretation of a singular vector as a coefficient vector. In particular, the left and right singular vectors are coefficients only when both within-set covariance blocks are identity matrices; otherwise they must be transformed by the appropriate inverse square roots.
A common mistake is to treat the entries of \(\Sigma_{XY}\) as canonical correlations, or to choose the largest entry and pair its variables. This ignores the scaling and correlations within \(X\) and \(Y\). The correction is to form \(K=\Sigma_{XX}^{-1/2}\Sigma_{XY}\Sigma_{YY}^{-1/2}\), find its singular values and vectors, and then transform those vectors back to coefficient vectors. For instance, in the first worked example, the raw cross-covariances are \(1\) and \(9/10\), but the canonical correlations are \(1/2\) and \(3/10\) after accounting for the variances of the variables.
Another issue is sign. If \(a\) is replaced by \(-a\) while \(b\) is unchanged, the correlation changes sign; if both are negated, it does not. Reporting a positive canonical correlation conventionally means choosing signs so the paired score covariance is nonnegative. If two singular values tie, the corresponding canonical directions are not uniquely determined: orthogonal rotations within the tied singular subspace give equally valid pairs.
Check Your Understanding
For each question, use the covariance-block calculation and distinguish whitened singular vectors from the final coefficient vectors.
- If both within-set covariance blocks are identity matrices, how do the canonical coefficient vectors relate to the singular vectors of the cross-covariance block?
- For diagonal within-set blocks and diagonal cross-covariance, what quantities determine the two canonical correlations?
- Why must the singular values of the whitened cross-covariance matrix be at most one for a valid joint covariance matrix?
- If both coefficient vectors in a canonical pair are multiplied by negative one, what happens to the score variances and correlation?
- What additional check should be made after computing singular values and coefficient vectors?