Ordering, Normalizing, and Signing Canonical Roots

Introduction

A canonical correlation analysis produces a sequence of associations between linear combinations of two variable sets. To interpret that sequence, three conventions matter: correlations are ordered from largest to smallest, each score is scaled to have variance one, and the signs of a pair are not uniquely determined. These are consequences of the singular-value representation, not arbitrary reporting choices.

Let \(X\) and \(Y\) be random column vectors of dimensions \(p\) and \(q\), with finite second moments and positive-definite covariance matrices \(\Sigma_{XX}\) and \(\Sigma_{YY}\). Write \(\Sigma_{XY}=\operatorname{Cov}(X,Y)\), and let \(r=\min(p,q)\). The discussion uses centered variables, or equivalently linear scores formed from centered \(X\) and \(Y\); means do not affect the covariance calculations.

The word “root” is often used for \(\lambda_i\), an eigenvalue in a generalized eigenvalue formulation; the canonical correlation itself is its nonnegative square root, \(\rho_i\). Keeping the two quantities distinct avoids confusing a correlation of \(0.6\) with its squared root \(0.36\).

Why the Canonical Correlations Are Ordered

Whitening removes the marginal covariance scales of the two sets. If \(K=U D V^T\) is a singular value decomposition, its singular values appear in nonincreasing order on the diagonal of \(D\). This order is also the order of the largest linear associations available between the two whitened sets. In coefficient form, \(a^T\Sigma_{XX}a=1\) and \(b^T\Sigma_{YY}b=1\) imply that \(u=\Sigma_{XX}^{1/2}a\) and \(v=\Sigma_{YY}^{1/2}b\) each have Euclidean length one, and \(\operatorname{Cov}(a^TX,b^TY)=u^TKv\).

Proof. For unit vectors \(u\in\mathbb{R}^p\) and \(v\in\mathbb{R}^q\), the covariance of the corresponding whitened scores is \(u^TKv\). For fixed \(v\), the Cauchy–Schwarz inequality gives \(|u^TKv|\leq\|u\|\,\|Kv\|=\|Kv\|\). If \(Kv\neq0\), equality is attained by taking \(u=Kv/\|Kv\|\); if \(Kv=0\), both sides are zero for every unit \(u\). Therefore the maximum over both unit vectors is \(\max_{\|v\|=1}\|Kv\|\). By the spectral theorem, \(\|Kv\|^2=v^TK^TKv\), whose maximum over unit \(v\) is the largest eigenvalue of \(K^TK\). Its square root is the largest singular value \(\rho_1\), proving the maximum claim. The remaining singular values are defined in nonincreasing order by the eigenvalues of \(K^TK\) and form the remaining canonical correlations. Since \(K\) has \(r\) singular values, the canonical-root list consists of the largest \(r\) eigenvalues of \(K^TK\), not necessarily every eigenvalue if \(q>p\). \(\square\)

This dimension detail is important. The matrix \(K^TK\) is \(q\times q\). If \(q>p\), it has \(q-p\) additional zero eigenvalues beyond the \(r=p\) squared singular values. Those extra zeros are not extra canonical roots. For instance, if \(p=1\), \(q=2\), and \(K=(1\ \ 0)\), then \(K^TK\) has eigenvalues \(1\) and \(0\), but \(r=1\), so the canonical-root list contains only \(1\). In all dimensions, the \(r\) roots are the largest \(r\) eigenvalues, or, equivalently, the squares of the \(r\) singular values.

Normalizing Coefficients to Give Unit-Variance Scores

Let \(u_i\) and \(v_i\) be unit-length left and right singular vectors paired with singular value \(\rho_i\), so \(Kv_i=\rho_i u_i\) and \(K^Tu_i=\rho_i v_i\) when \(\rho_i>0\). Convert these vectors back to coefficients for the original variable sets: \(a_i=\Sigma_{XX}^{-1/2}u_i\) and \(b_i=\Sigma_{YY}^{-1/2}v_i\). The inverse square roots undo the whitening in the score coefficients. The singular vectors have Euclidean length one, but the original coefficients are generally not Euclidean unit vectors. Their relevant normalization is with respect to their own covariance matrices.

Derivation. Using symmetry of the inverse square root and \(\Sigma_{XX}^{-1/2}\Sigma_{XX}\Sigma_{XX}^{-1/2}=I_p\), \(a_i^T\Sigma_{XX}a_i=u_i^Tu_i=1\). Likewise, \(b_i^T\Sigma_{YY}b_i=v_i^Tv_i=1\). Their cross-covariance is \(a_i^T\Sigma_{XY}b_i=u_i^T K v_i=\rho_i u_i^Tu_i=\rho_i\). Since both variances are one, the correlation is the covariance divided by \(\sqrt{1\cdot1}\), hence also \(\rho_i\). This proves the stated normalization and association. \(\square\)

Why the Roots Cannot Exceed One

Canonical correlations must lie between zero and one. This follows from the requirement that the joint covariance matrix be positive semidefinite, not from a numerical convention. Whitening the two marginal blocks transforms the joint covariance matrix to \(\begin{pmatrix}I_p&K\\K^T&I_q\end{pmatrix}\). For any \(y\in\mathbb{R}^q\), evaluate its quadratic form at the vector \((-Ky,y)^T\). Positive semidefiniteness gives \((-Ky)^T(-Ky)+2(-Ky)^TKy+y^Ty\geq0\), which simplifies to \(y^T(I_q-K^TK)y\geq0\). Thus \(I_q-K^TK\) is positive semidefinite. Every eigenvalue of \(K^TK\) is between zero and one, so every singular value of \(K\), and hence every canonical correlation, is between zero and one.

Sign Choices and What They Mean

If \((u_i,v_i)\) is a paired pair of singular vectors, then \((-u_i,-v_i)\) is also a valid pair: the two minus signs cancel in \(u_i^TKv_i\). The corresponding original coefficients change to \((-a_i,-b_i)\), and both scores change sign. Their variances remain one and their covariance remains \(\rho_i\). The data description changes orientation, but the paired association does not.

Changing just one sign is different. It changes the covariance from \(\rho_i\) to \(-\rho_i\), while leaving both score variances unchanged. Because a singular value is nonnegative, canonical correlations are conventionally reported as nonnegative; signs for the two coefficient vectors are chosen together to maintain that convention. If \(\rho_i=0\), the covariance is zero regardless of the paired sign choice. If a singular value is repeated, its singular vectors need not be unique even apart from sign; ordering fixes the values, not a unique direction within a tied subspace.

Common Pitfalls and Why the Conventions Matter

• Counting every eigenvalue of \(K^TK\) as a canonical root. This fails when \(q>p\), because \(K^TK\) then has additional zeros. The canonical-root list has exactly \(r=\min(p,q)\) entries: the largest \(r\) eigenvalues, counted with multiplicity.
• Normalizing coefficient vectors to Euclidean length one. That generally does not make their scores have variance one. The correct requirements are \(a_i^T\Sigma_{XX}a_i=1\) and \(b_i^T\Sigma_{YY}b_i=1\).
• Flipping only one score and calling the pair unchanged. Only simultaneous reversal preserves the signed covariance. Reversing one score changes its sign.
• Assuming singular vectors are uniquely determined. A sign can always be reversed, and tied singular values can allow more than one valid choice of directions. Reported roots may be fixed even when coefficient directions are not unique.

These conventions make the sequence comparable across variable sets with different measurement scales: each score has variance one, and the displayed correlations describe association rather than units. Descending order identifies the strongest available linear association first. Sign conventions make reported correlations nonnegative, while recognizing that the orientation of either score has no substantive meaning on its own.