Orthogonality of Successive Canonical Pairs

Why Successive Canonical Pairs Are Orthogonal

Canonical correlation analysis (CCA) constructs paired linear summaries of two random vectors. The first pair is chosen to have the greatest possible correlation; later pairs describe further association subject to orthogonality constraints. Those constraints are more than a convenient convention: they follow directly from the singular value decomposition used to define the canonical directions.

This tutorial establishes the covariance identities for successive pairs. We work with centered random vectors $X\in\mathbb{R}^p$ and $Y\in\mathbb{R}^q$, with finite second moments. Centering loses no covariance information. Write their covariance matrices and cross-covariance matrix as

\Sigma_{XX}=\operatorname{Cov}(X,X),\qquad \Sigma_{YY}=\operatorname{Cov}(Y,Y),\qquad \Sigma_{XY}=\operatorname{Cov}(X,Y).

Assume that $\Sigma_{XX}$ and $\Sigma_{YY}$ are positive definite, so their inverse square roots exist. Define the whitened cross-covariance matrix

K=\Sigma_{XX}^{-1/2}\Sigma_{XY}\Sigma_{YY}^{-1/2}.

Let $m=\min(p,q)$. Take a singular value decomposition $K=U D V^mathsf{T}$, with left and right singular vectors $u_i$ and $v_i$, and singular values $\rho_1\geq\cdots\geq\rho_m\geq 0$. For each $i$, define weights and canonical scores by

a_i=\Sigma_{XX}^{-1/2}u_i,\qquad b_i=\Sigma_{YY}^{-1/2}v_i,\qquad U_i=a_i^{\mathsf{T}}X,\qquad V_i=b_i^{\mathsf{T}}Y.

Here $U_i$ and $V_i$ are scalar random variables; the uppercase $U$ in the decomposition denotes the matrix of left singular vectors. The singular vectors are orthonormal within their respective spaces, and their signs are paired so that $Kv_i=\rho_i u_i$. When a singular value is repeated, the singular vectors within its singular subspace need not be unique, but they can be chosen as an orthonormal singular-vector basis.

The Covariance Identities

Theorem: Under the assumptions above, the canonical scores satisfy, for $1\leq i,j\leq m$, $$ \operatorname{Cov}(U_i,U_j)=\delta_{ij},\qquad \operatorname{Cov}(V_i,V_j)=\delta_{ij},\qquad \operatorname{Cov}(U_i,V_j)=\rho_i\delta_{ij}, $$ where $\delta_{ij}=1$ if $i=j$ and $0$ otherwise. Thus distinct scores in either set are uncorrelated, and a score in one set is uncorrelated with every nonmatching score in the other set.

Proof. The covariance of two linear combinations is obtained by multiplying the covariance matrix by their weight vectors. For scores in the $X$ set, this gives

\begin{aligned} \operatorname{Cov}(U_i,U_j) &=a_i^{\mathsf{T}}\Sigma_{XX}a_j\\ &=u_i^{\mathsf{T}}\Sigma_{XX}^{-1/2} \Sigma_{XX}\Sigma_{XX}^{-1/2}u_j\\ &=u_i^{\mathsf{T}}u_j =\delta_{ij}. \end{aligned}

The inverse square root is symmetric, and $\Sigma_{XX}^{-1/2}\Sigma_{XX}\Sigma_{XX}^{-1/2}=I$, which justifies the third expression. The same calculation for the $Y$ set gives

\begin{aligned} \operatorname{Cov}(V_i,V_j) &=b_i^{\mathsf{T}}\Sigma_{YY}b_j\\ &=v_i^{\mathsf{T}}\Sigma_{YY}^{-1/2} \Sigma_{YY}\Sigma_{YY}^{-1/2}v_j\\ &=v_i^{\mathsf{T}}v_j =\delta_{ij}. \end{aligned}

For a cross-covariance, use the definition of $K$:

\begin{aligned} \operatorname{Cov}(U_i,V_j) &=a_i^{\mathsf{T}}\Sigma_{XY}b_j\\ &=u_i^{\mathsf{T}}K v_j\\ &=u_i^{\mathsf{T}}(\rho_j u_j)\\ &=\rho_j\delta_{ij} =\rho_i\delta_{ij}. \end{aligned}

The last equality holds because when $i\ne j$, both sides are zero, and when $i=j$, $\rho_i=\rho_j$. These calculations prove all three identities. $\square$

In particular, the variance of each score is one, so its covariance with a matching score is also its correlation: $\operatorname{Corr}(U_i,V_i)=\rho_i$. The cross-covariance matrix of the first $m$ scores is diagonal. Its diagonal entries are the canonical correlations, and its off-diagonal entries vanish.

Worked Example: Diagonal Whitened Cross-Covariance

Suppose $X=(X_1,X_2)^{\mathsf{T}}$ and $Y=(Y_1,Y_2)^{\mathsf{T}}$ are jointly centered with $\Sigma_{XX}=\Sigma_{YY}=I$ and

\Sigma_{XY}=K= \begin{pmatrix} 0.8&0\\ 0&0.3 \end{pmatrix}.

The joint covariance matrix is positive definite: after ordering the coordinates as $(X_1,Y_1,X_2,Y_2)$, it has two blocks with eigenvalues $1\pm0.8$ and $1\pm0.3$, all of which are positive. The singular vectors are the coordinate vectors, and the singular values are $0.8$ and $0.3$. Thus $U_1=X_1,\ V_1=Y_1,\ U_2=X_2,\ V_2=Y_2$.

Within the first set, $\operatorname{Var}(U_1)=\operatorname{Var}(U_2)=1$ and $\operatorname{Cov}(U_1,U_2)=0$, because the off-diagonal entry of $\Sigma_{XX}$ is zero. The same is true for $V_1,V_2$. Across sets, $\operatorname{Cov}(U_1,V_1)=0.8$, $\operatorname{Cov}(U_2,V_2)=0.3$, and $\operatorname{Cov}(U_1,V_2)=\operatorname{Cov}(U_2,V_1)=0$. The matching pairs carry the association; the nonmatching pairs do not.

Orthogonality After a Change of Directions

The canonical scores need not coincide with individual coordinates. Singular vectors can combine several original variables. Orthogonality refers to covariance of the resulting scores, not necessarily to any one original variable or to a geometric angle in the unscaled coordinates.

Worked Example: Rotated Scores and a Repeated Calculation

Let both marginal covariance matrices be $I$, and suppose

\Sigma_{XY}=K= \begin{pmatrix} 0.4&0.3\\ 0.3&0.4 \end{pmatrix}.

This symmetric matrix has orthonormal eigenvectors $u_1=v_1=(1,1)^{\mathsf{T}}/\sqrt{2}$ and $u_2=v_2=(1,-1)^{\mathsf{T}}/\sqrt{2}$. Multiplication verifies the corresponding singular values:

K\begin{pmatrix}1\\1\end{pmatrix} =\begin{pmatrix}0.7\\0.7\end{pmatrix}, \qquad K\begin{pmatrix}1\\-1\end{pmatrix} =\begin{pmatrix}0.1\\-0.1\end{pmatrix}.

The scores are $U_1=(X_1+X_2)/\sqrt{2}$, $U_2=(X_1-X_2)/\sqrt{2}$, and the corresponding expressions in $Y$. Their within-set covariance is

\operatorname{Cov}(U_1,U_2) =\frac{1}{2}\operatorname{Cov}(X_1+X_2,X_1-X_2) =\frac{1}{2}\{1-0+0-1\}=0.

Similarly, $\operatorname{Cov}(V_1,V_2)=0$. For the cross-set calculations,

\begin{aligned} \operatorname{Cov}(U_1,V_1) &=\tfrac12(0.4+0.3+0.3+0.4)=0.7,\\ \operatorname{Cov}(U_1,V_2) &=\tfrac12(0.4-0.3+0.3-0.4)=0,\\ \operatorname{Cov}(U_2,V_1) &=\tfrac12(0.4+0.3-0.3-0.4)=0,\\ \operatorname{Cov}(U_2,V_2) &=\tfrac12(0.4-0.3-0.3+0.4)=0.1. \end{aligned}

The joint covariance is positive definite: in the rotated coordinates its cross-covariance is diagonal with entries $0.7$ and $0.1$, so its eigenvalues are $1\pm0.7$ and $1\pm0.1$, all positive. This example shows how score orthogonality can arise through cancellation even when the original variables have nonzero cross-covariances.

Why Nonsingular Joint Covariance Bounds the Roots

Positive definite marginal covariance matrices are enough to define the whitening transformation. If the full joint covariance matrix is also positive definite, then every canonical correlation is strictly less than one. This condition rules out a nonzero linear combination of one set being an exact copy of a linear combination of the other set.

Proposition: If the joint covariance matrix of $X$ and $Y$ is positive definite and both marginal covariance matrices are positive definite, then every singular value of $K$ satisfies $0\leq\rho_i<1$.

Proof. Whitening is an invertible linear transformation, so the covariance matrix of the whitened vectors is positive definite as well. It is

\begin{pmatrix} I&K\\ K^{\mathsf{T}}&I \end{pmatrix}.

Take any unit left and right singular vectors $u_i,v_i$ with singular value $\rho_i$. The variance of the scalar difference $u_i^{\mathsf{T}}\Sigma_{XX}^{-1/2}X-v_i^{\mathsf{T}}\Sigma_{YY}^{-1/2}Y$ is the quadratic form of this positive definite matrix at $(u_i,-v_i)$:

\begin{aligned} \operatorname{Var}\!\left( u_i^{\mathsf{T}}\Sigma_{XX}^{-1/2}X- v_i^{\mathsf{T}}\Sigma_{YY}^{-1/2}Y\right) &=u_i^{\mathsf{T}}u_i+v_i^{\mathsf{T}}v_i -2u_i^{\mathsf{T}}Kv_i\\ &=2-2\rho_i. \end{aligned}

The vector $(u_i,-v_i)$ is nonzero, so positive definiteness makes this variance strictly positive. Hence $2-2\rho_i>0$, which gives $\rho_i<1$. Singular values are nonnegative by definition, so $0\leq\rho_i<1$. $\square$

Worked Example: Unequal Marginal Scales

Suppose

\Sigma_{XX}= \begin{pmatrix}4&0\\0&1\end{pmatrix}, \qquad \Sigma_{YY}= \begin{pmatrix}1&0\\0&9\end{pmatrix}, \qquad \Sigma_{XY}= \begin{pmatrix}1&0\\0&0.6\end{pmatrix}.

The inverse square roots are $\Sigma_{XX}^{-1/2}=\operatorname{diag}(1/2,1)$ and $\Sigma_{YY}^{-1/2}=\operatorname{diag}(1,1/3)$. Therefore

K= \begin{pmatrix}1/2&0\\0&1\end{pmatrix} \begin{pmatrix}1&0\\0&0.6\end{pmatrix} \begin{pmatrix}1&0\\0&1/3\end{pmatrix} = \begin{pmatrix}0.5&0\\0&0.2\end{pmatrix}.

The singular values are $0.5$ and $0.2$, with coordinate singular vectors. The canonical weights and scores are

a_1=(1/2,0)^{\mathsf{T}},\quad a_2=(0,1)^{\mathsf{T}}, \qquad b_1=(1,0)^{\mathsf{T}},\quad b_2=(0,1/3)^{\mathsf{T}},

so $U_1=X_1/2,\ U_2=X_2,\ V_1=Y_1,\ V_2=Y_2/3$. Directly, $\operatorname{Var}(U_1)=4/4=1$, $\operatorname{Var}(U_2)=1$, $\operatorname{Var}(V_1)=1$, and $\operatorname{Var}(V_2)=9/9=1$. The within-set cross-covariances vanish because the marginal covariance matrices are diagonal. Across sets, $\operatorname{Cov}(U_1,V_1)=1/2=0.5$, $\operatorname{Cov}(U_2,V_2)=0.6/3=0.2$, and the two nonmatching covariances are zero. The unequal original scales have been accounted for by whitening.

The joint covariance is positive definite: after whitening, its eigenvalues are $1\pm0.5$ and $1\pm0.2$, each positive, and the whitening map is invertible. Thus this example also satisfies the strict-root condition.

Interpretation and Common Pitfall

The diagonal cross-covariance identity means that each canonical score is paired with the score of the same index in the other set. It does not mean that the two sets of scores are independent. Zero covariance alone does not imply independence for general distributions; independence follows from zero covariance for these scores if, for example, the full score vector is jointly multivariate normal.

Common mistake: “The first and second canonical pairs are orthogonal, so $U_1$ must be independent of $U_2$, and all four scores must be independent.” The covariance theorem establishes only that distinct scores within each set, and nonmatching scores across sets, are uncorrelated. It does not establish independence without additional distributional assumptions. Also, the matching scores are generally correlated: $\operatorname{Cov}(U_i,V_i)=\rho_i$, not zero. When singular values are tied, the particular score directions can vary with the orthonormal basis chosen in the tied subspace, but the stated covariance identities still hold for any such paired singular-vector basis.

Key takeaway: Whitening makes the marginal score variances one, while orthonormal singular vectors make distinct scores within each set uncorrelated. The singular-vector pairing makes the cross-covariance matrix diagonal, with canonical correlations on its diagonal.

Check Your Understanding

Use the covariance identities and assumptions above to answer these questions.

Which property of the left singular vectors gives $\operatorname{Cov}(U_i,U_j)=0$ when $i\ne j$?
What is $\operatorname{Cov}(U_i,V_j)$ when $i\ne j$, and what is it when $i=j$?
Why are the canonical scores in the unequal-scale example standardized to variance one?
Does zero covariance between $U_1$ and $U_2$ by itself imply that they are independent? State a condition under which independence would follow.
Why does positive definiteness of the full joint covariance matrix rule out a canonical correlation equal to one?