Why Successive Pairs Matter
Canonical correlation analysis (CCA) seeks linear scores from two data blocks that are strongly associated. Once the first pair has captured the strongest association, later pairs describe further association not already represented by that pair. This sequential construction is useful only if its constraints are understood: a second pair is not simply another high-correlation pair chosen without reference to the first.
Let \(X\) be an \(n \times p\) matrix and \(Y\) an \(n \times q\) matrix, with rows referring to the same \(n\) observations. Assume both matrices have been column-centered. Write their sample covariance matrices as \(S_{xx}=X^\mathsf{T}X/(n-1)\), \(S_{yy}=Y^\mathsf{T}Y/(n-1)\), and \(S_{xy}=X^\mathsf{T}Y/(n-1)\). For weight vectors \(a\) and \(b\), the corresponding scores are \(u=Xa\) and \(v=Yb\). Their sample variances and covariance are \(a^\mathsf{T}S_{xx}a\), \(b^\mathsf{T}S_{yy}b\), and \(a^\mathsf{T}S_{xy}b\).
The constraints are stated in terms of score covariances. For example, \(u_k\) must have zero sample covariance with each earlier \(u_j\), and \(v_k\) must have zero sample covariance with each earlier \(v_j\). They do not require the weight vectors themselves to have zero ordinary dot product.
The Sequential Optimization Problem
For the first pair, CCA maximizes the correlation between \(Xa\) and \(Yb\), subject to each score having variance 1. Once the first \(k-1\) pairs have been found, the \(k\)th pair maximizes correlation subject to the same unit-variance constraints and within-block orthogonality to every earlier score. With centered data, the correlation equals the covariance when both score variances are 1.
Here the objective is written with a positive sign for the covariance. Reversing the sign of either weight vector reverses its score, so a pair can be oriented to have nonnegative covariance without changing the substantive solution. In the standard full-rank setting, the successive maximum values are ordered \(\rho_1\geq \rho_2\geq \cdots\geq 0\).
For this ordinary formulation, \(S_{xx}\) and \(S_{yy}\) must be positive definite, so that each block has nonsingular sample covariance and the unit-variance constraints are meaningful in every direction. The observations must be paired across blocks, and both matrices must use the same centered observations. These are algebraic conditions for the solution described here, not claims that any particular data satisfy them. If a covariance matrix is singular, the inverse-square-root derivation below does not apply directly; that case requires separate treatment. No distributional assumption is needed to establish the sample covariance identities.
Why the Scores Are Orthogonal
When both within-block covariance matrices are positive definite, define the whitened cross-covariance matrix \(C=S_{xx}^{-1/2}S_{xy}S_{yy}^{-1/2}\). The symmetric inverse square roots rescale each block so that its covariance becomes the identity. Take a singular value decomposition \(C=P D Q^\mathsf{T}\), where the columns \(p_k\) of \(P\) and \(q_k\) of \(Q\) are orthonormal, and the diagonal entries of \(D\) are the singular values \(\rho_k\). Set \(a_k=S_{xx}^{-1/2}p_k\) and \(b_k=S_{yy}^{-1/2}q_k\).
These statements follow by direct substitution, not by an assumption that the original variables are mutually uncorrelated. For example, the covariance of two scores from the \(X\) block is \(a_i^\mathsf{T}S_{xx}a_j=p_i^\mathsf{T}p_j=\delta_{ij}\). In the \(Y\) block the calculation is \(b_i^\mathsf{T}S_{yy}b_j=q_i^\mathsf{T}q_j=\delta_{ij}\). For scores across blocks, \(a_i^\mathsf{T}S_{xy}b_j=p_i^\mathsf{T}Cq_j=\rho_j\delta_{ij}\), by the defining singular-vector relation. Thus each pair has covariance \(\rho_j\), and distinct pairs have zero cross-covariance as well as zero covariance within each block.
The SVD also justifies the sequential order. After the covariance normalization, the objective is \(p^\mathsf{T}Cq\) for unit vectors \(p\) and \(q\). Its maximum is the largest singular value of \(C\). Requiring \(p\) and \(q\) to be orthogonal to their previously selected singular vectors leaves the corresponding orthogonal subspaces; the maximum on those subspaces is the next singular value. Repeating this argument gives the descending sequence. If singular values are tied, the directions within the tied subspace need not be unique, even though the tied correlations are the same.
Worked Calculations
Worked Example: Covariance-Metric Orthogonality
Suppose a hypothetical \(X\) block has sample covariance \(S_{xx}=\begin{pmatrix}4&1\\1&2\end{pmatrix}\). Consider weights \(a_1=(1,0)^\mathsf{T}\) and \(a_2=(-1,4)^\mathsf{T}\). Check whether their scores are uncorrelated, and compare this with the ordinary dot product of the weights.
The score covariance is computed using \(S_{xx}\):
So the scores are uncorrelated. But the ordinary dot product is \(a_1^\mathsf{T}a_2=(1,0)(-1,4)^\mathsf{T}=-1\), not zero. The weights are orthogonal in the \(S_{xx}\) covariance metric, not in ordinary Euclidean geometry. They are not yet unit-variance weights: the score variances are \(a_1^\mathsf{T}S_{xx}a_1=4\) and \(a_2^\mathsf{T}S_{xx}a_2=28\). Dividing \(a_1\) by 2 and \(a_2\) by \(\sqrt{28}\) gives unit-variance scores and preserves their zero covariance.
Worked Example: Computing Two Pairs From a Small Dataset
Consider a hypothetical dataset of five paired observations and two variables in each block. Define four centered five-entry vectors: \(e_1=\sqrt{2}(1,-1,0,0,0)^\mathsf{T}\), \(e_2=(2/\sqrt{6})(1,1,-2,0,0)^\mathsf{T}\), \(e_3=(1/\sqrt{3})(1,1,1,-3,0)^\mathsf{T}\), and \(e_4=(1/\sqrt{5})(1,1,1,1,-4)^\mathsf{T}\). The data columns are \(X_1=e_1,\ X_2=e_2,\ Y_1=0.8e_1+0.6e_3,\) and \(Y_2=0.3e_2+\sqrt{0.91}e_4\). These definitions specify all 20 measurements; they give a compact exact representation of a small hypothetical dataset.
Each vector has sum zero. Their Euclidean squared lengths are 4, and their pairwise dot products are zero. Therefore, with \(n-1=4\), \(e_i^\mathsf{T}e_j/4=\delta_{ij}\). It follows that \(S_{xx}=I_2\). For \(Y_1\), its sample variance is \(0.8^2+0.6^2=0.64+0.36=1\); for \(Y_2\), it is \(0.3^2+(\sqrt{0.91})^2=0.09+0.91=1\). Their covariance is zero because the vectors used for \(Y_1\) are orthogonal to those used for \(Y_2\). Hence \(S_{yy}=I_2\).
The cross-block covariance entries follow from the same dot products:
Since both within-block covariance matrices are identity matrices, the whitened cross-covariance is this same diagonal matrix. Its singular values are 0.8 and 0.3, in that order, with corresponding directions \(a_1=b_1=(1,0)^\mathsf{T}\) and \(a_2=b_2=(0,1)^\mathsf{T}\). Thus \(u_1=X_1,\ v_1=Y_1,\ u_2=X_2,\) and \(v_2=Y_2\).
For the first pair, the variances are both 1 and the covariance is \(0.8\), so its correlation is \(0.8/(\sqrt{1}\sqrt{1})=0.8\). For the second pair, both variances are 1 and the covariance is \(0.3\), so its correlation is \(0.3\). The within-\(X\) covariance of the scores is \(e_1^\mathsf{T}e_2/4=0\), and the within-\(Y\) covariance is \(0.8(0.3)\operatorname{Cov}(e_1,e_2)+0.8\sqrt{0.91}\operatorname{Cov}(e_1,e_4)+0.6(0.3)\operatorname{Cov}(e_3,e_2)+0.6\sqrt{0.91}\operatorname{Cov}(e_3,e_4)=0\). The off-diagonal cross-block covariances are also zero: \(\operatorname{Cov}(u_1,v_2)=0\) and \(\operatorname{Cov}(u_2,v_1)=0\), by orthogonality of the four basis vectors. This verifies both pairs and their required orthogonality from the sample data.
Checks, Comparisons, and Failure Modes
The result gives practical checks on a computed CCA solution. Using the same centering and covariance convention as the fit, calculate score variances and covariances from the observed scores. Under unit-variance normalization, each diagonal within-block covariance should be 1, and off-diagonal within-block covariances should be near 0. The covariance between the \(k\)th matched scores should agree with the reported \(\rho_k\); off-diagonal cross-block covariances should be near 0. Small numerical deviations can arise from finite-precision arithmetic, but large discrepancies suggest a scaling, indexing, or implementation problem.
These properties are about linear sample covariance. Zero covariance does not generally imply statistical independence, and it does not imply that the corresponding original variables are independent. Nor do sample score correlations establish a causal relationship between the two blocks.
| Approach or quantity | What it does | How it differs from sequential CCA |
|---|---|---|
| Sequential CCA | Finds paired linear scores, maximizing their correlation subject to unit variance and within-block score orthogonality. | Its paired directions are chosen to represent ordered cross-block associations. |
| PCA within one block | Finds directions of high variance within a single block, with uncorrelated component scores. | High within-block variance is not the same objective as high correlation with the other block. |
| Whitened cross-covariance SVD | Computes singular vectors and values of \(S_{xx}^{-1/2}S_{xy}S_{yy}^{-1/2}\). | Under the stated full-rank conditions, this is a computational route to the same CCA pairs, not a different set of targets. |
A common mistake is to require \(a_i^\mathsf{T}a_j=0\) and \(b_i^\mathsf{T}b_j=0\), then treat failure of those equalities as evidence that the scores are not orthogonal. That reasoning confuses weights with scores. The correct checks are \(a_i^\mathsf{T}S_{xx}a_j=0\) and \(b_i^\mathsf{T}S_{yy}b_j=0\), because those are precisely the score covariances.
Other failure modes are using uncentered data when the intended analysis includes an intercept, standardizing scores inconsistently, or sorting correlations without keeping each weight vector paired with its corresponding score. Sign changes can make a reported score correlation negative without changing the fitted direction; a consistent sign convention resolves that presentation issue. Near-tied canonical correlations can make individual directions unstable: small data changes may rotate directions within a nearly tied subspace even when the corresponding correlations change little.
The ordinary SVD argument also has a clear limitation: it uses inverse square roots of \(S_{xx}\) and \(S_{yy}\). If either matrix is singular, or nearly singular numerically, the calculation may fail or become unstable; the full-rank derivation should not be treated as if it automatically resolves that situation. Moreover, score orthogonality is a property of the fitted sample construction, not a guarantee that the directions will remain orthogonal in a new sample.
Worked Example: Ordering and Sign Convention
In the hypothetical dataset above, the first and second canonical correlations are 0.8 and 0.3. Suppose software reports the second pair with \(a_2=(0,-1)^\mathsf{T}\) and \(b_2=(0,1)^\mathsf{T}\). The resulting scores are \(-X_2\) and \(Y_2\), so their covariance is \(-0.3\) and their correlation is \(-0.3\). Their absolute association is unchanged. Reversing the sign of \(a_2\) gives covariance and correlation \(0.3\) again, while leaving both within-block orthogonality checks unchanged. The pair still belongs second because its correlation magnitude is 0.3, below the first pair’s 0.8.
Check Your Understanding
Use the definitions and results above to answer each question. Brief solutions follow the questions.
- What covariance constraint expresses orthogonality between \(u_3\) and \(u_1\)?
- If the \(k\)th score pair has unit variances and covariance 0.42, what is its canonical correlation?
- In the small dataset, what is \(\operatorname{Cov}(u_1,v_2)\), and why?
- Why does \(a_1^\mathsf{T}a_2\neq 0\) not by itself contradict CCA orthogonality?
Solutions. (1) \(a_3^\mathsf{T}S_{xx}a_1=0\). (2) The correlation is \(0.42/(\sqrt{1}\sqrt{1})=0.42\). (3) It is 0 because \(e_1\) is orthogonal to both \(e_2\) and \(e_4\), the vectors that form \(v_2\). (4) Score orthogonality is defined by \(a_1^\mathsf{T}S_{xx}a_2=0\), not by the ordinary dot product of the weights.