Why the Choice Depends on the Question
Canonical correlation analysis (CCA) and two-block partial least squares (PLS) both form linear scores from two matched data blocks. Their first pair of scores can look similar, but the methods optimize different quantities. CCA seeks directions with large correlation; PLS seeks directions with large covariance. When feature scales and within-block variances differ, these objectives can favor different patterns in the same data.
Let \(X\) be a centered \(n\times p\) data matrix and \(Y\) a centered \(n\times q\) data matrix, with the same \(n\) individuals in corresponding rows. Define sample covariance matrices \(S_{XX}=X^\top X/(n-1)\), \(S_{YY}=Y^\top Y/(n-1)\), and \(S_{XY}=X^\top Y/(n-1)\). The examples use the first pair of scores; sequential components and deflation choices can affect later pairs.
The two-block PLS criterion here is the covariance-maximizing version defined in the earlier material on PLS as covariance maximization. It is not a claim that all procedures called PLS use the same objective: PLS regression, for example, has a response-prediction aim. The distinction in this tutorial is between CCA’s variance-normalized correlation and the stated two-block covariance objective.
What the Objectives Do to Scale
For CCA, suppose both covariance matrices are positive definite. Put \(u=S_{XX}^{1/2}a\) and \(v=S_{YY}^{1/2}b\). The CCA constraints become \(u^\top u=v^\top v=1\), and the covariance being maximized becomes \(u^\top S_{XX}^{-1/2}S_{XY}S_{YY}^{-1/2}v\). The maximum is therefore the largest singular value of the whitened cross-covariance matrix. Whitening expresses cross-block association relative to the variation within each block.
For PLS, the maximum of \(a^\top S_{XY}b\) under unit Euclidean weight constraints is the largest singular value of \(S_{XY}\). The corresponding left and right singular vectors give the weights. Unlike CCA’s whitened matrix, \(S_{XY}\) retains the original feature scales and within-block variation. Multiplying a feature by a constant can therefore change the PLS solution, even if its correlation pattern with the other block stays the same.
Worked Example: Unequal Variances Reverse the Preferred Direction
Consider four centered, mutually orthogonal sample vectors \(z_1,z_2,z_3,z_4\), each with sample variance 1. Treat them as hypothetical data patterns across the same individuals. Construct two features in each block as \(x_1=10z_1,\ x_2=z_2,\ y_1=0.3z_1+\sqrt{0.91}z_3,\ y_2=0.8z_2+0.6z_4\). These definitions give \(\operatorname{Var}(x_1)=100\), \(\operatorname{Var}(x_2)=1\), and \(\operatorname{Var}(y_1)=\operatorname{Var}(y_2)=1\). The within-block covariances are zero. The only nonzero cross-covariances are \(\operatorname{Cov}(x_1,y_1)=10(0.3)=3\) and \(\operatorname{Cov}(x_2,y_2)=0.8\). The covariance 3 is in the product of the units of \(x_1\) and \(y_1\); covariance has product units, not the units of either score alone.
Thus \(S_{XY}\) is diagonal with entries 3 and 0.8. Its largest singular value is 3, so PLS selects the first feature pair: \(a=(1,0)^\top\), \(b=(1,0)^\top\). The covariance of these scores is 3, and their variances are 100 and 1. Their correlation is \(3/\sqrt{100\cdot1}=0.3\).
For CCA, the two feature-pair correlations are \(3/\sqrt{100\cdot1}=0.3\) and \(0.8/\sqrt{1\cdot1}=0.8\). The whitened cross-covariance is diagonal with entries 0.3 and 0.8, so CCA selects the second pair, whose correlation is 0.8. CCA and PLS disagree because the first pair’s larger covariance is accompanied by much larger variance in \(x_1\). All covariance matrices here are nonsingular, so the ordinary first-component calculations are defined.
Collinearity and the Stability of Directions
Collinearity means that some features within a block are strongly linearly related. It can make a covariance matrix nearly singular. CCA’s whitening step then involves inverse square roots of small eigenvalues, which can amplify small perturbations in the data. The resulting canonical correlation may be high while the individual weight coefficients are unstable. PLS does not invert the within-block covariance matrices in its basic SVD solution, but its weights can still vary with sampling noise, scaling, or near-ties among singular values. Avoiding an inverse is not the same as guaranteeing stable or interpretable weights.
Worked Example: Collinearity Favors a Contrast in CCA
Suppose \(X\) has two features with covariance matrix \(S_{XX}=\begin{pmatrix}1&0.99\\0.99&1\end{pmatrix}\), and \(Y\) is one feature with variance 1. The eigenvectors of \(S_{XX}\) are the common direction \(u=(1,1)^\top/\sqrt{2}\), with eigenvalue 1.99, and the contrast direction \(v=(1,-1)^\top/\sqrt{2}\), with eigenvalue 0.01. Let the cross-covariance vector be \(c=S_{XY}=0.1u+0.01v\). This is a valid covariance specification because \(c^\top S_{XX}^{-1}c=0.1^2/1.99+0.01^2/0.01\approx0.0150<1=\operatorname{Var}(Y)\), so the implied joint covariance matrix is positive definite.
For the PLS objective with scalar \(Y\), the maximizing unit-length weight is \(a=c/\lVert c\rVert\). Its coordinates in the \(u,v\) directions are proportional to \((0.1,0.01)\), so it is mostly in the common, higher-variance direction. The maximum covariance is \(\lVert c\rVert=\sqrt{0.1^2+0.01^2}\approx0.1005\).
For CCA, the maximizing direction is proportional to \(S_{XX}^{-1}c\). In the eigenbasis its coordinates are \((0.1/1.99,\ 0.01/0.01)\approx(0.0503,\ 1)\), so it emphasizes the low-variance contrast. The squared canonical correlation is \(c^\top S_{XX}^{-1}c\approx0.0150\), giving a canonical correlation of about 0.1226. CCA’s correlation is greater than the correlation of the PLS-selected score, as expected from its maximizing objective. The contrast receives substantial weight because its covariance with \(Y\), though small, is large relative to its small variance.
This construction illustrates a potential instability, not a defect in the algebra: if the estimated eigenvalue 0.01 or the small cross-covariance component changes modestly, the inverse-eigenvalue weighting can change the CCA coefficients considerably. A near-zero eigenvalue also makes the data’s effective rank an important diagnostic.
Small Samples Can Make Sample Association Misleading
Both methods are estimated from the same matched observations, so a small sample can produce patterns that do not generalize. After centering, all columns lie in a space of dimension at most \(n-1\). If the column spaces of \(X\) and \(Y\) overlap, there are score vectors that are proportional across the observed individuals. Such scores have sample correlation 1 even if the overlap is a chance feature of the sample rather than a reproducible population relationship.
Worked Example: A Perfect Sample Correlation with Four Individuals
Take \(n=4\) individuals and three mutually orthogonal centered vectors \(z_1,z_2,z_3\), each with sample variance 1. Set \(X=(z_1,z_2)\) and \(Y=(z_1,z_3)\). Then \(S_{XX}=S_{YY}=I\), and \(S_{XY}=\begin{pmatrix}1&0\\0&0\end{pmatrix}\). Both methods select the first feature from each block, and their sample scores are both \(z_1\). Their sample correlation is exactly 1, and their covariance is 1 in the product of the units of the two scores. The construction is mathematically valid, but the perfect sample match alone does not establish a perfect population association.
More generally, if the centered column spaces have dimensions \(r_X\) and \(r_Y\) within the \((n-1)\)-dimensional centered observation space, their intersection has dimension at least \(r_X+r_Y-(n-1)\). When this lower bound is positive, a shared sample direction is guaranteed. This is one reason sample size must be considered relative to block dimensions, rather than treating a large in-sample canonical correlation as self-validating.
Choosing and Checking a Method
A practical choice starts with the scientific target, then checks whether the data support the intended calculation. The decision is not made by comparing in-sample maxima alone.
If the goal is association between score patterns after accounting for within-block variances, CCA’s correlation objective is aligned with that goal. If covariance in the original measurement scales is meaningful, covariance-maximizing PLS may be appropriate. State the scale convention, because changing it can change PLS.
Inspect block dimensions relative to \(n\), covariance eigenvalues, and condition numbers. Classical CCA’s inverse-covariance formulation requires nonsingular within-block covariance matrices. Singular or nearly singular matrices call for a method designed for that setting, such as ridge-regularized CCA, rather than an unexamined inverse.
Repeat the analysis under scientifically defensible feature scaling and assess whether weights and score associations are stable. Use held-out data or resampling to examine reproducibility. A strong training-sample association is not evidence by itself that the association will persist in new individuals.
Standardizing each feature to unit variance before analysis changes the question. It removes original marginal scale from PLS’s covariance criterion and can make the resulting objective resemble association among standardized features, but it does not make every feature equally reliable or cure collinearity. CCA is invariant to nonsingular rescaling of individual features in the exact, unregularized problem, because its score variances are normalized; its estimated weights and numerical stability can nevertheless be affected by near-singularity.
| Issue | CCA | Two-block covariance PLS |
|---|---|---|
| Target | Maximize score correlation | Maximize score covariance under unit-length weights |
| Feature rescaling | Exact canonical correlations are invariant to nonsingular rescaling | Can change the cross-covariance SVD and selected directions |
| Collinearity | Near-singularity can destabilize whitening and weights | No within-block inverse in the basic SVD, but weights can still be unstable |
| Small samples | Sample correlations can be spuriously high or perfect | Covariance directions can also overfit and need validation |
Common Mistakes and Limitations
- Comparing covariance values as if they were correlations. Covariance depends on the measurement scales and has product units; correlation is dimensionless. Compare each method using its own objective.
- Assuming PLS is automatically robust to collinearity. Its basic SVD avoids inversion, but correlated features can still yield unstable or difficult-to-interpret weights.
- Calling a large sample canonical correlation a population finding. It describes the fitted sample unless supported by an appropriate inferential analysis and evidence of generalization.
- Ignoring method variants. The criteria here concern classical CCA and covariance-maximizing two-block PLS. Regularization, feature selection, and regression-oriented PLS alter the problem and require their own assumptions and tuning choices.
Neither method establishes causation, identifies a unique biological mechanism, or guarantees that a linear score captures all relevant cross-block dependence. Both can miss nonlinear relationships, and both depend on which features were measured and how the observations were matched. In high dimensions, stability assessment and independent validation are central to interpretation, not optional additions.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In one sentence, state the difference between the first-component CCA and covariance-PLS objectives used here.
- If one \(X\) feature is multiplied by 5, what can happen to the covariance-PLS solution? What happens to the exact classical CCA correlations if the rescaling is nonsingular?
- In the collinearity example, why does CCA emphasize the contrast direction even though its cross-covariance with \(Y\) is smaller?
- For \(n=5\), suppose centered block column spaces have dimensions \(r_X=3\) and \(r_Y=2\). What does the dimension bound imply about their possible intersection?
Solutions. (1) CCA maximizes correlation with score variances normalized to one; the stated PLS maximizes covariance with Euclidean weight lengths normalized to one. (2) Multiplying one feature by 5 changes \(S_{XY}\), so it can change the PLS singular vectors and selected direction. The exact classical CCA correlations are invariant under nonsingular feature rescaling, although numerical conditioning and weight representations may change. (3) CCA weights cross-covariance relative to within-block variance: the contrast’s variance is only 0.01, so its covariance component 0.01 is large relative to its standard deviation. (4) The centered observation space has dimension at most \(n-1=4\), and the intersection dimension is at least \(3+2-4=1\); a shared sample direction is guaranteed under those stated dimensions.