Why Compare the Fits, Not Just the Variable Lists?
Sparse CCA and sparse PLS both produce low-dimensional scores from paired data blocks, and both can set some feature weights to zero. But their objectives differ: sparse CCA seeks association between scores after accounting for their variances, whereas sparse PLS seeks large cross-block covariance. A variable selected by one method is therefore not automatically selected by the other, and neither support alone establishes that an association will reproduce in new samples.
The comparison below uses an intentionally compact hypothetical example. The variables are centered, paired across the same samples, and expressed in their original measurement units. We impose a very strong sparsity budget—one active variable in each block—so the selection criterion can be calculated directly. This is a special, transparent case of sparse fitting, not a recommendation to restrict every analysis to one variable.
For a single selected feature \(X_j\) and a single selected feature \(Y_k\), the two criteria reduce to
Here \(s_{X_jY_k}\) is their sample covariance, and \(s_{X_j}\) and \(s_{Y_k}\) are their sample standard deviations. The correlation is unitless; covariance depends on the measurement units. In larger-support fits, the distinction is not limited to units: CCA’s score-variance normalization and PLS’s covariance objective can favor different combinations of correlated features.
A Worked Training-Set Comparison
Assume that 40 hypothetical training subjects have measurements on two features in each block. The reported sample variances and cross-covariances are as follows. The within-block covariances are zero in this example, so the listed variances fully specify each block’s covariance matrix.
| Quantity | Feature 1 | Feature 2 |
|---|---|---|
| \(X\) sample variance | 1 | 9 |
| \(Y\) sample variance | 1 | 4 |
| Cross-covariance | \(Y_1\) | \(Y_2\) |
|---|---|---|
| \(X_1\) | 0.75 | 0.20 |
| \(X_2\) | 0.30 | 1.20 |
Worked Example: Selecting One Feature From Each Block
With a budget of one active feature per block, calculate all four candidate correlations and identify each method’s training-set selection.
For CCA, divide each cross-covariance by the product of the corresponding sample standard deviations. The standard deviations are \(s_{X_1}=1\), \(s_{X_2}=3\), \(s_{Y_1}=1\), and \(s_{Y_2}=2\). Thus the four candidate correlations are
The largest absolute correlation is \(0.75\), so the one-pair sparse CCA fit selects \(X_1\) and \(Y_1\). For the corresponding one-pair sparse PLS comparison, the candidate covariances are \(0.75\), \(0.20\), \(0.30\), and \(1.20\). The largest absolute covariance is \(1.20\), so PLS selects \(X_2\) and \(Y_2\). Both selected associations are positive. These are training-set choices under the stated sparsity budget, not claims that either association is statistically significant or will generalize.
This calculation illustrates a central point: matching the broad goal of relating two blocks does not make the optimization criteria interchangeable. Here, CCA chooses the strongest unitless pairwise association, while PLS chooses the largest covariance in the original units. In an analysis with more than one active feature per block, the methods also assess weighted combinations, and within-block covariance affects the variance normalization in CCA.
Checking Selected Scores on Held-Out Subjects
Selection should be separated from evaluation. Suppose the two supports above were fixed using only the training data. For held-out subjects, form scores using those selected features and the corresponding fitted weights. The held-out samples must not have been used to choose features, tune sparsity, or otherwise alter the fitted model. For a one-feature score, multiplying a variable by a nonzero weight does not change its correlation with another one-feature score, though it can change the covariance.
Worked Example: Computing Held-Out Score Associations
For four hypothetical held-out subjects, let \(u=(-1,1,-1,1)\) and \(w=(-1,-1,1,1)\). Both vectors have mean zero, squared length 4, and dot product zero. Suppose the held-out CCA scores, up to positive scale factors, are \(u\) and \(0.6u+0.8w\). Their correlation is
For the PLS-selected features, suppose the held-out scores are \(3u\) and \(2(0.3u+\sqrt{0.91}w)\). The factors 3 and 2 are positive and do not affect correlation. Since \(u^\mathsf{T}w=0\) and \(\sqrt{0.3^2+0.91}=1\), the score correlation is
In this hypothetical split, the CCA-selected score pair has held-out correlation \(0.60\), and the PLS-selected pair has held-out correlation \(0.30\). These four held-out observations are for arithmetic illustration only: four subjects cannot provide a reliable performance estimate. In an application, report the held-out association with its sample size and uncertainty, and use a validation design that reflects the intended prediction or replication setting.
Why Supports Can Change
The numerical comparison above deliberately uses original measurement units. This makes the scale dependence visible. If \(X_2\) is converted to \(X_2/3\), its sample standard deviation becomes 1. Its covariance with \(Y_2\) becomes \(1.20/3=0.40\), while its correlation with \(Y_2\) remains \(0.20\). The covariance between \(X_1\) and \(Y_1\) stays \(0.75\). Under the same one-pair rule, PLS now selects \(X_1,Y_1\) rather than \(X_2,Y_2\); CCA’s four pairwise correlations and its selection are unchanged. This is not a change in the underlying association: only the unit used to record \(X_2\) changed.
Worked Example: Checking the Effect of Rescaling
Rescale \(X_2\) to \(X_2^*=X_2/3\). The PLS candidate covariance for the second pair is
The other cross-covariances are \(s_{X_1Y_1}=0.75\), \(s_{X_1Y_2}=0.20\), and \(s_{X_2^*Y_1}=0.30/3=0.10\). The largest is now \(0.75\), so PLS selects \(X_1,Y_1\). For CCA,
the same correlation as before; the other pairwise correlations are likewise unchanged by this positive rescaling. Therefore, CCA still selects \(X_1,Y_1\). The example shows why units and preprocessing must be reported when interpreting sparse PLS supports.
Even if features are standardized before fitting, do not assume the two methods must select identical supports. With multiple active variables, CCA accounts for the variances of both weighted scores, which depend on within-block covariance. PLS optimizes cross-block covariance under its own constraints or penalty. Correlated features, the sparsity penalty or budget, and the tuning rule can consequently alter which combination is preferred. Standardization removes differences in marginal units; it does not make the objectives identical.
Assumptions, Diagnostics, and Limits
A meaningful comparison requires that rows in \(X\) and \(Y\) refer to the same subjects or experimental units, with a consistent feature definition and preprocessing. Centering and scaling parameters must be estimated from training data and then applied unchanged to validation data. The chosen sparsity constraints, any penalty parameters, and the criterion for choosing them must be specified. For example, held-out association can guide tuning, but repeatedly consulting the same held-out set makes it part of model selection and biases its apparent performance upward. Use a separate test set or an appropriate nested validation procedure when tuning is required.
Useful diagnostics include checking that the selected score variances are nonzero, that the fitted objective actually improves over a null or permuted pairing, and that selected features and score associations are not driven by a small number of observations. Repeat the fit across resamples or folds to assess support stability. A high training association paired with weak or inconsistent held-out association is a warning of overfitting, not evidence that the held-out result should be ignored. When comparing methods, use the same outer splits and a clearly stated performance metric; otherwise, differences may reflect validation design rather than the fitting method.
Sparse supports are also conditional on the data representation and tuning choices. If two features carry nearly redundant information, one resample may select one and another resample the other, even when the score association is stable. Conversely, a stable feature list does not guarantee a strong reproducible association. Selected features and their weights describe a fitted multivariate association; they do not establish causal effects. Permutation tests for two-block association address a different question and should not be confused with support stability or held-out performance.
The main alternative is not simply “CCA or PLS” in isolation. One can alter scaling, sparsity constraints, penalties, or validation criteria, and these choices may change the answer. Classical CCA and classical PLS provide useful non-sparse reference fits, but neither replaces a validation assessment of sparse selections. The reported comparison should make clear which formulation was fit, how sparsity was imposed, and what association was evaluated out of sample.
Check Your Understanding
Answer the questions, then check the solutions below.
- In the training example, what correlation does CCA assign to \(X_2,Y_2\), and which one-pair support does sparse CCA choose?
- Why does dividing \(X_2\) by 3 change the PLS selection but not the CCA correlations?
- If feature scaling and support are fixed before evaluation, what does a held-out score correlation of 0.30 describe?
- Name two reasons that supports may vary across resamples even when an overall score association is similar.
Solutions. (1) The correlation is \(1.20/(3\cdot2)=0.20\); CCA chooses \(X_1,Y_1\), whose correlation is \(0.75\). (2) Dividing \(X_2\) by 3 divides its covariance with every \(Y\) feature by 3, so the PLS covariance ranking can change. Correlation divides by the standard deviation as well, so the positive rescaling cancels and the correlations remain unchanged. (3) It is the sample linear association, measured by correlation, between the two fixed scores on the held-out subjects; it is not a causal effect and does not by itself quantify uncertainty. (4) Correlated or redundant features can substitute for one another, and finite-sample variation can change which features pass the sparsity constraint. Tuning choices can also affect the selected support.