Why Interpretation Needs More Than a Large Association
CCA and PLS construct scores from two data blocks, but a striking association between fitted scores does not by itself establish a stable or scientifically interpretable relationship. In CCA, the score directions are chosen to maximize their sample correlation; in two-block PLS, they are chosen to maximize sample covariance under a normalization convention. Both objectives can produce results that are sensitive to the sample, the scaling, or the specific representation of the variables.
Let \(X\) be a centered \(n \times p\) data matrix and \(Y\) a centered \(n \times q\) data matrix, measured on the same \(n\) units. A pair of weight vectors \(a\) and \(b\) produces scores \(Xa\) and \(Yb\). Write \(\Sigma_{xx}\), \(\Sigma_{yy}\), and \(\Sigma_{xy}\) for the corresponding sample covariance matrices or cross-covariance matrix. CCA chooses directions to maximize the correlation of the scores, subject to nonzero score variances. A common two-block PLS-SVD formulation maximizes \(\operatorname{Cov}(Xa,Yb)\) subject to \(a'a=b'b=1\); its result depends on how the variables are scaled.
Inflated In-Sample Canonical Correlations
A large fitted canonical correlation can partly reflect the flexibility of the optimization. In each block, CCA searches across linear combinations, not just across the original variables. The more dimensions those combinations can use relative to the number of independent observations, the more opportunity there is to find sample scores that align by chance.
One way to see the mechanism is to rewrite the CCA objective in whitened coordinates. Suppose \(\Sigma_{xx}\) and \(\Sigma_{yy}\) are positive definite. Define \(u=\Sigma_{xx}^{1/2}a\) and \(v=\Sigma_{yy}^{1/2}b\). Then CCA’s objective becomes a bilinear form in \(u\) and \(v\), with matrix \(\Sigma_{xx}^{-1/2}\Sigma_{xy}\Sigma_{yy}^{-1/2}\). Maximizing over unit-length \(u\) and \(v\) gives the largest singular value of that matrix. This representation also exposes a danger: inverse square roots give considerable influence to directions with very small within-block variance.
When the feature counts are large relative to \(n\), the centered data matrices have rank at most \(n-1\), and their column spaces can occupy much of the same sample-centered space. In that setting, an optimized sample correlation can be perfect even when a perfect population relationship has not been established.
Worked Example: A Perfect Sample Correlation From Shared Sample Space
Consider four hypothetical matched observations. The centered sample space has dimension \(4-1=3\). Suppose each block has three columns that form a basis for this space. The bases need not contain the same columns, but each block can represent every centered four-vector.
Take the centered vector \(z=(1,-1,0,0)'\). Because it belongs to both column spaces, there are weights \(a\) and \(b\) such that \(Xa=z\) and \(Yb=z\). The sample variance of \(z\), using denominator \(n-1=3\), is \((1^2+(-1)^2)/3=2/3\), so each score has positive variance. Their sample correlation is
Thus the largest fitted canonical correlation is 1. This is a statement about these sample column spaces, not evidence that the underlying omics processes are perfectly related. If independent population blocks are sufficiently flexible relative to the sample size, their realized column spaces can still align unusually well. Out-of-sample assessment and study design are needed to judge generalizability.
Diagnostics should include the relationship between \(p\), \(q\), and \(n\), the effective ranks of the blocks, and whether the fitted association persists in held-out observations. Permutation tests, as introduced in the earlier material on permutation tests for two-block association, address evidence against a specified no-association null under the permutation scheme. They do not by themselves show that the estimated weights are stable or that the relationship generalizes to new data.
Unstable Weights and Near-Singular Covariance
CCA weights can change substantially when the data change slightly, especially when a block contains highly correlated variables or directions with very small variance. The objective depends on inverse within-block covariance. A small-variance direction can therefore receive a large coefficient, even if its estimated association is imprecise. Such a coefficient may reflect numerical sensitivity rather than a robust feature-level relationship.
Worked Example: A Small Covariance Change Reverses a Large Weight
Suppose a hypothetical two-variable block has \(\Sigma_{xx}=\begin{pmatrix}1&0\\0&0.0001\end{pmatrix}\), while the other block has one variable with variance 1. Let the cross-covariance vector be \(c=(0.2,0.005)'\). The squared maximum canonical correlation is \(c'\Sigma_{xx}^{-1}c\), provided the joint covariance is valid. Here it is \(0.2^2/1+0.005^2/0.0001=0.04+0.25=0.29\), which is below 1, so the implied squared correlation is feasible.
The unnormalized direction is \(\Sigma_{xx}^{-1}c=(0.2,50)'\). Its variance is \(0.2^2+50^2(0.0001)=0.29\), so the unit-variance CCA weights are
If a small change in the estimated second cross-covariance changes \(0.005\) to \(-0.005\), its standardized contribution changes sign. The squared canonical correlation remains \(0.29\), but the second weight becomes approximately \(-92.85\). The fitted association is the same size, while the direction assigned to the low-variance variable reverses. This is a warning against interpreting a large coefficient as strong, stable biological evidence.
Useful checks include examining the eigenvalues or condition number of each within-block covariance matrix, resampling observations to assess weight variability, and checking whether conclusions persist under reasonable preprocessing choices. Ridge penalties or sparsity constraints can reduce some forms of instability, but they change the estimation problem and do not guarantee that selected variables are reproducible. Selection and tuning must also be accounted for in validation.
Weights Are Not Loadings
Weights answer, “What coefficients form the score?” Loadings answer, “How strongly does each measured variable relate to that score?” These can differ sharply when variables are correlated. For a standardized CCA block and a score with variance 1, the loading vector is \(\Sigma_{xx}a\); its \(j\)th entry is the correlation between variable \(j\) and the score. For variables not standardized to unit variance, the correlation also divides by the variable’s standard deviation.
Worked Example: Large Opposing Weights, Modest Loadings
Suppose two standardized variables have covariance matrix \(\Sigma_{xx}=\begin{pmatrix}1&0.9\\0.9&1\end{pmatrix}\). Consider the weight vector \(a=(1,-1)'/\sqrt{0.2}\), so the score has variance \(a'\Sigma_{xx}a=1\). The weights are approximately \(2.236\) and \(-2.236\), but the loading vector is
The score is a contrast between two strongly correlated variables. The large, opposite coefficients define that contrast, but each variable has only a modest correlation with the resulting score. Reporting the weights as if they were variable-score correlations would overstate the individual variables’ relationships with the score. The same interpretive distinction matters in PLS: a weight constructs a latent score, while a loading summarizes a variable’s association with that score, with the precise loading convention stated.
Sign Ambiguity and Comparisons Between Fits
The sign of a latent direction is arbitrary. In CCA, changing \(a\) to \(-a\) reverses the first score; changing both \(a\) and \(b\) to their negatives leaves the paired score correlation unchanged. In PLS-SVD, the paired singular vectors can likewise both be sign-reversed without changing the fitted covariance. Consequently, a positive versus negative sign across separate fits is not automatically a biological reversal.
Worked Example: Equivalent PLS-SVD Signs
Suppose a two-block cross-covariance matrix is \(M=\begin{pmatrix}2&0\\0&0\end{pmatrix}\). One leading singular-vector pair is \(u=(1,0)'\), \(v=(1,0)'\), giving \(u'Mv=2\). The pair \(-u=(-1,0)'\), \(-v=(-1,0)'\) gives \((-u)'M(-v)=2\) as well. The two solutions describe the same association with opposite score orientations.
When comparing fits, choose and report a consistent orientation—for example, orient one score to have positive correlation with a prespecified reference score. If no defensible reference exists, describe the pattern as a relative sign structure, not as an absolute positive or negative direction.
CCA and PLS also answer different optimization questions. CCA targets correlation and can be especially sensitive to near-singular within-block covariance. The stated two-block PLS-SVD formulation targets covariance and its weights depend on variable scaling; it does not produce a canonical correlation merely because it produces paired scores. Ridge or sparse variants modify the objective or constrain the weights. These approaches may improve conditioning or interpretability, but none removes the need to inspect score quality, weight stability, and validation performance.
Common Pitfalls and Practical Checks
- Reading a fitted canonical correlation as generalizable. It is an optimized sample association. Check performance in held-out data and consider the feature counts and effective ranks.
- Treating the largest weight as the most important variable. Weights depend on scaling and correlations among variables. Inspect loadings and stability as well.
- Interpreting a sign flip as a changed biological effect. Latent score orientation is arbitrary. Align signs before comparing fits.
- Using a permutation p-value as a stability measure. A test of overall association and the reproducibility of individual weights are distinct questions.
- Assuming regularization solves every problem. Penalties impose trade-offs and can make results depend on tuning choices; validate the full procedure.
A defensible interpretation reports the method and scaling, distinguishes in-sample association from validation, describes both weights and loadings where relevant, and states whether the inferred pattern is stable under resampling or reasonable analysis choices. Even then, a fitted association is not evidence of causation.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- Why can two unrelated high-dimensional blocks yield a very large sample canonical correlation?
- What feature of \(\Sigma_{xx}\) can make CCA weights sensitive to small data changes?
- In a standardized CCA block with a unit-variance score, what does \(\Sigma_{xx}a\) represent?
- Why should the sign of a latent score not be interpreted without an orientation convention?
Solutions. (1) CCA optimizes over linear combinations, and high-dimensional sample column spaces may overlap substantially, allowing chance alignment. (2) A very small eigenvalue makes inverse covariance weighting amplify that direction and its estimation error. (3) Its entries are correlations between the variables and the score, that is, loadings, not the score-forming coefficients themselves. (4) Paired latent directions can be reversed without changing the association objective, so the sign is arbitrary until a reference orientation is fixed.