Loadings and Cross-Loadings in CCA

Why CCA Needs More Than Its Weights

Canonical correlation analysis (CCA) constructs linear scores from two sets of variables so that paired scores have a large correlation. The weights used to form those scores describe the construction; they do not, by themselves, describe how strongly each original variable is associated with a score. For interpretation, it is useful to calculate correlations between the original variables and the canonical scores.

These correlations are called loadings when the variable and score come from the same block, and cross-loadings when they come from opposite blocks. They answer related but distinct questions: a loading describes a variable’s association with its own block’s score, while a cross-loading describes its association with the paired score from the other block.

Definition: Let \(X_1,\ldots,X_p\) and \(Y_1,\ldots,Y_q\) be variables observed on the same units. For a canonical pair of scores \(U\) and \(V\), a loading is a within-block correlation, such as \(\operatorname{cor}(X_j,U)\) or \(\operatorname{cor}(Y_k,V)\). A cross-loading is a cross-block correlation, such as \(\operatorname{cor}(X_j,V)\) or \(\operatorname{cor}(Y_k,U)\). The weights in the score definitions, such as \(a_j\) in \(U=\sum_j a_jX_j\), are coefficients; they are not these correlations.

Calculating Loadings and Cross-Loadings

In a sample of \(n\) paired observations, let \(x_{ij}\) and \(y_{ik}\) be the recorded values of \(X_j\) and \(Y_k\). Suppose the corresponding canonical scores are \(u_i\) and \(v_i\). For any two variables \(A\) and \(B\), their sample correlation is the sample covariance divided by the product of their sample standard deviations. Thus, for a variable \(X_j\) and score \(U\),

\widehat{\operatorname{cor}}(X_j,U) = \frac{\sum_{i=1}^{n}(x_{ij}-\bar{x}_j)(u_i-\bar{u})} {\sqrt{\sum_{i=1}^{n}(x_{ij}-\bar{x}_j)^2 \sum_{i=1}^{n}(u_i-\bar{u})^2}}.

Replacing \(X_j,U\) with \(Y_k,V\), \(X_j,V\), or \(Y_k,U\) gives the other within-block or cross-block correlations. If scores are centered, then \(\bar{u}=\bar{v}=0\), apart from numerical rounding. The same individuals must contribute to each correlation; otherwise the correlation matrix no longer describes the paired CCA analysis.

For classical CCA, scores are often scaled to have variance one. This convention makes their correlations easy to compare, but it does not make the score weights equal to the loadings. A loading is calculated from the resulting score and original variable, so it incorporates both the weight vector and the covariance structure among variables.

Worked Example: A Zero Weight With a Nonzero Loading

Consider centered, standardized variables \(X_1,X_2,Y\) with hypothetical correlations \(\operatorname{cor}(X_1,X_2)=0.50\), \(\operatorname{cor}(X_1,Y)=0.60\), and \(\operatorname{cor}(X_2,Y)=0.30\). Take the score pair \(U=X_1\) and \(V=Y\), so both scores have variance one. Their correlation is \(0.60\).

The \(X\)-block weight vector for \(U\) is \((1,0)\): \(X_2\) has zero weight in this score. But its loading is \[ \operatorname{cor}(X_2,U)=\operatorname{cor}(X_2,X_1)=0.50. \] The \(X_1\) loading is \(\operatorname{cor}(X_1,U)=1\). Thus, \(X_2\) is associated with the score despite not appearing directly in its formula. Its association arises because it is correlated with \(X_1\).

The cross-loadings are \(\operatorname{cor}(X_1,V)=0.60\) and \(\operatorname{cor}(X_2,V)=0.30\). On the \(Y\) side, the loading is \(\operatorname{cor}(Y,V)=1\), and its cross-loading is \(\operatorname{cor}(Y,U)=0.60\). This example shows why coefficients and variable-to-score correlations must not be treated as interchangeable.

The Cross-Loading Identity

For a classical CCA pair, each cross-loading can be obtained by multiplying the corresponding within-block loading by the canonical correlation, provided the scores have unit variance and the same variables and observations are used. The result follows from the covariance equations that characterize a canonical pair.

Theorem: Let \(U=a^\mathsf{T}X\) and \(V=b^\mathsf{T}Y\) be a classical canonical pair, with \(\operatorname{Var}(U)=\operatorname{Var}(V)=1\) and \(\operatorname{cor}(U,V)=\rho\). Then \(\operatorname{cor}(X_j,V)=\rho\,\operatorname{cor}(X_j,U)\) and \(\operatorname{cor}(Y_k,U)=\rho\,\operatorname{cor}(Y_k,V)\). The identities apply to the corresponding population quantities, and to sample correlations when the sample canonical solution satisfies the usual covariance equations.

To justify the first identity, write \(\Sigma_{XX}\) for the covariance matrix of \(X\), \(\Sigma_{XY}\) for the cross-covariance matrix, and \(\Sigma_{YY}\) for the covariance matrix of \(Y\). The canonical equations for a pair with unit-variance scores include \(\Sigma_{XY}b=\rho\Sigma_{XX}a\). Since \(U=a^\mathsf{T}X\) and \(V=b^\mathsf{T}Y\), this gives

\operatorname{Cov}(X,V) =\Sigma_{XY}b =\rho\Sigma_{XX}a =\rho\operatorname{Cov}(X,U).

Taking the \(j\)th component yields \(\operatorname{Cov}(X_j,V)=\rho\operatorname{Cov}(X_j,U)\). Divide both sides by \(\operatorname{SD}(X_j)\), since both scores have standard deviation one, to obtain the stated correlation identity. The second identity follows from the companion equation \(\Sigma_{YX}a=\rho\Sigma_{YY}b\) in the same way. This argument also shows why the result depends on the canonical equations: it is not a general identity for any two arbitrarily chosen scores.

Worked Example: Checking the Identity Numerically

In the hypothetical example above, the canonical correlation is \(\rho=0.60\). The loading of \(X_2\) on \(U\) is \(0.50\), so the cross-loading identity gives \[ \operatorname{cor}(X_2,V)=0.60(0.50)=0.30. \] This agrees with the specified correlation between \(X_2\) and \(Y=V\). For \(X_1\), the loading is \(1\), so the predicted cross-loading is \(0.60(1)=0.60\), again matching the specified correlation. On the other side, the loading of \(Y\) on \(V\) is \(1\); its cross-loading on \(U\) is therefore \(0.60(1)=0.60\), which matches \(\operatorname{cor}(Y,X_1)\).

This check is not an independent proof that the selected pair is the optimal CCA solution. It verifies that the reported loadings, cross-loadings, and canonical correlation are mutually consistent with the canonical covariance equations for this pair.

Weights, Loadings, and Interpretation

Weights answer, “Which linear combination defines the score?” Loadings answer, “How strongly is each original variable correlated with that score?” Cross-loadings answer, “How strongly is each variable correlated with the paired score from the other block?” These distinctions matter when variables within a block are correlated: one variable’s loading can be appreciable even if its own score weight is small or zero.

Weights are also affected by the scale of the input variables. Rescaling an input variable changes the numerical coefficient needed to form the same score, whereas the correlation of that variable with the score is scale-free apart from sign. A large weight is therefore not, on its own, evidence that a variable is more substantively important. Loadings are often easier to interpret as associations, but they too are descriptive correlations, not causal effects or proof that a variable drives the score.

The sign convention for a canonical pair must be handled consistently. If \(U\) is replaced by \(-U\), its loadings and its correlations with variables in the other block change sign. If \(V\) is replaced by \(-V\), its loadings and correlations involving \(V\) change sign. Reversing both scores leaves their mutual correlation unchanged; reversing only one reverses the sign of that correlation. Positive rescaling of a score leaves its correlations unchanged, while negative rescaling reverses the signs of correlations involving that score. The cross-loading identity remains valid when \(\rho\) and the relevant loadings are updated consistently.

Worked Example: Computing a Loading From Observations

Suppose four hypothetical paired observations produce centered values \(X_1=(-1,1,-1,1)\) and canonical scores \(U=(-1,-1,1,1)\). Each vector has mean zero, and each sum of squares is \(4\). The sample correlation is \[ \widehat{\operatorname{cor}}(X_1,U) =\frac{(-1)(-1)+(1)(-1)+(-1)(1)+(1)(1)} {\sqrt{4\cdot4}} =\frac{1-1-1+1}{4}=0. \] Thus the estimated loading is zero in this small example: the score is not linearly correlated with \(X_1\) in these observations. This calculation uses a sample correlation, not the weight used to create \(U\). A zero observed loading does not establish that the population association is exactly zero.

Diagnostics, Alternatives, and Limitations

Check the calculation and interpretation of loadings and cross-loadings before drawing conclusions. Confirm that scores and variables are paired on the same units, that each variable has nonzero variance, and that the reported score scaling is known. Recalculate selected correlations directly from the observations. For a classical unit-variance canonical pair, compare cross-loadings with \(\rho\) times the corresponding loadings; discrepancies beyond rounding can indicate inconsistent scaling, mismatched observations, or a computation that is not the classical CCA solution.

A loading near \(1\) or \(-1\) indicates a strong positive or negative linear association in the analyzed data; a loading near zero indicates a weak linear association. It does not rule out nonlinear dependence. The sign can change with the arbitrary sign choice of a canonical score, so interpret signs only after stating the chosen orientation. Loadings and cross-loadings can also be unstable when variables are highly correlated, the sample is small relative to the number of variables, or the estimated CCA solution is sensitive to the data. They summarize the fitted sample relationship and do not by themselves quantify uncertainty or establish generalizability.

Direct correlations between original variables and scores are the standard descriptive alternative to interpreting weights alone. Weights remain necessary to reproduce or apply a score, while correlations between individual variables across blocks describe pairwise associations rather than associations with a canonical score. Neither alternative has the same interpretation as a loading or cross-loading. In particular, a high loading is not a causal contribution, and a cross-loading is not a separate canonical correlation.

Common mistake: “The variable has a zero score weight, so it cannot be related to the score.” This is incorrect when the variable is correlated with variables that do have nonzero weights. In the worked example, \(X_2\) has weight zero in \(U=X_1\), but its loading is \(0.50\) because \(\operatorname{cor}(X_2,X_1)=0.50\). Calculate the variable-to-score correlation rather than inferring it from the weight alone.

Check Your Understanding

Answer each question, then compare with the solutions.

In one sentence each, distinguish a loading, a cross-loading, and a score weight.
A unit-variance canonical pair has \(\rho=0.40\), and \(\operatorname{cor}(X_1,U)=0.75\). Under the stated classical CCA conditions, what is \(\operatorname{cor}(X_1,V)\)?
If \(U\) is replaced by \(-U\), what happens to \(\operatorname{cor}(X_j,U)\) and \(\operatorname{cor}(U,V)\) when \(V\) is unchanged?
Why can a variable with zero weight in \(U\) have a nonzero loading on \(U\)?

Solutions. (1) A loading is a correlation between a variable and its own block’s score; a cross-loading is a correlation with the paired score from the other block; a weight is a coefficient in the linear score formula. (2) The cross-loading is \(0.40(0.75)=0.30\). (3) Both correlations reverse sign: \(\operatorname{cor}(X_j,-U)=-\operatorname{cor}(X_j,U)\) and \(\operatorname{cor}(-U,V)=-\operatorname{cor}(U,V)\). (4) The variable may be correlated with variables that have nonzero weights, so it can correlate with the resulting score even though it does not enter that score directly.