Interpreting Weights, Structure Loadings, and Cross-Loadings

Three Quantities, Three Interpretations

A canonical variate is a weighted sum of variables. Its coefficients are essential for constructing the score, but they are not themselves correlations between the original variables and that score. For interpretation, it is useful to distinguish canonical coefficient weights, structure loadings, and cross-loadings. Each answers a different question: how is the score constructed, how strongly is a variable associated with its own set’s score, and how strongly is it associated with the opposite set’s score?

Let $X$ be a $p$-dimensional random vector and $Y$ a $q$-dimensional random vector, with finite second moments and positive variances for all component variables. Write their covariance matrix in blocks:

\Sigma = \begin{pmatrix} \Sigma_{XX} & \Sigma_{XY}\\ \Sigma_{YX} & \Sigma_{YY} \end{pmatrix}, \qquad \Sigma_{YX}=\Sigma_{XY}'.

For a canonical pair with coefficient vectors $a_i$ and $b_i$, define the scores $U_i=a_i'X$ and $V_i=b_i'Y$. We use the usual normalization $\operatorname{Var}(U_i)=\operatorname{Var}(V_i)=1$, and write their canonical correlation as $\rho_i=\operatorname{Corr}(U_i,V_i)$. The components may be centered first; adding constants to variables does not change covariances or correlations.

Definition: The entries of $a_i$ and $b_i$ are the canonical coefficient weights for the two scores. The structure loading of $X_j$ on $U_i$ is $\operatorname{Corr}(X_j,U_i)$; the structure loading of $Y_k$ on $V_i$ is $\operatorname{Corr}(Y_k,V_i)$. The cross-loading of $X_j$ on $V_i$ is $\operatorname{Corr}(X_j,V_i)$, and that of $Y_k$ on $U_i$ is $\operatorname{Corr}(Y_k,U_i)$.

Calculating Loadings From Covariance Blocks

A weight is a coefficient in a linear combination; a loading is a correlation. The covariance blocks convert weights into correlations. For any coefficient vectors $a$ and $b$, the score variances and the covariances with the original variables are

\begin{aligned} \operatorname{Var}(a'X)&=a'\Sigma_{XX}a, &\operatorname{Var}(b'Y)&=b'\Sigma_{YY}b,\\ \operatorname{Cov}(X,a'X)&=\Sigma_{XX}a, &\operatorname{Cov}(Y,b'Y)&=\Sigma_{YY}b,\\ \operatorname{Cov}(X,b'Y)&=\Sigma_{XY}b, &\operatorname{Cov}(Y,a'X)&=\Sigma_{YX}a. \end{aligned}

The covariance expressions involving a vector are column vectors: their $j$th or $k$th entries are the corresponding component covariances. Let $D_X$ and $D_Y$ be diagonal matrices whose diagonal entries are the standard deviations of the components of $X$ and $Y$. Dividing each covariance by the appropriate variable and score standard deviations gives the following formulas.

Covariance-block formulas: For any nonzero score variances, $$ \begin{aligned} \operatorname{Corr}(X,a'X) &=\frac{D_X^{-1}\Sigma_{XX}a}{\sqrt{a'\Sigma_{XX}a}},& \operatorname{Corr}(Y,b'Y) &=\frac{D_Y^{-1}\Sigma_{YY}b}{\sqrt{b'\Sigma_{YY}b}},\\ \operatorname{Corr}(X,b'Y) &=\frac{D_X^{-1}\Sigma_{XY}b}{\sqrt{b'\Sigma_{YY}b}},& \operatorname{Corr}(Y,a'X) &=\frac{D_Y^{-1}\Sigma_{YX}a}{\sqrt{a'\Sigma_{XX}a}}. \end{aligned} $$ Each expression is a vector of correlations, one for each original variable.

Under unit-variance normalization, the denominators for the canonical scores are one. The covariance blocks still matter: the own-set loadings use the within-set covariance block, while the cross-loadings use the between-set block. In particular, the weight vector alone does not generally determine the loadings without the relevant covariance structure.

Worked Example: Two Independent Variable Pairs

Suppose $X=(X_1,X_2)'$ and $Y=(Y_1,Y_2)'$ have covariance blocks $\Sigma_{XX}=I_2$, $\Sigma_{YY}=I_2$, and $\Sigma_{XY}=\begin{pmatrix}0.8&0\\0&0.3\end{pmatrix}$. These blocks form a valid covariance matrix: after reordering the variables into the pairs $(X_1,Y_1)$ and $(X_2,Y_2)$, the two covariance matrices have eigenvalues $1\pm0.8$ and $1\pm0.3$, all positive.

The first canonical pair is $a_1=(1,0)'$, $b_1=(1,0)'$. Both scores have variance one, and $\operatorname{Cov}(U_1,V_1)=0.8$, so $\rho_1=0.8$. Since both within-set covariance matrices are identity matrices, the first own-set loading vectors are $(1,0)'$ and $(1,0)'$. The cross-loading vectors are also $(0.8,0)'$: for example, $\operatorname{Cov}(X,V_1)=\Sigma_{XY}b_1=(0.8,0)'$.

Here the selected weights and loadings happen to match for the own-set scores because the variables within each set are uncorrelated and have unit variances. The cross-loadings are different: they report the associations with the opposite-set score.

Canonical Equations Link Cross-Loadings to Structure Loadings

For a nonzero canonical correlation, the normalized canonical vectors satisfy the canonical equations

\Sigma_{XY}b_i=\rho_i\Sigma_{XX}a_i, \qquad \Sigma_{YX}a_i=\rho_i\Sigma_{YY}b_i.

These equations show that each cross-loading vector is the corresponding own-set loading vector multiplied by the canonical correlation. This relation is useful for checking calculations, but it does not make loadings and weights interchangeable.

Theorem: Suppose $U_i$ and $V_i$ have unit variance and satisfy the canonical equations above. Then $$ \operatorname{Corr}(X,V_i)=\rho_i\operatorname{Corr}(X,U_i), \qquad \operatorname{Corr}(Y,U_i)=\rho_i\operatorname{Corr}(Y,V_i). $$

Proof. Since $\operatorname{Var}(V_i)=1$, the vector of correlations between $X$ and $V_i$ is $\operatorname{Cov}(X,V_i)$ divided componentwise by the standard deviations of $X$. Therefore

\operatorname{Corr}(X,V_i) =D_X^{-1}\operatorname{Cov}(X,b_i'Y) =D_X^{-1}\Sigma_{XY}b_i =\rho_iD_X^{-1}\Sigma_{XX}a_i.

Because $\operatorname{Var}(U_i)=1$, $D_X^{-1}\Sigma_{XX}a_i=\operatorname{Corr}(X,U_i)$. This proves the first equality. Similarly,

\operatorname{Corr}(Y,U_i) =D_Y^{-1}\Sigma_{YX}a_i =\rho_iD_Y^{-1}\Sigma_{YY}b_i =\rho_i\operatorname{Corr}(Y,V_i),

which proves the second equality. $\square$

The theorem assumes unit score variances. Without that normalization, the correlations must include the score standard deviations in their denominators, so the proportionality must be adjusted accordingly.

Worked Example: Loadings With Unequal Measurement Scales

Let $\Sigma_{XX}=\begin{pmatrix}4&0\\0&1\end{pmatrix}$, $\Sigma_{YY}=(9)$, and $\Sigma_{XY}=(3,1.2)'$. These blocks are valid: $\Sigma_{XX}$ is positive definite, and $\Sigma_{YY}-\Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XY}=9-3.69=5.31>0$. There is one variable in the $Y$ set, so there is one canonical pair.

For a scalar $Y$, the squared canonical correlation is $\Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XY}/\Sigma_{YY}$. Here,

\Sigma_{XX}^{-1}\Sigma_{XY} =\begin{pmatrix}0.75\\1.2\end{pmatrix}, \qquad \Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XY} =3(0.75)+1.2(1.2)=3.69, \qquad \rho=\sqrt{\frac{3.69}{9}}=\frac{\sqrt{3.69}}{3}\approx0.6403.

A unit-variance score for $Y$ has $b=1/3$. For $X$, take $a=(0.75,1.2)'/\sqrt{3.69}$, approximately $(0.3904,0.6247)'$. Its variance is one because $a'\Sigma_{XX}a=3.69/3.69=1$.

The own-set loading vector for $X$ is

D_X^{-1}\Sigma_{XX}a = \begin{pmatrix}1/2&0\\0&1\end{pmatrix} \begin{pmatrix}4&0\\0&1\end{pmatrix}a \approx \begin{pmatrix}0.7807\\0.6247\end{pmatrix}.

The cross-loading vector for $X$ on $V=Y/3$ is $\Sigma_{XY}b$ divided componentwise by the standard deviations $2$ and $1$:

\operatorname{Corr}(X,V) = \begin{pmatrix}1/2&0\\0&1\end{pmatrix} \begin{pmatrix}3\\1.2\end{pmatrix}\frac{1}{3} = \begin{pmatrix}0.5\\0.4\end{pmatrix}.

The $Y$ own-set loading is $1$, while the cross-loading of $Y$ on $U$ is $\rho\approx0.6403$. The theorem’s first relation is checked by $0.6403(0.7807)\approx0.5$ and $0.6403(0.6247)\approx0.4$. The coefficients and loadings differ, in part because the original variables use different measurement scales.

Weights Are Conditional Ingredients; Loadings Are Marginal Associations

A canonical coefficient weight describes how much a variable contributes to a particular linear combination, given the other variables in that combination. A structure loading instead describes the correlation between one original variable and the resulting score. When variables within a set are correlated, these quantities can differ substantially. A variable can have a modest or even negative weight while having a strong positive structure loading, because the weights combine variables to form a score rather than reporting each variable’s separate association with it.

For example, suppose $\Sigma_{XX}=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}$, $\Sigma_{YY}=(1)$, and $\Sigma_{XY}=(0.5,0)'$. The covariance matrix is positive definite because its Schur complement is $1-(0.5,0)\Sigma_{XX}^{-1}(0.5,0)'=2/3>0$. The canonical correlation is $1/\sqrt{3}$. A unit-variance $X$ score has weights $a=(2/\sqrt{3},-1/\sqrt{3})'$: direct calculation gives $a'\Sigma_{XX}a=1$. Yet its structure loadings are $\Sigma_{XX}a=(\sqrt{3}/2,0)'$. The second variable has a negative weight but zero loading on this score. Its coefficient helps construct a score that removes the part of $X_1$ predictable from the correlated $X_2$; it does not say that $X_2$ is negatively correlated with the score.

Loadings are also affected by the covariance structure and the variable standard deviations, whereas weights depend on the chosen coordinate scales. If a variable is rescaled, a coefficient must change to preserve the same score, while the corresponding correlation loading does not change. The score’s sign is arbitrary: replacing both $a_i$ and $b_i$ by their negatives leaves $\rho_i$ unchanged, but reverses both scores’ own-set and cross-loadings. Thus, signs should be interpreted only after specifying the chosen orientation of each canonical pair.

Worked Example: Checking the Three Quantities From Blocks

Use the covariance blocks from the preceding example: $\Sigma_{XX}=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}$, $\Sigma_{YY}=(1)$, and $\Sigma_{XY}=(0.5,0)'$. The canonical weights are $a=(2/\sqrt{3},-1/\sqrt{3})'$ and $b=1$.

Check the score variances. First, $\Sigma_{XX}a=(\sqrt{3}/2,0)'$, so $a'\Sigma_{XX}a=(2/\sqrt{3})(\sqrt{3}/2)+(-1/\sqrt{3})(0)=1$. Also, $b'\Sigma_{YY}b=1$.

Find the canonical correlation. The score covariance is $a'\Sigma_{XY}b=(2/\sqrt{3})(0.5)+(-1/\sqrt{3})(0)=1/\sqrt{3}$. Both score variances are one, so $\rho=1/\sqrt{3}\approx0.5774$.

Calculate own-set and cross-loadings. Since $X_1$ and $X_2$ each have standard deviation one, their structure loadings on $U$ are $\Sigma_{XX}a=(\sqrt{3}/2,0)'$. Their cross-loadings on $V$ are $\Sigma_{XY}b=(0.5,0)'$. The $Y$ structure loading on $V$ is $1$, and its cross-loading on $U$ is $1/\sqrt{3}$. The relation $(1/\sqrt{3})(\sqrt{3}/2)=0.5$ verifies the cross-loading identity for $X$.

This calculation separates what the negative second weight means from what the loading says: $X_2$ enters the score with a negative coefficient, but its correlation with the score is zero.

Common Mistake and Practical Use

Common mistake: treating a large coefficient as proof that the original variable is strongly associated with the canonical score, or interpreting the sign of a coefficient as the sign of that association. This reasoning is wrong because a coefficient is a component of a multivariable linear combination, while the loading is a correlation that also depends on the within-set covariance matrix and variable scale. Calculate the loadings from the covariance blocks before describing the original variables’ relationships with the scores.

In practice, report the canonical correlations together with clearly labelled weights and loadings. Weights are needed to reproduce the canonical scores; own-set loadings help describe how each original variable relates to its score; cross-loadings describe how each original variable relates to the opposite-set score. The distinctions matter whenever an interpretation is made in the original variables rather than only in terms of the constructed scores.

Key takeaway: Weights construct canonical scores; structure loadings are correlations with scores from the same variable set; cross-loadings are correlations with scores from the opposite set. Compute each from the covariance blocks rather than substituting one quantity for another.

Check Your Understanding

Use the covariance-block formulas and canonical equations to answer these questions.

Which covariance block is used to calculate the structure loadings of $X$ on $U_i$, and which block is used for the cross-loadings of $X$ on $V_i$?
Why can canonical coefficient weights differ from the structure loadings even when the original variables have unit variances?
If both canonical scores have unit variance and $\rho_i=0.6$, how are the $X$-on-$V_i$ cross-loadings related to the $X$-on-$U_i$ structure loadings?
What changes in the reported canonical correlations and loadings if both coefficient vectors in a canonical pair are multiplied by $-1$?
A variable has a negative canonical weight but a positive structure loading. Is this necessarily a contradiction? Explain.