Interpreting Canonical Results After Coordinate Changes

Why Coordinate Changes Matter in CCA

Canonical correlation analysis (CCA) summarizes association between two sets of variables using paired linear combinations. A change of measurement units or a nonsingular recoding can leave the underlying paired scores and canonical correlations unchanged, even while the reported coefficient values change substantially. Loadings and redundancy summaries require separate attention: they describe relationships with the particular variables being reported, so changing those variables can change their interpretation.

The key is to distinguish a change in coordinates from a change in the statistical relationship. We will follow each object through the recoding: canonical scores, weights, structure loadings, cross-loadings, and redundancy.

Definition: Let \(X\) and \(Y\) be centered random vectors of dimensions \(p\) and \(q\), with finite second moments and positive-definite covariance matrices \(\Sigma_{XX}\) and \(\Sigma_{YY}\). A canonical pair is selected by the CCA optimization: the first pair of linear scores \(U_1=a_1^\mathsf{T}X\) and \(V_1=b_1^\mathsf{T}Y\), each scaled to have variance one, maximizes their correlation, with orientations chosen so the correlation is nonnegative. Each subsequent pair maximizes correlation subject to unit score variances and the constraints \(\operatorname{Cov}(U_j,U_k)=\operatorname{Cov}(V_j,V_k)=0\) for \(k<j\). The resulting correlations are the canonical correlations.

This definition matters: two arbitrary unit-variance linear combinations are not necessarily a canonical pair. The weights are the vectors \(a_j\) and \(b_j\); the scores are the random variables they produce. A weight vector describes how a score is formed in the stated coordinates, not an association measure that is automatically comparable across different codings.

How Scores and Weights Transform

Consider separate nonsingular linear recodings \(X^*=AX\) and \(Y^*=BY\), where \(A\) and \(B\) are invertible matrices. Translations can be added without affecting centered covariance analysis. If the original score is \(U=a^\mathsf{T}X\), then the same score in the new coordinates is obtained with the weight \(a^*=A^{-\mathsf{T}}a\), because

(a^*)^\mathsf{T}X^*=(A^{-\mathsf{T}}a)^\mathsf{T}AX =a^\mathsf{T}A^{-1}AX=a^\mathsf{T}X.

The corresponding formula for \(V=b^\mathsf{T}Y\) is \(b^*=B^{-\mathsf{T}}b\). Thus weights transform in the opposite direction to the variables: they compensate for the recoding so that the score stays fixed. In particular, changing a measurement unit generally changes the numerical weight even when it changes nothing about the score.

Worked Example: Converting Units Changes Weights, Not Scores

Suppose \(X=(X_1,X_2)^\mathsf{T}\) has covariance matrix \(\operatorname{diag}(4,9)\), and a canonical score in the original coordinates is \(U=\tfrac12X_1\). Its variance is \((\tfrac12)^2(4)=1\). Record the first variable in units 1,000 times as large and the second in units one hundredth as large:

X^*= \begin{pmatrix}1000&0\\0&0.01\end{pmatrix}X.

Here \(A^{-\mathsf{T}}=\operatorname{diag}(0.001,100)\). The transformed weight is therefore \(a^*=(0.0005,0)^\mathsf{T}\), and the score is

(a^*)^\mathsf{T}X^*=0.0005(1000X_1)=0.5X_1=U.

The displayed coefficient has changed from \(0.5\) to \(0.0005\), but the score itself has not changed at all. Comparing these two coefficient values without accounting for the units would be misleading.

Theorem: Under nonsingular separate linear recodings \(X^*=AX\) and \(Y^*=BY\), the canonical correlations are unchanged. For any canonical pair whose scores are represented in the original coordinates by \(a^\mathsf{T}X\) and \(b^\mathsf{T}Y\), the transformed weights \(A^{-\mathsf{T}}a\) and \(B^{-\mathsf{T}}b\) represent exactly the same scores.

Proof. For any pair of weight vectors \(a,b\), the recoded weights \(a^*=A^{-\mathsf{T}}a\) and \(b^*=B^{-\mathsf{T}}b\) give the same two random variables. Their variances, covariance, correlation, and covariances with any earlier scores are therefore unchanged. Since \(A\) and \(B\) are invertible, this correspondence is one-to-one: every pair of recoded weights corresponds to original weights \(a=A^\mathsf{T}a^*\) and \(b=B^\mathsf{T}b^*\). The feasible pairs in each sequential CCA optimization correspond exactly, with the same objective value. Their successive maxima, and hence the canonical correlations, are equal. \(\square\)

This invariance concerns the canonical correlations and the attainable scores, not a unique printed coefficient vector in every circumstance. A canonical pair can be reversed in sign by reversing both score orientations. Moreover, if a canonical correlation is tied with another, the corresponding canonical directions can be represented by different bases for the same tied subspace. Such choices do not contradict invariance; they mean that comparisons of individual directions need a consistent orientation and, in a tied subspace, a consistent basis.

Loadings Describe the Chosen Coordinates

For unit-variance scores, a structure loading is a correlation between an original variable and a canonical score. A cross-loading is a correlation between a variable in one set and the canonical score from the other set. Let \(D_X\) be the diagonal matrix of standard deviations of the components of \(X\). For \(U=a^\mathsf{T}X\) with \(\operatorname{Var}(U)=1\), the vector of structure loadings is

\operatorname{Corr}(X,U)=D_X^{-1}\Sigma_{XX}a.

The vector of cross-loadings with \(V=b^\mathsf{T}Y\), also of variance one, is \(D_X^{-1}\Sigma_{XY}b\). The same formulas apply to the \(Y\) variables with the corresponding covariance matrices. After recoding \(X^*=AX\), the score may remain exactly the same, but its loadings with the recoded variables are correlations with \(AX\), not with \(X\). For example, the unstandardized covariance vector with the score becomes \(A\operatorname{Cov}(X,U)\); standardizing each new variable then divides by its new standard deviation. General mixing of variables therefore does not simply preserve the list of individual correlations.

Worked Example: Mixing Variables Changes Loadings

Suppose \(\operatorname{Var}(X)=I_2\), \(\operatorname{Var}(Y)=I_2\), and \(\operatorname{Cov}(X,Y)=\operatorname{diag}(0.8,0.3)\). The leading canonical pair is \(U=X_1\), \(V=Y_1\), with correlation \(0.8\). The \(X\)-structure loadings for \(U\) are \((1,0)^\mathsf{T}\).

Now recode \(X\) by \(X_1^*=X_1+X_2\) and \(X_2^*=X_2\). The recoding matrix is \(A=\begin{pmatrix}1&1\\0&1\end{pmatrix}\). Its inverse transpose maps the old leading weight \(e_1\) to \(a^*=(1,-1)^\mathsf{T}\), so \(U=X_1^*-X_2^*=X_1\), exactly as before. But \(\operatorname{Var}(X_1^*)=2\) and \(\operatorname{Var}(X_2^*)=1\). Consequently,

\operatorname{Corr}(X_1^*,U)=\frac{\operatorname{Cov}(X_1+X_2,X_1)}{\sqrt{2}\cdot1} =\frac{1}{\sqrt{2}}, \qquad \operatorname{Corr}(X_2^*,U)=0.

The canonical score and its correlation with \(V\) remain unchanged, while the structure loading on the first reported variable changes from \(1\) to \(1/\sqrt{2}\). The new variable is a mixture of two old coordinates; its loading answers a different descriptive question.

Redundancy: Invariant Under Unit Changes, Not Every Recoding

A common redundancy summary for the \(X\) set and the \(j\)th canonical pair is the average squared structure loading of the \(X\) variables on \(U_j\), multiplied by the squared canonical correlation. With \(p\) variables, define the variance-extracted average and the directional redundancy as

E_{X,j}=\frac{1}{p}\sum_{i=1}^{p}\operatorname{Corr}(X_i,U_j)^2, \qquad R_{X\leftarrow Y,j}=E_{X,j}\rho_j^2.

Here \(\rho_j=\operatorname{Corr}(U_j,V_j)\). The redundancy is a descriptive summary of how much variance in the specified \(X\) variables is associated with the paired canonical score from \(Y\), using this convention. It is not the same thing as the canonical correlation itself.

If each variable is rescaled by a nonzero constant, its correlation with a fixed score is unchanged or reversed in sign, so its squared loading is unchanged. Thus these averages and redundancies are invariant to separate changes of units and sign. Under a general nonsingular mixing, however, the variables in the average have changed. The average of their squared correlations need not be preserved.

Worked Example: A Redundancy Summary Can Change Under Mixing

Continue with the preceding model, where \(\rho_1=0.8\). Before recoding, the squared \(X\)-loadings on \(U_1=X_1\) are \(1\) and \(0\). Therefore,

E_{X,1}=\frac{1+0}{2}=0.5, \qquad R_{X\leftarrow Y,1}=0.5(0.8)^2=0.5(0.64)=0.32.

After the mixing recoding, the squared loadings are \(1/2\) and \(0\). The new average and redundancy are

E_{X^*,1}=\frac{1/2+0}{2}=0.25, \qquad R_{X^*\leftarrow Y,1}=0.25(0.64)=0.16.

The canonical correlation has not changed, but the redundancy summary has halved because the set of variables being summarized changed. Neither value is an arithmetic error: each describes redundancy relative to its own coordinates.

Sign Orientation and Interpretation

Canonical scores have an orientation convention, not an intrinsic positive direction. If both scores in a pair are multiplied by \(-1\), their correlation stays positive and the paired canonical result is unchanged. Their within-set structure loadings reverse signs, while squared loadings and redundancy summaries stay the same. If only one score is reversed, the signed correlation reverses; the usual convention is to orient the pair so that the canonical correlation is nonnegative.

This distinction is useful when comparing output from software or from analyses using different coding choices. Compare the canonical correlations first. Then verify that the score definitions correspond, allowing for sign reversals or, for tied roots, alternative bases. Interpret weights in the stated units and coding, and interpret loadings as correlations with the particular variables listed. Redundancy summaries should be compared only when their variable sets and averaging conventions match.

Common pitfall: It is incorrect to conclude that a canonical direction changed substantively because a weight became numerically smaller after converting units. The weight is a coordinate-dependent coefficient: under \(X^*=AX\), it changes to \(A^{-\mathsf{T}}a\). Check the resulting score. If it is the same random variable, the difference is representational. Conversely, it is also incorrect to assume that every summary remains unchanged because the canonical correlations do. Loadings and redundancy refer to the chosen variables and can change under mixing.

Identify the recoding.
Write the new variables as \(X^*=AX\) and \(Y^*=BY\), and check that the matrices are nonsingular.

Track the scores.
Use \(a^*=A^{-\mathsf{T}}a\) and \(b^*=B^{-\mathsf{T}}b\) to determine whether the canonical scores are preserved.

Recompute descriptive quantities.
Calculate loadings and redundancy using the variables and standard deviations in the new coordinates; do not infer their values from canonical-correlation invariance alone.

What Is Invariant, and What Is Not?

Quantity	Effect of nonsingular recoding	Interpretation
Canonical correlation	Invariant	The sequential optimization has the same attainable scores and correlations.
Canonical score	Can be represented identically with transformed weights	Weights compensate for the new coordinates.
Weight values	Generally change	They depend on units and coding.
Structure or cross-loadings	Generally change under mixing	They describe correlations with the listed variables.
Redundancy	Unchanged by nonzero separate rescalings; generally changed by mixing	It averages squared loadings for a specified variable set.

The practical lesson is not to treat one kind of output as a substitute for another. Canonical correlations summarize optimized association; weights define scores in coordinates; loadings describe how named variables relate to those scores; redundancy aggregates squared loadings over a specified set. Their different transformation behavior is part of their meaning.

Check Your Understanding

Use the transformation rules to explain what should be compared across two CCA reports.

If \(X^*=5X\), what weight represents the same score as \(a^\mathsf{T}X\)? Show the score equality.
Why are two arbitrary unit-variance linear combinations not necessarily a canonical pair?
Under a nonsingular recoding of both sets, which feature of the CCA optimization establishes that the canonical correlations are unchanged?
Why can a structure loading change when the canonical score itself does not?
State a condition under which a redundancy summary is unchanged, and give a type of recoding under which it may change.
What sign change occurs in within-set loadings if both scores in a pair are reversed? What happens to squared loadings?