Invariance Under Separate Linear Reparameterizations

Why Coordinate Changes Matter

A multivariate measurement can be recorded in different units or represented by different linear combinations of its variables. For example, a laboratory might report concentrations in different units, or replace two original measurements with their sum and difference. These choices change the coordinates used to describe a variable set, but should not create or remove association with a second set when the change is invertible. Canonical correlation analysis makes this principle precise: its canonical correlations are unchanged, while the coefficient vectors must be adjusted to describe the same scores in the new coordinates.

The key distinction is between changing coordinates and discarding information. A nonsingular transformation is reversible, so it changes the representation but not the information in either variable set. A singular transformation can collapse distinct directions together or remove directions altogether; it need not preserve all canonical correlations.

Setup and Transformation Rule

Let \(X\in\mathbb{R}^p\) and \(Y\in\mathbb{R}^q\) be random vectors with finite second moments. Write their covariance blocks as \(\Sigma_{XX}=\operatorname{Cov}(X,X)\), \(\Sigma_{YY}=\operatorname{Cov}(Y,Y)\), and \(\Sigma_{XY}=\operatorname{Cov}(X,Y)\). Assume \(\Sigma_{XX}\) and \(\Sigma_{YY}\) are positive definite. This ensures every nonzero linear combination within either set has positive variance and that the usual canonical correlations are well-defined.

A pair of coefficient vectors \(a\in\mathbb{R}^p\) and \(b\in\mathbb{R}^q\) defines centered canonical scores \(a^T(X-E[X])\) and \(b^T(Y-E[Y])\). Now choose nonsingular matrices \(A\in\mathbb{R}^{p\times p}\) and \(B\in\mathbb{R}^{q\times q}\), and record the transformed vectors \(X^*=AX\) and \(Y^*=BY\). Their covariance blocks are

If the original score uses coefficient \(a\), its coefficient in the transformed coordinates is not generally \(Aa\). It is \(A^{-T}a\), where \(A^{-T}=(A^{-1})^T\). Indeed,

Similarly, \(b\) becomes \(B^{-T}b\). This inverse-transpose rule is the contragredient transformation of coefficient vectors: the coordinates change one way, while the linear functional that extracts a score changes in the corresponding inverse way.

Invariance of the Canonical Correlations

The canonical correlations are the ordered maximum correlations between linear combinations of the two sets, with successive pairs subject to the standard within-set uncorrelatedness constraints. Under a nonsingular transformation, every transformed coefficient pair corresponds to exactly one original pair: set \(a=A^T a^*\) and \(b=B^T b^*\). The score identity above shows that this correspondence preserves the scores themselves, not merely their correlations.

Proof. For any coefficient pair \((a^*,b^*)\) in the transformed coordinates, let \(a=A^T a^*\) and \(b=B^T b^*\). Since \(A\) and \(B\) are nonsingular, this is a bijection between transformed and original coefficient pairs. The centered scores satisfy

Consequently, the two scores have the same variances and covariance under the correspondence, and hence the same correlation whenever both coefficients are nonzero. In particular, the set of correlations attainable by nonzero linear combinations is unchanged. The first canonical correlation, the maximum over this set, is therefore unchanged.

For each later canonical pair, the admissible coefficient pairs are additionally constrained so that their scores are uncorrelated with the earlier canonical scores within their respective sets. The earlier scores are preserved exactly by the same correspondence. Thus a candidate score is uncorrelated with an earlier score before transformation if and only if it is uncorrelated with that score afterward: each covariance in the constraint is the same covariance of the same random variables. At every successive maximization, the feasible sets and the objective correlations are therefore identical under the bijection. This proves equality of the ordered canonical correlations and the stated coefficient transformation. \(\square\)

Whitening Gives a Matrix View

The same invariance can be seen through the whitened cross-covariance matrix. With positive-definite within-set covariance matrices, define

The canonical correlations are the singular values of \(C\), including zero values up to \(\min(p,q)\). For the transformed variables, define \(C^*=\Sigma_{X^*X^*}^{-1/2}\Sigma_{X^*Y^*}\Sigma_{Y^*Y^*}^{-1/2}\). Set

Both matrices are orthogonal. For example,

using \(\Sigma_{X^*X^*}=A\Sigma_{XX}A^T\); the corresponding calculation gives \(Q_YQ_Y^T=I_q\). Substitution of the transformed covariance blocks now yields

Multiplication on the left and right by orthogonal matrices does not change singular values. To see this, \((C^*)^T C^*=Q_Y C^T C Q_Y^T\), which is an orthogonal similarity of \(C^T C\), so the two matrices have the same eigenvalues. Their nonnegative square roots, the singular values, are therefore equal. This proves the invariance again from the whitened representation.

Common Pitfalls and Practical Meaning

Coordinate invariance is useful when variables have different units or when an analyst chooses a numerically convenient set of linear combinations. Rescaling, rotating, or applying an invertible shear within either set may change the numerical entries of covariance matrices and the displayed coefficients. Those changes do not alter the canonical correlations. The coefficient vectors are coordinate-dependent descriptions of scores; the scores and their correlations can be represented unchanged.

The claim has limits. The transformation must be nonsingular and act separately within each set. A singular map can discard a direction, and a transformation that combines variables across the two sets changes which quantities are being associated. Neither case is covered by the invariance theorem.

Common mistake: “The variables have been rescaled or mixed, so the canonical correlations must change because the covariance matrices changed.” The correction is to compare the full optimization problem, not individual covariance entries. Under nonsingular separate transformations, coefficient pairs correspond bijectively and produce exactly the same centered scores. A related error is to apply \(A\) to the coefficient vector; the correct rule is \(A^{-T}\), as required by the score identity.