Canonical Scores and Weight Vectors

From Weight Vectors to Sample Scores

Canonical correlation analysis (CCA) describes linear combinations of two data blocks that have high correlation across the same observational units. The weight vectors specify the combinations; the resulting canonical scores are the values of those combinations for each observed unit. Keeping those objects distinct matters: a weight vector has one entry per feature, whereas a score vector has one entry per observation.

Let \(X\) be an \(n\times p\) data matrix and \(Y\) an \(n\times q\) data matrix, with the same \(n\) observations in corresponding rows. Assume each feature column has been centered across observations. Write \(a\in\mathbb{R}^p\) and \(b\in\mathbb{R}^q\) for weight vectors. The two score vectors are \(u=Xa\) and \(v=Yb\), each of length \(n\).

For centered columns, define the sample covariance matrices \(S_{xx}=X^\mathsf{T}X/(n-1)\), \(S_{yy}=Y^\mathsf{T}Y/(n-1)\), and \(S_{xy}=X^\mathsf{T}Y/(n-1)\). The sample variance of \(Xa\) is \(a^\mathsf{T}S_{xx}a\), and the sample covariance between the two score vectors is \(a^\mathsf{T}S_{xy}b\). In classical CCA, weights are commonly scaled so that each score has sample variance one; among those choices, a canonical pair has the largest possible sample correlation.

With unit-variance scores, their sample correlation is simply \(a^\mathsf{T}S_{xy}b\). More generally, for nonzero score variances it is that covariance divided by the square root of the product of the two variances. These identities follow by substituting \(u=Xa\) and \(v=Yb\) into the usual sample covariance formulas. They are useful both for understanding the optimization and for checking computed scores.

Scaling and Interpretation

A standard classical CCA convention imposes \(a^\mathsf{T}S_{xx}a=1\) and \(b^\mathsf{T}S_{yy}b=1\). Thus the scores, not necessarily the weights, have sample variance one. The entries of the weights need not sum to one, have unit Euclidean length, or be directly comparable in magnitude across differently scaled features. Their numerical values depend on the units and preprocessing of the features.

If a set of nonzero weights is rescaled by a positive constant, its scores are rescaled by the same constant. The correlation is unchanged, although the score variance changes. For example, to rescale a weight vector \(a\) to give a unit-variance score, divide it by \(\sqrt{a^\mathsf{T}S_{xx}a}\), provided that this quantity is positive. This is covariance-based normalization. It is not in general the same as dividing \(a\) by its Euclidean length \(\sqrt{a^\mathsf{T}a}\).

Feature scaling also changes the coordinates used for the weights. If a feature is measured in different units, its corresponding weight can change when the data are transformed, even when the resulting score is the same after the weights are transformed consistently. Therefore, report which preprocessing and score-normalization convention was used. A score is a computed linear-combination value, not a feature coefficient; neither alone establishes a causal relationship between the blocks.

Why This Normalization Works

The covariance identities explain the connection between CCA's constraints and its scores. For centered \(u=Xa\), the sample variance is \(u^\mathsf{T}u/(n-1)\). Substituting \(u=Xa\) gives \(a^\mathsf{T}X^\mathsf{T}Xa/(n-1)=a^\mathsf{T}S_{xx}a\); the corresponding calculation for \(v\) gives \(b^\mathsf{T}S_{yy}b\). Likewise, \(u^\mathsf{T}v/(n-1)=a^\mathsf{T}S_{xy}b\). Dividing this covariance by the two score standard deviations gives the sample correlation.

Thus, when both variances equal one, maximizing the score correlation is equivalent to maximizing \(a^\mathsf{T}S_{xy}b\) subject to the two variance constraints. The normalization makes the comparison of correlations well-defined: without constraints, multiplying either weight vector by an arbitrarily large constant would change the covariance without changing the underlying correlation. Classical CCA assumes the paired rows are correctly aligned and the within-block covariance matrices support the required normalization. When a covariance matrix is singular, some nonzero directions can have zero variance, so this classical normalization may fail; that case requires separate treatment.

Sign Ambiguity and Equivalent Representations

A canonical pair does not have a uniquely determined sign. Replacing \(a\) and \(b\) simultaneously by \(-a\) and \(-b\) changes \(u,v\) to \(-u,-v\), but leaves both variances and their correlation unchanged. If only one weight vector is negated, the correlation changes sign. Since the two directions can be oriented after fitting, software may choose different signs while representing the same canonical relationship. A common reporting convention is to orient the pair so that its score correlation is nonnegative.

Checks, Alternatives, and Common Pitfalls

After obtaining weights and scores, verify the dimensions, centering, and normalization rather than relying on a displayed output alone. For \(n\) paired observations, both score vectors must have length \(n\). With the stated sample-covariance convention, check that \(u^\mathsf{T}u/(n-1)\) and \(v^\mathsf{T}v/(n-1)\) equal one if unit-variance normalization is claimed. Then compute \(u^\mathsf{T}v/(n-1)\) and compare it with the reported correlation. Small discrepancies can arise from rounding; large discrepancies often indicate inconsistent scaling, row order, or preprocessing.

In implementations based on QR factorization or singular value decomposition, the reported vectors may be expressed in transformed coordinates. The weights used for scores in the original feature columns must be mapped back to those coordinates before computing \(Xa\) or \(Yb\). When features are centered or standardized, apply the same fitted transformations to the data used to produce the scores. Misaligned rows or using one set of preprocessing constants for the weights and another for the scores can produce plausible-looking numbers that are not the intended canonical scores.

It is also important not to confuse weight-based scores with other summaries. A principal-component score summarizes variation within one block; a CCA score is formed to participate in a paired, cross-block correlation. Nor should a score be mistaken for a weight, or for a feature correlation with a score. Those are distinct quantities with different interpretations. The subsequent study of loadings and cross-loadings addresses such feature-to-score relationships; they are not needed to calculate \(Xa\) and \(Yb\).

• Wrong: “The weights have unit variance because the score variance is one.” Correction: The unit-variance constraint is \(a^\mathsf{T}S_{xx}a=1\) for \(Xa\); it does not generally imply \(a^\mathsf{T}a=1\).
• Wrong: “The software returned a different sign, so it found a different solution.” Correction: A simultaneous sign reversal gives the same canonical pair; compare aligned scores or absolute correlation.
• Wrong: “The \(j\)th entry of \(a\) is the score for observation \(j\).” Correction: The \(j\)th weight refers to a feature. The \(i\)th score is the dot product of observation \(i\)'s feature row with the full weight vector.

The earlier material on QR and SVD provides computational routes for obtaining CCA directions. Regardless of the route, the final interpretation depends on the same direct check: weights combine features, and their multiplication by the paired data matrices produces one score per observation. The main limitations here are those of the fitted CCA itself, including unstable directions when the data do not adequately support the covariance calculations. Correctly computing scores does not remove that instability or make an association causal.