Deriving CCA Generalized Eigenvalue Equations

From the CCA Objective to an Eigenvalue Problem

Canonical correlation analysis (CCA) looks for linear combinations of two data blocks whose scores have the largest correlation. The constrained optimization formulation expresses this by choosing weight vectors with unit within-block variance and maximizing their cross-covariance. This tutorial derives the generalized eigenvalue equations that follow from that formulation and explains why their eigenvalues are squared canonical correlations.

Let \(X\) be a centered \(n\times p\) data matrix for one block and \(Y\) a centered \(n\times q\) data matrix for the other block. The rows correspond to the same \(n\) observations. Define the empirical covariance matrices using divisor \(n-1\):

Thus \(S_{xx}\) is \(p\times p\), \(S_{yy}\) is \(q\times q\), and \(S_{xy}\) is \(p\times q\). For weight vectors \(a\in\mathbb{R}^p\) and \(b\in\mathbb{R}^q\), the scores are \(Xa\) and \(Yb\). Their covariance is \(a^\top S_{xy}b\), and their variances are \(a^\top S_{xx}a\) and \(b^\top S_{yy}b\). The first CCA pair solves

The variance constraints make the objective the correlation of the two scores. The derivation below assumes \(S_{xx}\) and \(S_{yy}\) are positive definite, so both are invertible and every nonzero weight vector has positive within-block variance. This is the classical nonsingular setting; the equations need modification or special handling when a covariance matrix is singular.

Deriving Both Generalized Eigenvalue Equations

Introduce Lagrange multipliers \(\eta_x\) and \(\eta_y\) for the two variance constraints. With the objective written as a maximization, one convenient Lagrangian is

Setting the gradients with respect to \(a\) and \(b\) equal to zero gives the stationarity equations. Since the covariance matrices within each block are symmetric, differentiating each quadratic form gives:

Multiply the first equation on the left by \(a^\top\). Because \(a^\top S_{xx}a=1\), this gives \(a^\top S_{xy}b=\eta_x\). Multiplying the second equation by \(b^\top\), and using \(b^\top S_{yy}b=1\), gives \(b^\top S_{yx}a=\eta_y\). The two left sides are the same scalar because \(S_{yx}=S_{xy}^\top\). Therefore \(\eta_x=\eta_y\), and both equal the objective value. Denote this value by \(\rho\), the correlation of the normalized scores. The stationary equations become

For a nonzero \(\rho\), solve the second equation for \(b\), then substitute into the first. Since \(S_{yy}\) is invertible, \(b=\rho^{-1}S_{yy}^{-1}S_{yx}a\), so

This is the \(p\)-dimensional generalized eigenvalue problem. Eliminating \(a\) instead gives the \(q\)-dimensional counterpart:

If a canonical correlation is zero, the corresponding stationarity equations still apply; elimination by division by \(\rho\) is not available for that pair. The generalized equations themselves describe the zero eigenvalues as well.

Why the Eigenvalues Are Squared Correlations

The square appears because eliminating one weight vector substitutes an equation containing \(\rho\) into another equation that also contains \(\rho\). For example, substituting \(b=\rho^{-1}S_{yy}^{-1}S_{yx}a\) into \(S_{xy}b=\rho S_{xx}a\) yields a factor \(\rho^{-1}\) on the left and \(\rho\) on the right; multiplying through by \(\rho\) produces \(\rho^2\).

The matrix \(S_{xy}S_{yy}^{-1}S_{yx}\) is symmetric and positive semidefinite: for any \(a\), its quadratic form is \((S_{yx}a)^\top S_{yy}^{-1}(S_{yx}a)\), which is nonnegative because \(S_{yy}\) is positive definite. The generalized eigenvalues are therefore nonnegative. They are also at most 1 for a valid joint covariance matrix: every pair of score vectors has correlation with magnitude at most 1, and CCA maximizes that correlation under unit-variance constraints. Consequently \(0\leq\lambda\leq 1\), and \(\rho=\sqrt{\lambda}\) for the nonnegative canonical correlation.

A generalized eigenvector determines a direction of weights, not their absolute scale: multiplying \(a\) by a nonzero constant does not change its direction. After obtaining an eigenvector, it is normalized so that \(a^\top S_{xx}a=1\), and the associated \(b\) is normalized to satisfy \(b^\top S_{yy}b=1\). Changing the sign of one vector changes the sign of the score correlation but not its magnitude or squared value.

Assumptions, Diagnostics, and Alternatives

The derivation uses paired rows, centered data, the stated empirical covariance definitions, and positive-definite \(S_{xx}\) and \(S_{yy}\). Centering ensures the score covariance and variances have the usual covariance interpretation. Positive definiteness ensures the variance constraints define valid nonzero directions and permits the matrix inverses used in elimination. These are algebraic requirements for this classical derivation, not a claim that the data must follow a particular probability distribution.

When computing the equations, useful checks include:

• Confirm that \(S_{xy}\) has dimensions \(p\times q\), so the \(X\)-side product is \(p\times p\) and the \(Y\)-side product is \(q\times q\).
• Check that the within-block covariance matrices are symmetric and positive definite to the accuracy expected from the data and computation.
• Check that the resulting eigenvalues are nonnegative and no greater than 1, up to small numerical error, and that recomputed score correlations agree with their square roots.
• Check the original constraints after normalizing weights: \(a^\top S_{xx}a=1\) and \(b^\top S_{yy}b=1\).

A materially negative eigenvalue, or one materially above 1, signals a calculation or data problem, such as an incorrect covariance block, inconsistent pairing, or severe numerical error. Tiny departures can arise from rounding. If a covariance matrix is singular or nearly singular, its inverse may not exist or may amplify small perturbations. That is a limitation of the classical equations; it is not resolved by simply ignoring the problem or interpreting unstable weights as reliable. Repeated eigenvalues can also make individual weight directions nonunique even when the corresponding canonical correlations are well determined.

The constrained optimization statement is the direct conceptual definition: it makes clear which score correlation is being maximized and under what normalization. The generalized eigenvalue formulation converts that optimization into a matrix eigenproblem that can be solved using linear algebra routines. They describe the same classical solution under the assumptions above. The eigenvalue formulation is especially useful for identifying canonical correlations, while the original constrained objective remains a useful check on computed weights and scores.

Common Mistake

A frequent error is to report an eigenvalue as the canonical correlation. For example, if an eigenvalue is \(0.64\), the canonical correlation is \(\sqrt{0.64}=0.80\), not \(0.64\). The eigenvalue is the squared correlation because elimination produces \(\rho^2\). Another error is to invert a within-block covariance matrix without checking that it is nonsingular; the classical derivation requires that condition.