Deriving the Generalized Eigenvalue Equations

From a Constrained Objective to an Eigenvalue Equation

Canonical correlation compares two vectors through linear scores. Its weights are constrained: without fixing the score variances, the weights could be rescaled indefinitely, even though rescaling by a positive constant leaves correlation unchanged. A negative rescaling reverses the correlation's sign, and a zero rescaling makes it undefined. Unit-variance constraints fix the scale; the signs can then be chosen so that the covariance is nonnegative.

Let \(X\) and \(Y\) be centered random vectors of dimensions \(p\) and \(q\), with finite second moments. Write their covariance matrix in blocks

Assume \(\Sigma_{XX}\) and \(\Sigma_{YY}\) are positive definite. This ensures that every nonzero weight vector gives a score with positive variance and that the inverse covariance matrices used below exist. For weights \(a\in\mathbb{R}^p\) and \(b\in\mathbb{R}^q\), the scores are \(a'X\) and \(b'Y\).

Thus the constrained problem is to maximize \(a'\Sigma_{XY}b\) subject to the two unit-variance constraints. The following derivation turns its stationary conditions into a pair of generalized eigenvalue equations.

The Lagrange Multiplier Equations

Set \(C=\Sigma_{XY}\). Use one multiplier for each variance constraint:

The covariance blocks \(\Sigma_{XX}\) and \(\Sigma_{YY}\) are symmetric. Differentiating with respect to \(a\) and \(b\), respectively, gives the stationary conditions

At a feasible stationary point, multiply the first equation on the left by \(a'\) and the second by \(b'\). Since \(a'\Sigma_{XX}a=b'\Sigma_{YY}b=1\), both multipliers equal the attained covariance:

Write this common value as \(\rho\). At the maximizing solution it is the nonnegative canonical correlation. The stationary equations therefore become

For \(\rho>0\), the second equation gives \(b=\rho\Sigma_{YY}^{-1}C'a\). Substituting this expression into the first yields one equation involving \(a\) alone. Eliminating \(a\) instead gives the corresponding equation for \(b\):

Why the Leading Eigenvalue Gives the Maximum

The equations alone identify stationary pairs. To establish which eigenvalue corresponds to the first canonical correlation, it helps to compare the constrained objective with a generalized Rayleigh quotient. Define

For a fixed nonzero \(a\), maximizing \(a'Cb\) over \(b\) with \(b'\Sigma_{YY}b=1\) gives

To verify this, put \(u=\Sigma_{YY}^{1/2}b\), so \(u'u=1\). Then \(a'Cb=(\Sigma_{YY}^{-1/2}C'a)'u\). The largest inner product with a unit vector is the Euclidean norm of the other vector, which is \(\sqrt{a'C\Sigma_{YY}^{-1}C'a}\). If this vector is zero, the maximum is zero and the equality still holds.

Consequently, the largest attainable covariance is the square root of the maximum of \(a'Aa\) subject to \(a'\Sigma_{XX}a=1\). This maximum is the largest generalized eigenvalue of \(Av=\eta\Sigma_{XX}v\). Here is a direct justification: factor the positive definite matrix \(\Sigma_{XX}=R'R\), and set \(z=Ra\). Then the constraint is \(z'z=1\), and

The matrix \(R^{-T}AR^{-1}\) is symmetric and positive semidefinite: for every \(v\), \(v'Av=(C'v)'\Sigma_{YY}^{-1}(C'v)\geq 0\). By the spectral theorem, the maximum of its quadratic form over unit vectors is its largest eigenvalue, attained at a corresponding eigenvector. Its eigenvalue equation transforms back to \(Av=\eta\Sigma_{XX}v\). This proves that the constrained maximum of \(a'Aa\) is the largest generalized eigenvalue. Since the maximizing covariance is its square root, the first canonical correlation is the nonnegative square root of that eigenvalue. \(\square\)

The same reasoning shows that these generalized eigenvalues are real and nonnegative. They are also at most one: for any \(a,b\), the covariance inequality gives \((a'Cb)^2\leq(a'\Sigma_{XX}a)(b'\Sigma_{YY}b)\). Taking \(b=\Sigma_{YY}^{-1}C'a\), and writing \(t=a'Aa=b'\Sigma_{YY}b=a'Cb\), yields \(t^2\leq(a'\Sigma_{XX}a)t\). If \(t>0\), division by \(t\) gives \(t\leq a'\Sigma_{XX}a\); the same bound holds when \(t=0\). Thus every generalized Rayleigh quotient is at most one.

The argument also clarifies a zero-root case. If \(C=0\), all cross-covariances vanish, \(A=0\), and the maximum canonical correlation and all the generalized eigenvalues are zero. The elimination using division by \(\rho\) is only needed for positive roots; the stationary equations themselves remain valid when \(\rho=0\).

Worked Examples

Common Mistakes and Why the Equations Matter

The generalized eigenvalue form is useful because it replaces optimization over two weight vectors with a matrix eigenvalue problem. It also makes explicit that the squared association is determined by both cross-covariance and within-set covariance: large cross-covariances alone do not determine canonical correlation if the score variances are also large.

• Claiming arbitrary rescaling leaves correlation unchanged. Positive rescaling leaves its sign and value unchanged; negative rescaling reverses the sign, and zero makes the score variance zero. The unit-variance constraints fix scale, while a sign choice makes the covariance nonnegative.
• Using one multiplier without checking it. The two constraints introduce two multipliers. Multiplying each stationary equation by its corresponding weight vector shows that both equal the attained covariance at a feasible stationary point.
• Forgetting the square. The eigenvalue is \(\rho^2\), not \(\rho\). A generalized eigenvalue of \(0.36\) corresponds to canonical correlation \(0.6\).
• Dividing by a zero canonical correlation. The elimination step that substitutes \(b=\rho\Sigma_{YY}^{-1}C'a\) requires \(\rho>0\). Treat the zero case using the original stationary equations.
• Dropping the covariance metric. In general the equation is \(C\Sigma_{YY}^{-1}C'a=\rho^2\Sigma_{XX}a\), not an ordinary eigenvalue equation with \(\Sigma_{XX}\) omitted.