The Variational Definition of Canonical Correlation

From Two Variable Sets to a Variational Definition

Suppose a study records two related collections of measurements on each subject. Canonical correlation analysis asks whether a linear combination of the first collection can be strongly associated with a linear combination of the second. The central idea is an optimization: among all such pairs of combinations, select the pair whose correlation is largest, then repeat subject to constraints that distinguish later pairs from earlier ones.

Let \(X\) be a \(p\)-dimensional random vector and \(Y\) a \(q\)-dimensional random vector, with finite second moments. Write their covariance partition as \[ \operatorname{Cov}\begin{pmatrix}X\\Y\end{pmatrix} = \begin{pmatrix} \Sigma_{XX} & \Sigma_{XY}\\ \Sigma_{YX} & \Sigma_{YY} \end{pmatrix}, \] where \(\Sigma_{XY}=\operatorname{Cov}(X,Y)\) and \(\Sigma_{YX}=\Sigma_{XY}^{\top}\). For coefficient vectors \(a\in\mathbb R^p\) and \(b\in\mathbb R^q\), the corresponding scalar scores are \(a^\top X\) and \(b^\top Y\). Their covariance is \(a^\top\Sigma_{XY}b\).

Definition: Assume \(\Sigma_{XX}\) and \(\Sigma_{YY}\) are positive definite. The first canonical correlation is the maximum correlation between \(a^\top X\) and \(b^\top Y\), over coefficient vectors satisfying \[ a^\top\Sigma_{XX}a=1,\qquad b^\top\Sigma_{YY}b=1. \] Equivalently, it is the maximum of \(a^\top\Sigma_{XY}b\) subject to these constraints. A maximizing pair of coefficient vectors is a first canonical pair, and its scores are the first canonical variates.

The constraints say that each score has variance one: \(\operatorname{Var}(a^\top X)=a^\top\Sigma_{XX}a\), and likewise for \(b^\top Y\). With both variances equal to one, their covariance equals their correlation. Maximizing a signed covariance is appropriate because replacing \(b\) by \(-b\) reverses the sign of the covariance without changing either variance. Thus the maximum is nonnegative and agrees with the largest attainable correlation.

Why Normalize the Scores?

Without a constraint on scale, maximizing covariance would not define a meaningful problem. If \(a^\top\Sigma_{XY}b>0\), replacing \(a\) by \(c a\) for \(c>0\) multiplies the covariance by \(c\), with no finite upper bound as \(c\) increases. Correlation, in contrast, does not change when either score is multiplied by a positive constant. Unit-variance constraints fix that arbitrary scale while preserving the substantive goal: finding the most strongly associated linear combinations.

If both scores have positive variance, their correlation can be written as \[ \operatorname{Corr}(a^\top X,b^\top Y) = \frac{a^\top\Sigma_{XY}b} {\sqrt{(a^\top\Sigma_{XX}a)(b^\top\Sigma_{YY}b)}}. \] Rescaling \(a\) and \(b\) to make both variances one gives exactly the constrained definition. Positive definiteness ensures that every nonzero coefficient vector has positive score variance, so these normalizations are available.

Worked Example: The Scalar Case

Let \(X\) and \(Y\) be scalar random variables with variances \(4\) and \(9\), and covariance \(3\). For scalar coefficients \(a\) and \(b\), the unit-variance constraints are \(4a^2=1\) and \(9b^2=1\). Hence \(a=\pm 1/2\) and \(b=\pm 1/3\).

The covariance of the two scores is \(ab\operatorname{Cov}(X,Y)=3ab\). Choosing the signs to make it positive gives \(3(1/2)(1/3)=1/2\). The maximum correlation is therefore \(1/2\). Directly, the original variables have correlation \(3/(\sqrt{4}\sqrt{9})=3/6=1/2\), as expected: with only one variable in each set, there is no choice of direction beyond changing scale and sign.

Successive Canonical Pairs

After selecting a first pair, a later pair should capture a new association rather than reproduce the first scores. The successive variational definition imposes uncorrelatedness with earlier scores within each variable set. It does not require the two new scores to be uncorrelated across sets; their cross-set correlation is the quantity being maximized.

Definition: Let \((a_j,b_j)\), for \(j=1,\ldots,k-1\), be previously selected canonical pairs, each normalized to unit variance. The \(k\)th canonical correlation \(\rho_k\) is the maximum of \(\operatorname{Corr}(a^\top X,b^\top Y)\) over pairs satisfying \[ a^\top\Sigma_{XX}a=1,\qquad b^\top\Sigma_{YY}b=1, \] and, for every \(j<k\), \[ a^\top\Sigma_{XX}a_j=0,\qquad b^\top\Sigma_{YY}b_j=0. \] A maximizing pair \((a_k,b_k)\) gives the \(k\)th canonical variates. This definition is used for \(k=1,\ldots,\min(p,q)\), choosing any maximizer if there is a tie.

For example, \(a^\top\Sigma_{XX}a_j=\operatorname{Cov}(a^\top X,a_j^\top X)\), so the first orthogonality condition says that the new score from \(X\) is uncorrelated with each earlier score from \(X\). The condition for \(Y\) has the corresponding interpretation. These constraints make the feasible choices for each successive pair more restricted.

Worked Example: Two Distinct Pairs With Different Strengths

Let \(X_1,X_2,\varepsilon_1,\varepsilon_2\) be mutually independent standard normal variables, and set \[ X=\begin{pmatrix}X_1\\X_2\end{pmatrix},\qquad Y=\begin{pmatrix}0.8X_1+0.6\varepsilon_1\\0.3X_2+\sqrt{0.91}\,\varepsilon_2\end{pmatrix}. \] Each component of \(Y\) has variance one: \(0.8^2+0.6^2=1\), and \(0.3^2+0.91=1\). The two covariance matrices within sets are therefore both the \(2\)-by-\(2\) identity matrix, while \[ \Sigma_{XY}=\begin{pmatrix}0.8&0\\0&0.3\end{pmatrix}. \]

For unit-variance scores, write \(a=(a_1,a_2)^\top\) and \(b=(b_1,b_2)^\top\), so \(a_1^2+a_2^2=b_1^2+b_2^2=1\). Their covariance is \(0.8a_1b_1+0.3a_2b_2\). Since \(0.3\leq 0.8\), \[ |0.8a_1b_1+0.3a_2b_2| \leq 0.8\bigl(|a_1b_1|+|a_2b_2|\bigr) \leq 0.8\sqrt{a_1^2+a_2^2}\sqrt{b_1^2+b_2^2} =0.8. \] The second inequality is Cauchy–Schwarz. The bound is attained by \(a=b=(1,0)^\top\), so the first canonical correlation is \(0.8\).

For the second pair, the within-\(X\) constraint against \(a_1=(1,0)^\top\) gives \(a_2=0\) for the new coefficient vector; unit variance then requires its second coordinate to be \(1\) or \(-1\). The within-\(Y\) constraint gives the same restriction on \(b\). Taking both new vectors to be \((0,1)^\top\) yields covariance \(0.3\), and choosing opposite signs gives \(-0.3\). Thus the maximum is \(0.3\), the second canonical correlation. The example shows how successive maximization selects a new, weaker association after excluding the first pair.

Existence and Ordering of the Maxima

The variational definition describes actual maxima, not just suprema, under the stated positive-definiteness assumptions. It also implies that successive canonical correlations cannot increase. Both facts follow directly from the constrained optimization.

Theorem: If \(\Sigma_{XX}\) and \(\Sigma_{YY}\) are positive definite, then each successive maximization in the definition has a feasible pair and attains its maximum for \(k\leq\min(p,q)\). The resulting values satisfy \(\rho_1\geq\rho_2\geq\cdots\geq\rho_{\min(p,q)}\geq0\).

Proof. Suppose the first \(k-1\) pairs have been selected and are orthonormal within each set under the covariance inner products \(u^\top\Sigma_{XX}v\) and \(s^\top\Sigma_{YY}t\). In \(\mathbb R^p\), the subspace satisfying the \(k-1\) linear constraints \(a^\top\Sigma_{XX}a_j=0\) has dimension at least \(p-k+1\), which is positive for \(k\leq p\). Choose a nonzero vector in it and divide by the square root of its positive variance \(a^\top\Sigma_{XX}a\). This gives a feasible unit-variance coefficient vector. The same argument in \(\mathbb R^q\) gives a feasible vector for \(Y\), so the feasible set of pairs is nonempty.

To see that a maximum is attained, the unit-variance constraint sets are closed and bounded in Euclidean coefficient space. In fact, positive definiteness gives a constant \(c_X>0\) such that \(a^\top\Sigma_{XX}a\geq c_X\|a\|^2\), so unit variance implies \(\|a\|^2\leq 1/c_X\); similarly the \(b\) vectors are bounded. The orthogonality constraints are closed linear constraints. The feasible set of pairs is therefore closed and bounded, hence compact. The correlation is a continuous function of \(a,b\) on this set, so the extreme value theorem guarantees an attained maximum. Replacing \(b\) by \(-b\) preserves feasibility and reverses the correlation, proving that the maximum is nonnegative.

Finally, the feasible set at step \(k+1\) is a subset of the feasible set at step \(k\): it has all the earlier constraints and one additional within-\(X\) and within-\(Y\) orthogonality constraint. Maximizing the same correlation over a smaller set cannot yield a larger value. Thus \(\rho_{k+1}\leq\rho_k\). \(\square\)

Worked Example: A Weighted Combination Against One Score

Suppose \(\operatorname{Cov}(X)=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}\), \(\operatorname{Var}(Y)=1\), and \(\operatorname{Cov}(X,Y)=(0.6,0.6)^\top\). These moments can arise by taking \(X_1,X_2\) to have unit variances and covariance \(0.5\), and setting \(Y=0.4(X_1+X_2)+\varepsilon\), where \(\varepsilon\) is independent of \(X\) and has variance \(0.52\). Indeed, \(\operatorname{Var}(X_1+X_2)=1+1+2(0.5)=3\), so \(\operatorname{Var}(Y)=0.4^2(3)+0.52=1\), and each covariance \(\operatorname{Cov}(X_i,Y)\) is \(0.4(1+0.5)=0.6\).

For a coefficient vector \(a\), the covariance of \(a^\top X\) with \(Y\) is \(a^\top c\), where \(c=(0.6,0.6)^\top\), and the score variance is \(a^\top\Sigma_{XX}a\). Define \(a_0=(0.4,0.4)^\top\). Direct multiplication gives \[ \Sigma_{XX}a_0=(0.6,0.6)^\top=c,\qquad a_0^\top\Sigma_{XX}a_0=(0.4,0.4)\begin{pmatrix}0.6\\0.6\end{pmatrix}=0.48. \] For any unit-variance score \(a^\top X\), Cauchy–Schwarz in the inner product \(u^\top\Sigma_{XX}v\) gives \[ a^\top c=a^\top\Sigma_{XX}a_0 \leq \sqrt{a^\top\Sigma_{XX}a}\sqrt{a_0^\top\Sigma_{XX}a_0} =\sqrt{0.48}. \] Equality holds when \(a\) is a positive multiple of \(a_0\). The normalized choice is \(a=a_0/\sqrt{0.48}\), whose variance is one; its covariance with \(Y\), and hence its correlation with \(Y\), is \(\sqrt{0.48}\approx0.693\). This calculation makes clear that the maximizing combination need not select just one measured variable.

Common Pitfalls and Why the Definition Matters

The unit-variance formulation separates association from arbitrary units of measurement. A coefficient vector is not itself the scientific result: multiplying all its entries by a nonzero constant changes the score's scale, but not its correlation with the other score. The constraints choose a normalized representative so that the optimization is well-defined.

Maximizing covariance without controlling scale. Increasing a coefficient vector can increase covariance indefinitely. Impose unit score variances, or optimize the correlation ratio directly.
Confusing covariance with correlation. Under the unit-variance constraints they are equal; without normalization, covariance depends on the units and scale of the scores.
Treating the sign as substantive. Flipping the sign of one score reverses the correlation. The maximum is nonnegative by convention, and the sign of a coefficient vector is not uniquely determined by the association.
Leaving out successive constraints. Repeating the first maximization alone could select the same pair again. Later pairs are defined with within-set uncorrelatedness constraints against earlier scores.

Key takeaway: Canonical correlation is a sequence of constrained correlation maxima. Unit variances fix scale, while successive within-set orthogonality constraints ensure each later pair is chosen from a more restricted set of linear combinations.

Check Your Understanding

Answer each question using the variational definition.

Why does maximizing covariance without a scale constraint fail to define a finite optimization problem?
What covariance constraints express that a new \(X\)-score and a new \(Y\)-score are each uncorrelated with their earlier scores?
Why can the maximum signed correlation be taken to be nonnegative?
In the diagonal example, why does the second pair have correlation \(0.3\) rather than \(0.8\)?
What role does positive definiteness of the within-set covariance matrices play in the definition and existence argument?