Canonical Correlation as a Constrained Optimization Problem

From Two Data Blocks to Two Scores

In multi-omics studies, each observational unit may have measurements in two blocks, such as an expression block and a protein block. Canonical correlation analysis (CCA) asks whether a linear score from one block can be paired with a linear score from the other so that their correlation is large. The weights defining those scores are not arbitrary: their scale must be fixed, or the covariance objective can be increased merely by multiplying a weight vector by a constant.

Let \(X\) be an \(n\times p\) matrix and \(Y\) an \(n\times q\) matrix. Each row refers to the same observational unit in both matrices. Assume the columns have been centered, so each column has sample mean zero. For weight vectors \(a\in\mathbb{R}^p\) and \(b\in\mathbb{R}^q\), the two score vectors are \(Xa\) and \(Yb\). Define the sample covariance matrices

The sample variances of the scores are \(a^\mathsf{T}S_{xx}a\) and \(b^\mathsf{T}S_{yy}b\), and their sample covariance is \(a^\mathsf{T}S_{xy}b\). Thus, when both score variances are positive, their sample correlation is

Why the Constraints Are the Right Normalization

A correlation does not change if either score is multiplied by a positive constant. The variance constraints select a scale for each score without changing the directions in the two feature spaces that make the association strong. A sign change reverses a score, so the sign of one weight vector can be chosen to make the optimized covariance nonnegative.

Under the constraints, the denominator of the correlation is \(\sqrt{1\cdot 1}=1\). Consequently, maximizing the correlation is exactly maximizing the score covariance. Without constraints, the covariance has no finite maximum whenever some pair of directions has nonzero covariance: replacing \(a\) by \(ca\) multiplies that covariance by \(c\). The variance constraints prevent that meaningless scale inflation.

The standard formulation assumes the same \(n\) paired observational units are used for both blocks, the columns are centered, and the covariance matrices are calculated from finite observations. For the usual smooth constrained optimization, assume \(S_{xx}\) and \(S_{yy}\) are positive definite. This ensures that every nonzero weight has positive score variance and that each unit-variance constraint defines a bounded ellipsoid. No multivariate normality assumption is needed to define or solve this sample optimization problem.

Deriving the Stationary Equations

For positive-definite within-block covariance matrices, the feasible sets \(a^\mathsf{T}S_{xx}a=1\) and \(b^\mathsf{T}S_{yy}b=1\) are closed and bounded. Their product is compact, and the objective \(a^\mathsf{T}S_{xy}b\) is continuous. Therefore a global maximum exists. At a regular constrained optimum, the Lagrange multiplier conditions give the stationary equations.

Use multipliers \(\lambda\) and \(\mu\) for the two constraints, with the Lagrangian

The covariance matrices \(S_{xx}\) and \(S_{yy}\) are symmetric. Differentiating with respect to \(a\) and \(b\), and setting each gradient to zero, yields

These equations are the stationary conditions for the constrained optimization. They state that the cross-block covariance direction associated with one score must be proportional to the within-block covariance direction associated with the other score. The proportionality constants are not arbitrary. Premultiply the first equation by \(a^\mathsf{T}\) and use \(a^\mathsf{T}S_{xx}a=1\); premultiply the second by \(b^\mathsf{T}\) and use \(b^\mathsf{T}S_{yy}b=1\). Since \(b^\mathsf{T}S_{yx}a=a^\mathsf{T}S_{xy}b\), both multipliers equal the objective value:

This derivation supplies the stationary conditions, not a complete method for finding all canonical pairs. In particular, the equations can be combined into other matrix problems, but those reformulations are a separate step. The equations themselves are useful for understanding what a candidate solution must satisfy and for checking numerical results.

Interpreting the Formulation and Checking Its Limits

CCA is not the only way to construct paired linear scores. Partial least squares (PLS) is a useful comparison: its two-block covariance-maximization formulation seeks directions with large cross-block covariance, whereas CCA maximizes correlation by scaling each score to unit variance. When features have different variances, this distinction can change the selected directions. CCA accounts for within-block scales through its constraints; PLS does not impose these same unit-variance constraints. Neither objective alone establishes that a score association is causal or biologically meaningful.

The positive-definiteness assumption matters in high-dimensional omics data. If \(p\) or \(q\) is large relative to \(n\), a within-block sample covariance matrix may be singular. Then some nonzero weights can have zero score variance, the unit-variance constraint may not define a bounded ellipsoid, and the standard stationary derivation may not apply as stated. Regularization or a treatment of singular covariance matrices is needed rather than silently using an inverse or assuming the optimization is well behaved.

Other warning signs include misaligned rows, uncentered columns when the intended analysis uses centered covariance, and score variances that are nearly zero. These can make calculated score correlations misleading or numerically unstable. A practical check is to verify that both computed score variances are one (up to numerical tolerance), that the reported correlation equals the constrained covariance, and that the stationary residuals are small:

A common mistake is to maximize \(a^\mathsf{T}S_{xy}b\) while omitting the variance constraints, then interpret the result as the largest correlation. That objective can grow just by rescaling a weight vector and is not the CCA problem. The correction is to impose unit variance in each block; only then is the covariance objective equal to the score correlation. A second mistake is to treat any solution to the stationary equations as automatically the global maximum. The equations are necessary at a regular optimum, but candidate solutions must still be compared or otherwise shown to attain the maximum.