Sparse CCA Objectives and Constraints

Why Add Sparsity to CCA?

Canonical correlation analysis (CCA) finds linear combinations of two data blocks whose scores are strongly associated. In a multi-omics analysis, each block may contain many measured features, and an ordinary CCA weight vector can assign nonzero weights to nearly all of them. That can make the resulting combinations difficult to summarize and unstable when features are numerous or correlated.

Sparse CCA modifies the optimization problem so that some weights are exactly zero. The remaining nonzero weights identify features that contribute to the fitted combinations. This is a form of variable selection, not proof that selected features are biologically causal or uniquely important. The central design choice is how to encourage zeros while preserving a well-defined association objective.

Correlation Objectives With L1 Constraints

The sample correlation between the two scores is

This expression is unchanged if either weight vector is multiplied by a positive constant. Therefore, an L1 bound alone would not restrict the solution: a vector could simply be rescaled to meet the bound without changing the correlation. The usual correlation-based constrained formulation fixes each score variance to one and then imposes L1 bounds:

Under the variance constraints, the objective is the score correlation. Here \(\|\mathbf{a}\|_1=\sum_j|a_j|\), and \(c_x,c_y\) set the allowed L1 sizes. The constants must be large enough for the corresponding variance constraint to be feasible. A tighter bound generally restricts the weights more strongly, though it does not always produce a strictly smaller support at every value of the bound.

A common penalized version keeps the variance normalizations but subtracts L1 costs from the objective:

The nonnegative parameters \(\lambda_x\) and \(\lambda_y\) control the penalty strengths. This penalized objective is not itself the unpenalized score correlation: it trades off association against weight complexity. A constrained formulation and a penalized formulation express related preferences, but their tuning parameters are not generally interchangeable by a simple conversion.

What the Penalty Changes

A penalty makes the cost of a weight depend on its absolute size. Since the L1 norm sums absolute weights, adding a feature with a small nonzero weight can still incur a cost. This differs from merely reporting a dense solution and then dropping small coefficients afterward: the penalty changes which solution is optimal in the first place.

For intuition, consider a candidate with unit-length weights in both blocks and identity within-block covariance matrices. For this candidate, the penalized objective is its cross-covariance score minus \(\lambda_x\|\mathbf{a}\|_1+\lambda_y\|\mathbf{b}\|_1\). The following calculation compares particular candidates; it does not assert that these candidates exhaust all possible weight vectors.

Normalization, Scaling, and the Meaning of the Objective

The L1 constraints act on weights, so their practical effect depends on how features are scaled. If one feature is measured in large numerical units and another in small units, the corresponding weights can have different magnitudes for reasons unrelated to scientific importance. A common preparation is to center and scale features before fitting, while checking that this choice is suitable for the measurement types and scientific question. The covariance matrices and all weights must then be interpreted on that chosen scale.

The importance of normalization is also visible when the within-block covariance matrices are not identity matrices. The cross-covariance entry alone does not give the correlation between a pair of individual features. That correlation divides the cross-covariance by the product of their standard deviations.

Assumptions, Diagnostics, and Alternatives

The optimization itself is a descriptive procedure and does not require normally distributed measurements. It does require matched rows across the two blocks: each row of \(\mathbf{X}\) must correspond to the same observational unit as the row of \(\mathbf{Y}\). Centering, scaling, missing-data handling, and any removal of zero-variance features should be decided before fitting and applied consistently.

For the variance-normalized formulation, each chosen weight vector must yield a well-defined, nonzero score variance. Positive-definite within-block covariance matrices make the normalization straightforward. If a covariance matrix is singular, some directions may have zero variance or may not be uniquely normalized; sparse constraints do not automatically solve that issue. Inspect the covariance structure and use an appropriate regularized formulation if necessary.

Useful diagnostics include checking the scale of each feature, the number of nonzero weights in each block, sensitivity of the supports to modest changes in the constraints or penalties, and the size of the association on data not used to choose the weights. Highly correlated features can exchange weight or selection across fits, so a selected support need not be stable even when the overall score association is similar. A strong in-sample objective is not, by itself, evidence that the association will replicate.

A common mistake is to treat an L1 penalty as a cosmetic adjustment to a classical CCA solution, or to interpret every selected feature as uniquely important. The correction is that the penalty changes the optimization trade-off, and correlated features may substitute for one another. Sparse CCA provides a compact fitted representation under chosen modeling and scaling decisions; it does not establish causation, guarantee support stability, or automatically provide valid inferential conclusions.