Group-Sparse CCA and PLS

Why Group Sparsity?

In a two-block analysis, an ordinary sparse method can retain isolated features from across an assay, even when the scientific question concerns coherent sets such as a pathway, a gene module, or a predefined metabolite class. Group sparsity instead encourages a whole predefined set of features to enter or leave a variate together. The groups must be specified before fitting; the penalty does not discover biological groups from the data.

The goal is not necessarily to make every coefficient in a selected group nonzero. A group penalty controls whether a group contributes at all. Within a retained group, coefficients can still vary in size, and some may be zero by chance or through an additional penalty. This distinction matters when interpreting a selected assay-defined set.

A common default is \(w_g=\sqrt{|G_g|}\), where \(|G_g|\) is the number of features in group \(g\). This partly adjusts for the fact that the norm of a noise-only group tends to grow with its size. Other weights are possible, but they change how much penalty groups of different sizes receive. Weights and feature scaling should be chosen deliberately, not treated as harmless implementation details.

Objectives for CCA and PLS

Write \(u\) and \(v\) for the coefficient vectors defining scores \(Xu\) and \(Yv\). With sample cross-covariance matrix \(S_{XY}=X^\mathsf{T}Y/n\), a group-penalized two-block PLS objective can be written as

Here \(H_1,\ldots,H_L\) are the predefined groups in the \(Y\) block, \(\Omega_H(v)=\sum_{\ell=1}^{L}a_\ell\|v_{H_\ell}\|_2\), and \(\lambda_X,\lambda_Y\geq0\) control the penalties. The cross-block term rewards covariance between the scores; the penalties discourage retaining many groups.

CCA targets association while accounting for the within-block score variances. In its usual unpenalized form, it maximizes \(u^\mathsf{T}S_{XY}v\) subject to \(u^\mathsf{T}S_{XX}u\leq1\) and \(v^\mathsf{T}S_{YY}v\leq1\), where \(S_{XX}=X^\mathsf{T}X/n\) and \(S_{YY}=Y^\mathsf{T}Y/n\). Group penalties can be added to this objective, but the covariance-metric constraints affect computation: ordinary Euclidean group thresholding is not generally an exact update under a non-diagonal metric. A Euclidean-coordinate or regularized formulation may allow a simple group-thresholding step; otherwise, the update must respect the chosen CCA metric. The exact formulation and coefficient scale therefore matter when reporting a group-sparse CCA fit.

These formulations assume the two blocks describe the same samples, features have been centered, and the groups have been defined consistently. Scaling is consequential: a feature measured on a large numerical scale can dominate a covariance objective and influence selection. Standardize features when that is appropriate for the scientific question, or state and justify a different scaling. CCA also requires a workable score-variance geometry; singular covariance matrices need an appropriate regularization or dimension-reduction treatment rather than an unexamined inverse.

How Group Soft-Thresholding Works

The group-L1 penalty has a simple proximal update in Euclidean coordinates. For a vector \(z\), the proximal operator minimizes squared distance to \(z\) plus a weighted group penalty. The minimization separates across groups, so each group is either shrunk toward zero as a unit or set exactly to zero.

To justify the formula, consider minimizing \(\frac12\|a-z_{G_g}\|_2^2+t w_g\|a\|_2\). For a nonzero minimizer, the derivative condition is \(a-z_{G_g}+t w_g a/\|a\|_2=0\). This requires \(a\) to point in the direction of \(z_{G_g}\). Writing \(a=c z_{G_g}\) for \(c>0\) gives \(c=1-tw_g/\|z_{G_g}\|_2\), which is positive only when \(\|z_{G_g}\|_2>tw_g\). If that inequality fails, the subgradient condition at zero is satisfied, so the minimizer is zero. Combining the cases gives the stated operator.

In an alternating Euclidean-coordinate algorithm, one can hold \(v\) fixed, take a gradient-ascent step for the smooth cross-block term in \(u\), apply group soft-thresholding, and then enforce the coefficient norm constraint if needed. The corresponding update for \(v\) is analogous. Alternating updates do not make the joint objective globally concave: the method can depend on initialization, and convergence to a stationary solution does not guarantee the best possible solution. Metric-constrained CCA requires a metric-aware update or a suitable transformed formulation.

Assay-Defined Sets and Penalty Choices

The scientific interpretation depends on both the groups and the fitted support. A group might be a predefined gene set measured in one assay, a panel of related proteins, or a metabolite class. If a group is retained, the defensible conclusion is that its features contribute jointly to this fitted variate under the selected model and tuning choices. It is not evidence that every member is individually associated, that the group is biologically coherent in this sample, or that the group causes variation in the other block.

Featurewise L1 penalties answer a different selection question: they can retain some variables within a set while removing others, but do not directly select entire sets. Group-L1 penalties select sets but generally do not produce sparsity within a selected set. A sparse-group penalty combines both:

This can select groups and then remove individual coefficients inside retained groups. It adds another tuning choice and should not be described as the same procedure as group L1 alone.

Diagnostics, Limitations, and Common Pitfalls

Group-sparse solutions are useful when coherent feature sets are central to the analysis, but a selected support should be checked rather than read off as a definitive discovery. At minimum, inspect whether the fitted groups change substantially under reasonable changes in tuning, scaling, or sample resampling; whether one very large or highly correlated set dominates; and whether the groups were defined without using the outcome information that the model is intended to evaluate. Held-out association can inform tuning, but it does not remove instability from small samples or correlated features.

A key limitation is that predefined groups can be incomplete, overlapping, or biologically heterogeneous. The formula above assumes a nonoverlapping partition. Overlapping groups require a separately specified penalty and algorithm; simply counting a feature in several groups changes its effective penalty. In addition, group selection can be unstable when sets are correlated: different but related groups may carry similar cross-block signal, so a modest data perturbation can change which one is retained.

• Wrong: “The group penalty is zero for unselected sets, so every feature in each selected set must have a nonzero coefficient.” Correction: The group penalty applies to the group norm; group retention does not guarantee that every coefficient is nonzero.
• Wrong: “A selected assay group is statistically significant and causes the other block’s score.” Correction: Selection is a model-dependent association pattern, not a significance test or causal conclusion.
• Wrong: “The same numeric penalty is comparable across any group definitions and scaling choices.” Correction: Feature scale, group size, weights, and score geometry all affect the effective penalty.
• Wrong: “The Euclidean threshold formula is automatically an exact CCA update.” Correction: It is exact for the stated Euclidean proximal subproblem; a non-diagonal CCA covariance metric requires a metric-aware treatment.

Use group L1 when the unit of interpretation is a predefined feature set and within-set sparsity is not the primary goal. Use featurewise penalties when individual features are the target of selection, or a sparse-group formulation when both levels matter. In every case, report group definitions, weights, scaling, tuning choices, and the relevant CCA or PLS normalization so that the support can be interpreted in context.