Why Penalize PLS Directions?
In multi-omics analysis, two matched data blocks may each contain many features, while only a subset may contribute to their shared pattern. Two-block partial least squares (PLS) seeks directions with high cross-block covariance. Sparse PLS adds an L1 penalty to encourage directions with zero coefficients for some features. The result is a covariance-focused objective that can produce more selective directions than ordinary PLS, while making clear that selection depends on the chosen scaling and penalty strengths.
The construction below assumes the rows of \(X\in\mathbb{R}^{n\times p}\) and \(Y\in\mathbb{R}^{n\times q}\) represent the same \(n\) observations in the same order. Both blocks are centered columnwise, so each column has sample mean zero. Define their sample cross-covariance matrix by \(S_{xy}=X^\top Y/(n-1)\). The directions \(u\in\mathbb{R}^p\) and \(v\in\mathbb{R}^q\) produce scores \(Xu\) and \(Yv\), whose sample covariance is \(u^\top S_{xy}v\).
The objective uses the covariance between the two scores, not their correlation. The penalty parameters must therefore be chosen on a scale compatible with \(S_{xy}\): multiplying the measurements in one block changes the covariance term, but does not automatically change the numerical values of \(\lambda_x\) and \(\lambda_y\). An alternative common formulation maximizes covariance subject to L1 bounds on the directions as well as Euclidean normalization. Such constraints and penalties both encourage sparsity, but their tuning parameters need not correspond exactly in a finite, nonconvex problem.
Why Singular Vectors Appear
First set both penalties to zero. With Euclidean norms bounded by one, the problem becomes maximizing \(u^\top S_{xy}v\). Write a singular value decomposition \(S_{xy}=U D V^\top\), where the diagonal entries of \(D\) are nonnegative and ordered \(d_1\geq d_2\geq\cdots\). Let \(a=U^\top u\) and \(b=V^\top v\). Orthogonal transformations preserve Euclidean norms, so \(\|a\|_2\leq 1\) and \(\|b\|_2\leq 1\), and \(u^\top S_{xy}v=\sum_k d_k a_k b_k\).
The middle inequality is Cauchy–Schwarz. Taking \(u\) and \(v\) to be leading left and right singular vectors attains \(d_1\), so the unpenalized maximum is the largest singular value of \(S_{xy}\). With positive L1 penalties, the same singular-vector solution generally does not maximize the penalized objective: a direction with slightly lower covariance may score better if it uses fewer features.
The penalty terms are not jointly separable from the covariance term: \(u\) and \(v\) occur together in \(u^\top S_{xy}v\). Thus the full optimization is generally nonconvex as a joint problem, even though fixing either direction makes the covariance term linear in the other. This formulation motivates algorithms that update directions in turn, but it does not make every such algorithm globally optimal. In particular, a solution returned by an iterative procedure may depend on its starting values.
Worked Example: Computing Cross-Covariances and Candidate Scores
Consider three hypothetical matched observations, with columns already centered: $$ X=\begin{pmatrix}-1&1\\0&-2\\1&1\end{pmatrix}, \qquad Y=\begin{pmatrix}-2&0.5\\0&-1\\2&0.5\end{pmatrix}. $$ There are \(n=3\) observations, so \(S_{xy}=X^\top Y/2\). Direct multiplication gives $$ X^\top Y= \begin{pmatrix}4&0\\0&3\end{pmatrix}, \qquad S_{xy}= \begin{pmatrix}2&0\\0&1.5\end{pmatrix}. $$
Set \(\lambda_x=\lambda_y=0.25\). For the sparse candidate directions \(u=v=(1,0)^\top\), the covariance is \(2\) and each L1 norm is \(1\). Its objective value is \(2-0.25-0.25=1.50\). For \(u=v=(0,1)^\top\), the value is \(1.5-0.25-0.25=1.00\).
Now consider the dense candidate \(u=v=(1/\sqrt{2},1/\sqrt{2})^\top\). Its covariance is \(2(1/2)+1.5(1/2)=1.75\). Each L1 norm is \(\sqrt{2}\), so its objective is \(1.75-0.25\sqrt{2}-0.25\sqrt{2}\approx1.043\). Among these three candidates, the first sparse direction has the highest objective. This comparison illustrates the trade-off; it does not establish that the first candidate is the global maximizer over all possible directions.
How This Differs from Sparse CCA
PLS and canonical correlation analysis (CCA) use different targets. PLS seeks directions with high covariance between scores. CCA seeks directions with high correlation, which divides score covariance by the product of the score standard deviations. When \(S_{xx}\) and \(S_{yy}\) are nonsingular, the classical CCA problem can be represented through the whitened cross-covariance matrix \(S_{xx}^{-1/2}S_{xy}S_{yy}^{-1/2}\). The whitening accounts for within-block variances and correlations. In contrast, the unpenalized two-block PLS-SVD uses \(S_{xy}\) itself.
Sparse CCA therefore penalizes directions within a variance-normalized association problem; sparse PLS penalizes directions within a covariance-maximization problem. A sparse PLS procedure is not obtained simply by using a sparse CCA update and changing its label. The normalization and the matrix being optimized differ, so the selected directions can differ even with the same observations. The earlier material on CCA and PLS establishes the classical distinction; the examples here show how it persists when sparsity is introduced.
Worked Example: Covariance and Correlation Can Favor Different Features
Suppose hypothetical centered blocks have covariance matrices $$ S_{xx}=\begin{pmatrix}100&0\\0&1\end{pmatrix}, \qquad S_{yy}=\begin{pmatrix}1&0\\0&1\end{pmatrix}, \qquad S_{xy}=\begin{pmatrix}4&0\\0&1\end{pmatrix}. $$ These matrices are compatible with a joint covariance matrix: after scaling each block to unit marginal variance, the cross-block entries are \(0.4\) and \(1\), both within the allowable correlation range.
For PLS, the singular values of \(S_{xy}\) are \(4\) and \(1\), so the leading coordinate directions use feature 1 in both blocks. For CCA, the whitened cross-covariance is $$ S_{xx}^{-1/2}S_{xy}S_{yy}^{-1/2} = \begin{pmatrix}0.1&0\\0&1\end{pmatrix} \begin{pmatrix}4&0\\0&1\end{pmatrix} = \begin{pmatrix}0.4&0\\0&1\end{pmatrix}. $$ The leading CCA directions instead use feature 2, with correlation \(1\), rather than feature 1, with correlation \(0.4\). PLS favors the larger covariance, while CCA favors the larger variance-normalized association. This example concerns the classical objectives; adding sparsity penalties can further change either solution.
Scaling, Diagnostics, and Limitations
Because the covariance term depends on measurement scale, preprocessing is part of the sparse PLS specification. Centering is needed for the stated sample covariance formula. Standardizing columns to unit variance before forming \(S_{xy}\) changes the target toward a cross-correlation matrix and can prevent high-variance features from dominating solely because of their units. It also changes the analysis: a standardized feature contributes according to a different scale than it does in the original measurement units. The preprocessing and penalty values should therefore be reported together.
A useful diagnostic is to compare the objective and selected support across reasonable initial values, penalty choices, and scaling decisions. If the procedure converges to substantially different supports or objective values from different starts, the nonconvex optimization may be finding different local solutions. Also inspect whether the returned directions actually yield nonzero scores and whether their observed score covariance agrees with \(u^\top S_{xy}v\). If many features have nearly identical cross-covariances, small data changes can swap selected features even when the overall covariance changes little.
Sparse PLS describes a covariance-oriented association in the observed sample. A nonzero coefficient is not, by itself, evidence of a causal effect or a uniquely important biological feature. The method also does not automatically provide valid post-selection uncertainty estimates. In high dimensions, \(S_{xy}\) can be noisy; sparsity can simplify the directions but does not remove sampling error or guarantee stable feature selection. Held-out evaluation and stability assessment address different questions from maximizing the training-sample objective.
Practice and Solutions
Check Your Understanding
Answer each question, then check the worked solutions below.
- For an unpenalized cross-covariance matrix with leading singular value \(3.2\), what is the maximum covariance between unit-norm directions?
- For \(\lambda_x=\lambda_y=0.2\), calculate the penalized objective at \(u=v=(1,0)^\top\) when \(u^\top S_{xy}v=1.4\).
- Why might PLS and CCA select different coordinate directions when one feature has much larger variance than another?
- What should be checked if repeated runs of a sparse PLS algorithm return different supports?
Solutions. (1) The maximum is the leading singular value, \(3.2\), by the singular-value argument above. (2) Each L1 norm is \(1\), so the objective is \(1.4-0.2-0.2=1.00\). (3) PLS ranks directions by covariance, whereas CCA normalizes covariance by the score standard deviations; a large-variance feature can have high covariance but a lower correlation. (4) Compare objective values, starting values, scaling, and penalty choices. Different supports at similar objective values can also indicate that correlated features are interchangeable or that selection is unstable.