Soft Thresholding in Sparse PLS

Why Threshold a PLS Weight?

A two-block PLS direction seeks variables in one data block whose weighted combination is associated with a weighted combination in the other block. In high-dimensional omics data, an unpenalized direction can assign nonzero weights to many variables, making it difficult to identify a smaller set of variables contributing to the fitted association. Soft thresholding gives a direct way to set some weights to zero while retaining graded weights for those that remain.

The earlier material on Sparse PLS Through Penalized Covariance motivates penalizing a covariance-based direction. Here the focus is the coordinate update: given the current direction in one block, how do we calculate a sparse weight in the other block? The update uses a vector of cross-block covariance scores, applies a common threshold to its coordinates, and normalizes the result.

Definition: Let $X$ be an $n\times p$ matrix and $Y$ an $n\times q$ matrix of paired observations, with columns centered across the $n$ observations. For a current weight $b$ on the $Y$ block, the cross-block covariance score vector for the $X$ block is $c_X=S_{XY}b$, where $S_{XY}=X^\top Y/(n-1)$. Its $j$th coordinate is the sample covariance between $X_j$ and the current score $Yb$.

The score $c_X$ indicates the direction and size of each variable’s covariance with the current score in the other block. The coordinate scale depends on how the variables and blocks have been scaled. If variables are intended to compete on a comparable scale, their preprocessing should make that reasonable before a shared threshold is applied. The score has covariance units; therefore the threshold $\lambda$ must use the same units.

Soft Thresholding and the Sparse Update

For a real score $c_j$, soft thresholding subtracts $\lambda$ from its magnitude when that magnitude exceeds the threshold, keeps its sign, and otherwise returns zero. The threshold is nonnegative. Applying the operation coordinate by coordinate gives a vector; normalizing that vector produces a unit-length PLS weight.

Definition: The soft-thresholding operator at threshold $\lambda\geq 0$ is $S(c_j,\lambda)=\operatorname{sign}(c_j)(|c_j|-\lambda)_+$, where $r_+=\max(r,0)$. For a score vector $c_X$, the sparse PLS update is $$ \widetilde a=S(c_X,\lambda),\qquad a=\frac{\widetilde a}{\|\widetilde a\|_2}, $$ provided $\|\widetilde a\|_2>0$. The selected variables are the coordinates with $|c_{Xj}|>\lambda$.

The strict inequality in the support rule matters: a coordinate whose score magnitude equals $\lambda$ becomes exactly zero. Soft thresholding shrinks each surviving score toward zero by the same amount; it does not merely retain the original score unchanged. Normalization then rescales all surviving coordinates by the same positive factor, preserving their relative magnitudes and signs.

Why the Update Has This Form

Soft thresholding is the solution to a standard one-dimensional penalized least-squares problem. For a score $c_j$, consider the value $z$ that minimizes squared discrepancy from the score plus an absolute-value penalty. The quadratic term favors $z=c_j$; the penalty favors $z=0$.

\widehat z=\arg\min_z\left\{\frac{1}{2}(z-c_j)^2+\lambda|z|\right\}.

For $z>0$, the derivative is $z-c_j+\lambda$; setting it to zero gives $z=c_j-\lambda$, which is valid only when $c_j>\lambda$. For $z<0$, the derivative is $z-c_j-\lambda$; setting it to zero gives $z=c_j+\lambda$, valid only when $c_j<-\lambda$. At $z=0$, the minimum occurs when the slope from the quadratic, $-c_j$, is within the range of slopes contributed by $\lambda|z|$, namely $[-\lambda,\lambda]$. Thus $z=0$ when $|c_j|\leq\lambda$. Together these cases give $S(c_j,\lambda)$.

Applying this coordinate-wise yields the thresholded vector. In the sparse PLS update, that vector supplies the direction, while unit-length normalization enforces the usual weight scale. When at least one coordinate survives, this update also maximizes the penalized linear score $c_X^\top a-\lambda\|a\|_1$ over unit vectors $a$. To see why, write $d_j=|c_{Xj}|-\lambda$. For any unit vector,

c_X^\top a-\lambda\|a\|_1 \leq \sum_j d_j|a_j| \leq \sum_j (d_j)_+|a_j| \leq \left\|S(c_X,\lambda)\right\|_2\|a\|_2 =\left\|S(c_X,\lambda)\right\|_2.

The last inequality is Cauchy–Schwarz. Equality is attained by taking $a$ proportional to the signed thresholded vector, so the normalized update is a maximizer under this condition. If no coordinate exceeds $\lambda$, the thresholded vector is zero and cannot be normalized; the stated sparse update is then undefined. This is an important boundary case, not a weight vector with all zero entries.

Calculating Thresholded Weights

Worked Example: Thresholding a Given Covariance Score

Suppose the score vector for four hypothetical molecular variables is $c_X=(0.42,-0.28,0.19,0.05)^\top$, measured in covariance units. Let $\lambda=0.12$ in the same units. Apply soft thresholding, normalize, and identify the selected variables.

The coordinate calculations are $S(0.42,0.12)=0.42-0.12=0.30$, $S(-0.28,0.12)=-(0.28-0.12)=-0.16$, $S(0.19,0.12)=0.19-0.12=0.07$, and $S(0.05,0.12)=0$, since $0.05\leq0.12$. Thus $\widetilde a=(0.30,-0.16,0.07,0)^\top$.

Its squared norm is $0.30^2+(-0.16)^2+0.07^2=0.0900+0.0256+0.0049=0.1205$, so $\|\widetilde a\|_2=\sqrt{0.1205}\approx0.3471$. The normalized weight is

a=\frac{(0.30,-0.16,0.07,0)^\top}{\sqrt{0.1205}} \approx(0.8642,-0.4609,0.2017,0)^\top.

The selected variables are the first three: their score magnitudes exceed $0.12$. The fourth receives zero weight. The weight is unit length up to rounding, and its signs retain the signs of the corresponding covariance scores.

Worked Example: Obtaining Scores From a Cross-Covariance Matrix

For hypothetical centered and appropriately scaled blocks, suppose the cross-covariance matrix is

S_{XY}= \begin{pmatrix} 0.30&0.10\\ -0.20&0.05\\ 0.08&0.24\\ 0.04&-0.12 \end{pmatrix}, \qquad b= \begin{pmatrix} 0.6\\ 0.8 \end{pmatrix}.

The current weight has unit length because $0.6^2+0.8^2=1$. Multiplying gives $c_X=S_{XY}b=(0.30(0.6)+0.10(0.8),-0.20(0.6)+0.05(0.8),0.08(0.6)+0.24(0.8),0.04(0.6)-0.12(0.8))^\top=(0.26,-0.08,0.24,-0.072)^\top$.

Set $\lambda=0.10$. The thresholded vector is $\widetilde a=(0.16,0,0.14,0)^\top$, because only $0.26$ and $0.24$ have magnitudes greater than $0.10$. Its norm is $\sqrt{0.16^2+0.14^2}=\sqrt{0.0452}\approx0.2126$. Therefore $a\approx(0.7526,0,0.6585,0)^\top$, with variables 1 and 3 selected.

Worked Example: Comparing Thresholds on the Same Scores

Return to $c_X=(0.42,-0.28,0.19,0.05)^\top$. At $\lambda=0.20$, soft thresholding gives $(0.22,-0.08,0,0)^\top$. The norm is $\sqrt{0.22^2+0.08^2}=\sqrt{0.0548}\approx0.2341$, so $a\approx(0.9398,-0.3417,0,0)^\top$. Variables 1 and 2 are selected.

At $\lambda=0.30$, the thresholded vector is $(0.12,0,0,0)^\top$, since the second score has magnitude $0.28<0.30$ and the third has magnitude $0.19<0.30$. Normalization gives $a=(1,0,0,0)^\top$: only variable 1 is selected. At $\lambda=0.42$, every score magnitude is at most the threshold, so the thresholded vector is zero and no normalized update exists.

Increasing $\lambda$ can remove variables from the support, but the normalized weights need not simply decrease in magnitude: once only one coordinate survives, its normalized weight is 1 or $-1$. The threshold controls selection before normalization.

Using the Update and Checking Its Behavior

In a two-block alternating procedure, the update is applied to one block using the current weight in the other block. The roles are then reversed: compute $c_Y=S_{XY}^\top a$, soft-threshold it at the chosen threshold for the $Y$ block, and normalize if any coordinates survive. Repeating these updates changes the scores as the paired directions change. This tutorial concerns the thresholding step, not the choice of threshold by held-out association.

Form a covariance score.
For the $X$ update, calculate $c_X=S_{XY}b$ from the current $Y$-block weight $b$.

Apply the threshold.
Compute $S(c_{Xj},\lambda)$ for every variable. A variable remains only if $|c_{Xj}|>\lambda$.

Check and normalize.
If the thresholded vector is nonzero, divide it by its Euclidean norm. If it is zero, there is no direction to normalize.

Inspect the result.
Confirm the support, signs, unit norm, and consistency of the score and threshold scales before using the direction in the next alternating update.

A few diagnostics help distinguish a valid sparse update from a computational or interpretive problem:

No surviving coordinates: if $\max_j|c_{Xj}|\leq\lambda$, normalization would divide by zero. The threshold must be reduced or the update must be handled by an explicitly defined algorithmic fallback.
Unexpectedly different selection across blocks: the two blocks may have different scales. Check centering and scaling, and remember that a common numerical threshold need not imply comparable stringency across differently scaled scores.
Weights do not have unit norm: check that normalization was performed after thresholding and that the norm was calculated from all surviving coordinates.
Support changes abruptly: this is expected near a threshold boundary. A small change in a score can change whether $|c_{Xj}|>\lambda$, especially for weak scores.

Alternatives and Limitations

Approach	Effect on a score	Implication for sparse PLS
Soft thresholding	Subtracts $\lambda$ from the magnitude of each surviving score	Produces exact zeros and shrinks retained coordinates
Hard thresholding	Sets scores below a cutoff to zero but leaves survivors unchanged	Produces exact zeros without the same continuous shrinkage; selection can be more abrupt
Ridge-style shrinkage	Shrinks scores continuously without setting coordinates exactly to zero	Can stabilize or reduce weights but does not provide a sparse support by itself

Soft thresholding is a simple, transparent update, but it has limitations. It does not establish that selected variables are biologically important, nor does a nonzero weight establish causation. Results depend on preprocessing and on the chosen threshold; correlated variables may substitute for one another as the direction changes. A zero coordinate means only that its current covariance score did not exceed the threshold in this update. Also, alternating updates can depend on their starting weights, so a single thresholded step is not a guarantee of a unique final solution.

Common Mistake

A common mistake is to say, “The thresholded vector has three nonzero entries, so those entries are the final weights.” That confuses the unnormalized thresholded scores with the PLS weight. The support is determined by the thresholded vector, but the weight used for the direction is that vector divided by its Euclidean norm. Conversely, if every thresholded entry is zero, assigning an all-zero vector as a unit PLS weight is not valid: its norm is zero, not one.

Key takeaway: For a fixed weight in the opposite block, compute the cross-block covariance scores, apply coordinate-wise soft thresholding, and normalize the result only when it is nonzero. The threshold determines which variables survive; normalization sets the scale of the resulting direction.

Check Your Understanding

Try each question, then compare your work with the solutions below.

For $c=(0.30,-0.20,0.10)^\top$ and $\lambda=0.10$, calculate the thresholded vector, normalized weight, and selected variables.
Why must $\lambda$ and the coordinates of $c_X$ be expressed on compatible scales?
What should be reported if $\max_j|c_{Xj}|\leq\lambda$?
Explain why a surviving coordinate’s normalized weight is not generally equal to its original covariance score.

Solutions. (1) The thresholded vector is $(0.20,-0.10,0)^\top$, because the third score equals the threshold and is set to zero. Its norm is $\sqrt{0.20^2+(-0.10)^2}=\sqrt{0.05}\approx0.2236$. Thus $a\approx(0.8944,-0.4472,0)^\top$; variables 1 and 2 are selected. (2) The comparison $|c_{Xj}|>\lambda$ has meaning only if both quantities use the same units. If variables have very different scales, a common threshold may favor variables with larger-scale scores. (3) The thresholded vector is zero, so no unit-length update can be formed by normalization; report that no coordinate survives and use a specified response such as reducing the threshold. (4) Soft thresholding first subtracts $\lambda$ from each surviving score’s magnitude, then normalization divides all surviving values by their joint Euclidean norm. Both operations generally change their numerical values.

Approach	Effect on a score	Implication for sparse PLS
Soft thresholding	Subtracts \(\lambda\) from the magnitude of each surviving score	Produces exact zeros and shrinks retained coordinates
Hard thresholding	Sets scores below a cutoff to zero but leaves survivors unchanged	Produces exact zeros without the same continuous shrinkage; selection can be more abrupt
Ridge-style shrinkage	Shrinks scores continuously without setting coordinates exactly to zero	Can stabilize or reduce weights but does not provide a sparse support by itself