Deriving Soft-Thresholded Sparse CCA Updates

Why the Conditional Update Has a Threshold

In sparse canonical correlation analysis (CCA), a weight vector combines variables from one data block into a score. Sparsity is useful when a block has many variables: it can produce a score that uses only a subset, making the fitted direction easier to inspect. In an alternating algorithm, one weight vector is updated while the direction for the other block is held fixed. The resulting one-block problem has a simple form: maximize alignment with a fixed vector while penalizing the L1 norm.

The update is not just a rule for deleting small coefficients. It is the optimizer of a particular conditional objective under an L2 constraint. The objective, the constraint, and the scale of the data determine the meaning of the threshold.

Definition: For a scalar $z$ and threshold $\lambda\geq 0$, the soft-thresholding operator is $$ S_\lambda(z)=\operatorname{sign}(z)(|z|-\lambda)_+,\qquad (a)_+=\max(a,0). $$ Applied to a vector, it acts coordinate by coordinate: $S_\lambda(c)_j=S_\lambda(c_j)$. It preserves the sign of a surviving coordinate, reduces its magnitude by $\lambda$, and sets it to zero when $|c_j|\leq\lambda$.

The Conditional Sparse CCA Problem

Let $X$ and $Y$ denote two centered data blocks measured on the same $n$ observations. Write $C$ for their cross-product or cross-covariance matrix, with its scale specified by the analyst. For example, $C=X^\mathsf{T}Y/n$ is a cross-covariance matrix when the columns are centered. If $v$ is held fixed, define $c=Cv$. The vector $c$ records the linear alignment of each variable in the $X$ block with the direction in the $Y$ block represented by $v$.

The conditional update for $u$ considered here maximizes the penalized alignment over the closed unit L2 ball:

\max_{\|u\|_2\leq 1}\left\{u^\mathsf{T}c-\lambda_u\|u\|_1\right\}, \qquad c=Cv,\quad \lambda_u\geq 0.

The opposite-block direction $v$, the matrix $C$, and the nonnegative penalty $\lambda_u$ are fixed during this update. The same reasoning gives the update for $v$ when $u$ is fixed, using $C^\mathsf{T}u$ and its own penalty. This conditional problem is the one-step objective; it does not by itself guarantee that alternating updates find a global optimum of the joint sparse CCA problem.

The L2-ball constraint is important. It allows the solution to have norm less than one, including the zero vector if no coordinate has enough alignment to overcome the penalty. When a nonzero update is normalized to unit length, its direction is the soft-thresholded vector divided by its L2 norm.

Deriving the Update

Consider one coordinate of the objective. If $c_j$ is fixed, the coordinate contribution is $c_j u_j-\lambda_u|u_j|$. A coordinate aligned in sign with $c_j$ gains linear alignment; the L1 penalty reduces that gain by $\lambda_u$ per unit magnitude. If $|c_j|\leq\lambda_u$, this coordinate cannot contribute a positive gain. If $|c_j|>\lambda_u$, its positive gain per unit magnitude is $|c_j|-\lambda_u$, with the sign of $c_j$.

This coordinate reasoning can be made into a complete optimization argument. For each coordinate and every real $u_j$,

c_j u_j-\lambda_u|u_j|\leq S_{\lambda_u}(c_j)u_j.

Summing over coordinates and applying the Cauchy–Schwarz inequality gives

u^\mathsf{T}c-\lambda_u\|u\|_1 \leq S_{\lambda_u}(c)^\mathsf{T}u \leq \|S_{\lambda_u}(c)\|_2\|u\|_2 \leq \|S_{\lambda_u}(c)\|_2.

If $S_{\lambda_u}(c)\ne 0$, set $u=S_{\lambda_u}(c)/\|S_{\lambda_u}(c)\|_2$. The nonzero coordinates of this $u$ have the same signs as $c$, so the first inequality is an equality on those coordinates; the vectors in the Cauchy–Schwarz inequality point in the same direction, and $\|u\|_2=1$. Thus this choice attains the upper bound and is an optimizer. If the soft-thresholded vector is zero, the upper bound is zero and $u=0$ attains it.

Conditional update: If $S_{\lambda_u}(Cv)\ne 0$, an optimizer of the conditional problem is $$ u=\frac{S_{\lambda_u}(Cv)}{\|S_{\lambda_u}(Cv)\|_2}. $$ Its nonzero coordinates are exactly those for which $|(Cv)_j|>\lambda_u$. If $S_{\lambda_u}(Cv)=0$, the ball-constrained problem has maximum value zero, attained at $u=0$; there is no nonzero soft-thresholded direction to normalize.

The derivation also explains why the constraint and penalty cannot be left implicit. A unit-norm equality constraint excludes $u=0$, so the zero-vector case above would not apply. In addition, a different scaling of $C$ changes the scale of $Cv$, and therefore changes which coordinates exceed a fixed numerical threshold. The penalty must be interpreted relative to the scaling used in the objective.

Worked Example: Updating a Four-Variable Direction

Suppose a hypothetical conditional alignment vector is $c=(0.35,-0.20,0.50,0.08)$, and the threshold is $\lambda_u=0.15$. The corresponding soft-thresholded coordinates are

S_{0.15}(c)=(0.35-0.15,\;-(0.20-0.15),\;0.50-0.15,\;0) =(0.20,-0.05,0.35,0).

The last coordinate is zero because $0.08\leq0.15$. The norm of the thresholded vector is

\|S_{0.15}(c)\|_2 =\sqrt{0.20^2+(-0.05)^2+0.35^2} =\sqrt{0.165} \approx 0.4062.

Normalizing gives

u\approx(0.20/0.4062,\;-0.05/0.4062,\;0.35/0.4062,\;0) =(0.4924,-0.1231,0.8616,0).

The update uses three of the four variables. The surviving signs match those of $c$, and the normalized weights have L2 norm one, up to rounding.

How the Threshold Changes the Selected Variables

For fixed $c$, increasing $\lambda_u$ can only shrink the magnitudes of the soft-thresholded coordinates and can remove coordinates from the support. Here the support means the set of indices with nonzero weights. A coordinate enters the support exactly when its absolute conditional alignment exceeds the threshold. This is a statement about the conditional update, not a claim that a selected variable is uniquely important or causally related to the other block.

Worked Example: Comparing Three Thresholds

Take the hypothetical alignment vector $c=(0.18,-0.12,0.22)$. At $\lambda_u=0.10$, the thresholded vector is $(0.08,-0.02,0.12)$. Its norm and normalized update are

\sqrt{0.08^2+(-0.02)^2+0.12^2} =\sqrt{0.0212}\approx0.1456, \qquad u\approx(0.5494,-0.1374,0.8242).

At $\lambda_u=0.15$, the thresholded vector becomes $(0.03,0,0.07)$. Therefore

\sqrt{0.03^2+0.07^2} =\sqrt{0.0058}\approx0.07616, \qquad u\approx(0.3939,0,0.9191).

The second variable is now excluded because its absolute alignment, $0.12$, does not exceed $0.15$. The first and third variables remain, although their normalized weights change. At $\lambda_u=0.22$, every coordinate is at or below the threshold, so the thresholded vector is zero. The ball-constrained conditional problem then has no nonzero update direction.

Normalization is consequential. Before normalization, increasing the threshold reduces the surviving magnitudes. After normalization, however, the relative weights can change in either direction: a variable that survives may receive a larger share of the unit length because other coordinates have been removed. Therefore, comparing normalized weights across thresholds is not the same as comparing the unnormalized amounts of shrinkage.

Worked Example: Computing the Conditional Alignment

Suppose a hypothetical cross-covariance matrix and fixed unit direction are

C= \begin{pmatrix} 0.40&0.00\\ -0.10&0.30\\ 0.20&0.10 \end{pmatrix}, \qquad v= \begin{pmatrix}0.60\\0.80\end{pmatrix}.

The conditional alignment vector is obtained by multiplying each row of $C$ by $v$:

c=Cv= \begin{pmatrix} 0.40(0.60)+0.00(0.80)\\ -0.10(0.60)+0.30(0.80)\\ 0.20(0.60)+0.10(0.80) \end{pmatrix} = \begin{pmatrix}0.24\\0.18\\0.20\end{pmatrix}.

With $\lambda_u=0.15$, the thresholded vector is $(0.09,0.03,0.05)$, and

\sqrt{0.09^2+0.03^2+0.05^2} =\sqrt{0.0115}\approx0.1072, \qquad u\approx(0.8393,0.2797,0.4663).

All three coordinates survive because each entry of $c$ is greater than $0.15$ in absolute value. This calculation illustrates why the other-block direction must be held fixed while forming $c=Cv$: changing $v$ changes the alignment vector, and hence may change the selected variables even if the threshold stays constant.

Assumptions, Diagnostics, and Limitations

The update is an algebraic optimizer of the stated conditional objective; it does not require a Gaussian data model. Its use in a CCA analysis does require that the two blocks have matched observations and that the analyst has defined a meaningful, fixed scaling for their variables. Centering and any standardization should be chosen before interpreting thresholds, since changing units changes the numerical entries of $C$. During the update, $C$, $v$, and $\lambda_u$ must remain fixed.

Several checks help identify an update that is being misapplied:

Check the norm. If the thresholded vector is nonzero, the normalized update should have L2 norm one, apart from rounding.
Check the support and signs. A coordinate is zero when $|c_j|\leq\lambda_u$; a surviving coordinate has the sign of $c_j$.
Check for a zero update before dividing. If every $|c_j|\leq\lambda_u$, do not divide by a zero norm. The conditional ball-constrained optimizer can be $u=0$, but it cannot be represented as a normalized nonzero direction.
Check scale and numerical precision. Values close to the threshold can change status under small changes in scaling or rounding. Apply the threshold to the unrounded values when possible.
Check the alternating objective. The conditional update should maximize its stated one-block objective with the opposite direction fixed. But because the joint problem is nonconvex, successful conditional updates do not establish a global optimum; different initial directions can lead to different solutions.

A useful comparison is ridge shrinkage. In a simple L2-penalized linear update, coefficients are reduced continuously but are generally not set exactly to zero. Soft thresholding differs because the L1 penalty creates an exact zero region, $|c_j|\leq\lambda_u$. Hard thresholding also creates exact zeros, but it deletes coordinates abruptly while leaving the magnitude of a retained coordinate unchanged before any subsequent normalization. Soft thresholding instead subtracts the threshold from the magnitude of each retained coordinate.

A constrained L1 formulation, in which the L1 norm is bounded rather than penalized, can also encourage sparse weights. Its trade-off is set by a constraint value rather than by the fixed $\lambda_u$ used in this conditional problem; the two formulations should not be treated as the same numerical update without establishing their relationship. Finally, the support and weights depend on the fitted cross-block alignment and scaling. They are not, by themselves, evidence of a unique biological mechanism or of causal effects.

Common pitfall: Subtracting $\lambda_u$ from every coefficient, or normalizing $c$ before thresholding, is not the derived update. The correct order is to apply $S_{\lambda_u}$ coordinatewise to the conditional alignment vector $c=Cv$, then normalize the result only if it is nonzero.

Check Your Understanding

Answer each question, then compare with the concise solutions below.

If $c_j=-0.31$ and $\lambda_u=0.11$, what is the soft-thresholded coordinate?
For $c=(0.10,0.20)$ and $\lambda_u=0.10$, which coordinates survive, and is there a nonzero normalized update?
Why does a change of units in one block potentially alter the selected variables for the same numerical threshold?
What is the key difference between soft thresholding and ridge shrinkage in whether a coordinate becomes exactly zero?

Solutions. (1) The sign remains negative and the magnitude is reduced by $0.11$: $S_{0.11}(-0.31)=-(0.31-0.11)=-0.20$. (2) The thresholded vector is $(0,0.10)$, since equality to the threshold gives zero for the first coordinate. Its norm is $0.10$, so the normalized update is $(0,1)$. (3) Changing units changes entries of $C$, and hence of $Cv$; a fixed numerical threshold can then be crossed by a different set of coordinates. (4) Soft thresholding sets a coefficient to zero when its absolute conditional alignment is no greater than the threshold, whereas ridge shrinkage generally reduces coefficients without producing exact zeros.