Lasso Soft Thresholding for Orthogonal Predictors

Why Orthogonality Makes the Lasso Transparent

The lasso combines a squared-error fit with a penalty on the sum of the absolute values of the coefficients. Because the absolute-value penalty can produce exact zeros, it can yield a model using fewer predictors. In general, however, changing one lasso coefficient can affect the others through the predictors’ relationships. When centered predictor columns are orthogonal, that interaction disappears: the full minimization separates into independent one-coefficient problems.

This special case reveals the lasso’s soft-thresholding rule. Each least-squares coefficient is pulled toward zero by a threshold determined by the penalty and the length of its predictor column. A coefficient whose magnitude does not exceed that threshold is set exactly to zero.

Model, Centering, and the Penalty Convention

Suppose there are \(n\) observations and \(p\) predictors. Let \(y\) be the response vector after centering, and let \(x_j\) denote the centered column for predictor \(j\). Assume every column has positive squared length \(d_j=x_j^\top x_j>0\), and the columns are mutually orthogonal: \(x_j^\top x_k=0\) for \(j\ne k\). The intercept is handled separately and is not penalized. With centered \(y\) and centered predictors, its fitted value is the response mean.

For clarity, this tutorial uses the following lasso objective. The factor \(1/2\) in the squared-error term is part of the convention, so the numerical value of the threshold depends on it.

\underset{\beta_1,\ldots,\beta_p}{\operatorname{minimize}} \quad \frac{1}{2}\left\|y-\sum_{j=1}^{p}x_j\beta_j\right\|^2 +\lambda\sum_{j=1}^{p}|\beta_j|, \qquad \lambda\ge 0.

The first term rewards fitting the data; the second penalizes large absolute coefficients. Increasing \(\lambda\) increases the cost of nonzero coefficients. The computation of the minimizing coefficients under this specified design does not require a normality assumption.

Definition: For a real number \(z\) and threshold \(t\ge 0\), the soft-thresholding function is \(S(z,t)=\operatorname{sign}(z)(|z|-t)_+\), where \((a)_+=\max(a,0)\). Equivalently, \(S(z,t)=z-t\) if \(z>t\), \(S(z,t)=0\) if \(|z|\le t\), and \(S(z,t)=z+t\) if \(z<-t\).

Deriving the Soft-Thresholding Formula

Let \(\widehat{\beta}^{\,\mathrm{LS}}_j\) be the least-squares coefficient for predictor \(j\) in this centered model. Orthogonality gives \(\widehat{\beta}^{\,\mathrm{LS}}_j=x_j^\top y/d_j\): the least-squares problem separates in the same way as the penalized problem, but without the absolute-value terms.

To see why the lasso also separates, expand the squared-error term. Every cross-product \(x_j^\top x_k\) for \(j\ne k\) is zero. The terms involving a particular coefficient \(\beta_j\), up to constants that do not depend on that coefficient, are therefore

\frac{d_j}{2}\beta_j^2-(x_j^\top y)\beta_j+\lambda|\beta_j| = \frac{d_j}{2}\bigl(\beta_j-\widehat{\beta}^{\,\mathrm{LS}}_j\bigr)^2 +\lambda|\beta_j|+\text{constant}.

Thus, for each \(j\), the task is to minimize \(g(\beta)=\frac{d_j}{2}(\beta-z)^2+\lambda|\beta|\), where \(z=\widehat{\beta}^{\,\mathrm{LS}}_j\). Away from zero, differentiate separately on the positive and negative sides. For \(\beta>0\), the stationary point is \(\beta=z-\lambda/d_j\), which is in the positive region only when \(z>\lambda/d_j\). For \(\beta<0\), the stationary point is \(\beta=z+\lambda/d_j\), which is in the negative region only when \(z<-\lambda/d_j\).

At zero, the left and right derivatives are \(-d_jz-\lambda\) and \(-d_jz+\lambda\), respectively. Zero minimizes the function when the left derivative is at most zero and the right derivative is at least zero. Those two inequalities are equivalent to \(|d_jz|\le\lambda\), or \(|z|\le\lambda/d_j\). Since \(d_j>0\), the squared term is strictly convex, so these cases identify the unique minimum.

Theorem: With centered, mutually orthogonal predictor columns and the objective stated above, the lasso coefficient for predictor \(j\) is \(\widehat{\beta}^{\,\mathrm{lasso}}_j=S(\widehat{\beta}^{\,\mathrm{LS}}_j,\lambda/d_j)\). In particular, it is exactly zero if and only if \(|\widehat{\beta}^{\,\mathrm{LS}}_j|\le\lambda/d_j\).

This is called soft, rather than hard, thresholding because a coefficient outside the zero region is not left unchanged: its magnitude is reduced by the threshold. Its sign is retained, and its magnitude decreases by \(\lambda/d_j\).

Worked Example: A Coefficient at the Zero Threshold

Consider four hypothetical observations and two centered predictor columns, \(x_1=(1,-1,1,-1)^\top\) and \(x_2=(1,1,-1,-1)^\top\). Both have squared length 4, and \(x_1^\top x_2=0\). Let the centered response be \(y=2x_1-0.5x_2+(1,-1,-1,1)^\top\). The third vector is also centered and orthogonal to both predictors, so \(x_1^\top y=8\) and \(x_2^\top y=-2\). The least-squares coefficients are \(8/4=2\) and \(-2/4=-0.5\).

Take \(\lambda=2\). For each predictor the threshold is \(\lambda/d_j=2/4=0.5\). Therefore \(\widehat{\beta}^{\,\mathrm{lasso}}_1=S(2,0.5)=2-0.5=1.5\), while \(\widehat{\beta}^{\,\mathrm{lasso}}_2=S(-0.5,0.5)=0\). The second coefficient is exactly zero at the threshold, because its least-squares magnitude equals \(0.5\).

The least-squares residual is the third vector, whose squared length is 4. At the lasso fit, the residual is \(0.5x_1-0.5x_2+(1,-1,-1,1)^\top\). Orthogonality gives its sum of squared errors as \(4(0.5^2+0.5^2+1^2)=6\). Shrinking coefficients has increased the squared error, as expected, while the penalty can reward the simpler fit.

How the Penalty Changes the Coefficients

For a fixed set of orthogonal columns, increasing \(\lambda\) widens each coefficient’s zero region. A coefficient becomes zero once \(\lambda\) reaches \(d_j|\widehat{\beta}^{\,\mathrm{LS}}_j|\); above that value it remains zero. A coefficient that is nonzero has the same sign as its least-squares counterpart, and its magnitude falls linearly with \(\lambda\) until it reaches zero.

Worked Example: Tracking a Short Thresholding Path

Use the centered, mutually orthogonal vectors \(x_1=(1,-1,1,-1)^\top\), \(x_2=(1,1,-1,-1)^\top\), and \(x_3=(1,-1,-1,1)^\top\), each with squared length 4. Let the hypothetical response be \(y=1.2x_1-0.8x_2+0.4x_3=(0.8,-2.4,1.6,0)^\top\). The third vector is orthogonal to the two predictors in the model, so the least-squares coefficients for \(x_1,x_2\) are \(1.2\) and \(-0.8\). Each threshold is \(\lambda/4\).

At \(\lambda=1.6\), the threshold is \(0.4\), giving \(\widehat{\beta}_1=S(1.2,0.4)=0.8\) and \(\widehat{\beta}_2=S(-0.8,0.4)=-0.4\).

At \(\lambda=3.2\), the threshold is \(0.8\), so \(\widehat{\beta}_1=S(1.2,0.8)=0.4\) and \(\widehat{\beta}_2=S(-0.8,0.8)=0\). At \(\lambda=4.8\), the threshold is \(1.2\), and both coefficients are zero. The zeroing values are \(4(1.2)=4.8\) for the first coefficient and \(4(0.8)=3.2\) for the second.

At \(\lambda=3.2\), the fitted coefficients are \(0.4\) and \(0\), and the residual is \(0.8x_1-0.8x_2+0.4x_3\). Its sum of squared errors is \(4(0.8^2+0.8^2+0.4^2)=5.76\). This calculation uses orthogonality to add the squared contributions.

Why Predictor Length and Scaling Matter

The threshold is \(\lambda/d_j\), not just \(\lambda\), when the predictor columns have different squared lengths. A larger \(d_j\) means a smaller threshold for the same \(\lambda\). Consequently, the amount of shrinkage depends both on the penalty and on how each predictor is measured. This is one reason predictor scaling matters before penalization: changing units changes column lengths and therefore changes the numerical thresholds.

Worked Example: Orthogonal Predictors With Unequal Lengths

Consider the centered hypothetical columns \(x_1=(1,-1,1,-1)^\top\) and \(x_2=(2,2,-2,-2)^\top\). They are orthogonal, with squared lengths \(d_1=4\) and \(d_2=16\). Let \(y=1.5x_1+0.75x_2+0.5(1,-1,-1,1)^\top=(3.5,-0.5,-0.5,-2.5)^\top\). The final vector is centered and orthogonal to both columns. Thus the least-squares coefficients are \(1.5\) and \(0.75\).

For \(\lambda=4\), the thresholds are \(4/4=1\) for \(x_1\) and \(4/16=0.25\) for \(x_2\). Hence \(\widehat{\beta}^{\,\mathrm{lasso}}_1=S(1.5,1)=0.5\), and \(\widehat{\beta}^{\,\mathrm{lasso}}_2=S(0.75,0.25)=0.5\). Both coefficients shrink, but by different amounts because their columns have different lengths. If the predictors were rescaled, these numerical thresholds would change; comparisons of penalized coefficients require attention to the scaling used.

What the Formula Does—and Does Not—Say

Soft thresholding is an exact coefficient formula for the stated orthogonal design and penalty convention. With correlated predictors, the squared-error expansion contains cross-terms, so minimizing each coefficient separately by this formula is generally not valid. The formula also depends on how the loss is scaled: if an objective divides the sum of squared errors by \(n\), its penalty parameter must be interpreted using that objective’s own normalization.

A zero lasso coefficient means the fitted lasso model assigns that predictor a coefficient of zero at the chosen penalty, given the other modeling choices and the data. It does not by itself establish that the predictor has no relationship with the response or that its population effect is zero. Nor does this algebraic threshold provide ordinary least-squares inference for a coefficient chosen by penalization.

Common Mistakes

Subtracting \(\lambda\) directly from every coefficient. The threshold under this objective is \(\lambda/d_j\), which depends on the predictor’s squared length. Use \(\lambda/n\) only when \(d_j=n\), such as for centered columns scaled to that squared length.
Calling the rule hard thresholding. Coefficients outside the zero region are still shrunk. For example, \(S(2,0.5)=1.5\), not 2.
Assuming a zero coefficient proves no association. It only describes the penalized fit under the selected penalty and model setup; it is not a statement of causal absence or a test of a population null hypothesis.
Applying the separate formula to correlated predictors. Orthogonality is what removes the cross-terms and makes the minimization separable. Without it, this derivation does not give the general solution.

Key takeaway: For centered orthogonal predictors under the objective used here, lasso applies \(S(\widehat{\beta}^{\,\mathrm{LS}}_j,\lambda/d_j)\) to each least-squares coefficient. The coefficient is zero precisely when its least-squares magnitude is no greater than \(\lambda/d_j\); otherwise, its magnitude is reduced by that threshold.

Check Your Understanding

Use the objective and notation defined above.

A centered orthogonal predictor has \(d_j=9\), least-squares coefficient \(1.2\), and \(\lambda=5.4\). What is its lasso coefficient?
For \(d_j=4\) and \(\lambda=3\), what range of least-squares coefficients is set to zero?
For a negative least-squares coefficient whose magnitude exceeds the threshold, how does soft thresholding change its value?
Why can the one-coefficient derivation not be applied separately to each predictor when predictor columns are correlated?
If two orthogonal columns have different squared lengths, which one has the smaller threshold at a common positive \(\lambda\), and why?