Why Orthogonality Makes Lasso Calculations Simpler
The lasso estimates regression coefficients by balancing fit to the data against an absolute-value penalty. In general, changing one coefficient can affect how the other coefficients should be estimated. When the predictor columns are orthogonal, that interaction disappears: each coefficient can be calculated separately from its ordinary least-squares value. The result is a precise shrinkage rule called soft thresholding.
This rule is useful both for hand calculations and for understanding what the penalty does. A coefficient is pulled toward zero, and it becomes exactly zero once its ordinary least-squares magnitude is no greater than a threshold. The threshold depends not only on the penalty parameter but also on the scale of the predictor and the scaling convention used for the objective.
The centered formulation removes the intercept from the displayed calculations. Equivalently, one may include an intercept, leave it unpenalized, and center the response and predictors before estimating the slopes. The result requires the predictor columns to be orthogonal in the data used for fitting. If columns are correlated, the objective does not separate into independent one-coefficient problems, so this formula cannot generally be applied coefficient by coefficient.
The definition uses a specific penalty convention: the squared-error term is divided by \(2n\). Some texts or software use a different multiplier on the squared-error term or penalty. The numerical value of the threshold changes with that convention, so the objective must be read before using a formula.
Deriving the Soft-Thresholding Rule
Write the loss as a sum of contributions from individual predictors. Orthogonality makes the cross-products between distinct predictor columns zero. After dropping terms that do not depend on \(\beta_j\), the part of the objective involving coefficient \(j\) is
To see why this expression has its minimum at the soft-threshold value, consider the three possible regions for \(\beta_j\). If \(\beta_j>0\), then \(|\beta_j|=\beta_j\), and differentiating the expression gives \(q_j(\beta_j-b_j)+\lambda\). Setting this derivative equal to zero gives the candidate \(\beta_j=b_j-\lambda/q_j\). It is positive exactly when \(b_j>\lambda/q_j\).
If \(\beta_j<0\), then \(|\beta_j|=-\beta_j\), and the derivative is \(q_j(\beta_j-b_j)-\lambda\). Setting it to zero gives \(\beta_j=b_j+\lambda/q_j\), which is negative exactly when \(b_j<-\lambda/q_j\). In the remaining case, \(-\lambda/q_j\le b_j\le\lambda/q_j\), neither nonzero candidate lies in its required region. The minimum is then at zero. The objective is convex, so these region-by-region minima give its global minimum.
Combining the cases yields
Equivalently, the magnitude is reduced by \(\lambda/q_j\) when the coefficient survives, while its sign is preserved. At the boundary \(|b_j|=\lambda/q_j\), the estimate is zero. If the columns are scaled so that \(X_j^\mathsf{T}X_j/n=1\), then \(q_j=1\) and the threshold is simply \(\lambda\). Orthogonality alone does not imply this scaling.
Worked Example: Thresholding Standardized Predictors
Suppose three centered, mutually orthogonal predictor columns have \(q_j=1\), and their ordinary least-squares coefficients are \(b_1=2.4\), \(b_2=-1.1\), and \(b_3=0.5\). Using the objective \(\frac{1}{2n}\lVert y-X\beta\rVert^2+\lambda\sum_j|\beta_j|\), calculate all lasso coefficients at \(\lambda=0.6\), then recalculate them at \(\lambda=0.25\).
Method. Since each \(q_j=1\), the threshold for each coefficient is \(\lambda/q_j=\lambda\).
At \(\lambda=0.6\), the threshold is \(0.6\). For the first coefficient, \(2.4>0.6\), so \(\hat\beta_1=2.4-0.6=1.8\). For the second, \(|-1.1|=1.1>0.6\), so its negative sign is retained and \(\hat\beta_2=-1.1+0.6=-0.5\). For the third, \(|0.5|\le0.6\), so \(\hat\beta_3=0\). Thus the estimates are \((1.8,-0.5,0)\).
At \(\lambda=0.25\), the threshold is \(0.25\). The first estimate is \(2.4-0.25=2.15\). The second is \(-1.1+0.25=-0.85\). The third is \(0.5-0.25=0.25\). The estimates are \((2.15,-0.85,0.25)\).
Check. At both penalty values, each surviving coefficient has its original sign and a smaller magnitude than its ordinary least-squares coefficient. The third coefficient is zero at the larger penalty but nonzero at the smaller one. The common mistake avoided here is subtracting the threshold from a negative coefficient without preserving its sign.
Predictor Scale Determines the Threshold
For a predictor with \(q_j\ne1\), the threshold is \(\lambda/q_j\), not \(\lambda\). Holding \(\lambda\) fixed, a larger \(q_j\) gives a smaller threshold in coefficient units. Thus, coefficients measured against predictors with different scales should not be compared using the standardized-predictor rule unless the columns have actually been scaled to make their \(q_j\) values equal to one.
The threshold at which coefficient \(j\) first becomes zero as \(\lambda\) increases is \(q_j|b_j|\). Indeed, the coefficient is nonzero when \(|b_j|>\lambda/q_j\), or equivalently when \(\lambda<q_j|b_j|\). At equality it is zero. This identifies the zeroing point for an individual coefficient under orthogonality; it does not require comparing the fit of different models.
Worked Example: Unequal Predictor Norms
Two centered, orthogonal predictors have \(q_1=2\) and \(q_2=0.5\), with ordinary least-squares coefficients \(b_1=1.5\) and \(b_2=-2\). Under the stated \(\frac{1}{2n}\)-scaled objective, calculate the estimates at \(\lambda=0.5\), and find the penalty at which each coefficient becomes zero.
Method. Calculate each predictor-specific threshold using \(\lambda/q_j\). For predictor 1, the threshold is \(0.5/2=0.25\). Since \(1.5>0.25\), its estimate is \(1.5-0.25=1.25\). For predictor 2, the threshold is \(0.5/0.5=1\). Since \(|-2|=2>1\), its estimate is \(-2+1=-1\). Thus \((\hat\beta_1,\hat\beta_2)=(1.25,-1)\).
The zeroing penalty is \(q_j|b_j|\). For predictor 1 it is \(2(1.5)=3\). For predictor 2 it is \(0.5(2)=1\). Therefore, as the penalty increases, predictor 2 reaches zero at \(\lambda=1\), while predictor 1 reaches zero at \(\lambda=3\). At those exact values the corresponding coefficient is zero.
Check. At \(\lambda=0.5\), both coefficients are nonzero because \(0.5<1\) and \(0.5<3\). The smaller \(q_2\) produces a larger threshold in coefficient units, which explains why predictor 2 is shrunk more. The common mistake avoided is assigning both predictors the same threshold despite their different norms.
Calculating from Cross-Products
Sometimes a problem supplies the predictor-response cross-products rather than the ordinary least-squares coefficients. For orthogonal columns, calculate \(q_j=X_j^\mathsf{T}X_j/n\) and \(b_j=X_j^\mathsf{T}y/(X_j^\mathsf{T}X_j)\) first. Then apply the soft-thresholding formula. This order keeps the predictor norm and the penalty convention visible.
Worked Example: From Sufficient Statistics to Lasso Estimates
A centered regression has \(n=10\) observations and three mutually orthogonal predictor columns. The supplied cross-products are \(X_1^\mathsf{T}X_1=20\), \(X_2^\mathsf{T}X_2=5\), \(X_3^\mathsf{T}X_3=10\), and \(X_1^\mathsf{T}y=30\), \(X_2^\mathsf{T}y=-10\), \(X_3^\mathsf{T}y=4\). Under the objective \(\frac{1}{2n}\lVert y-X\beta\rVert^2+0.5\sum_j|\beta_j|\), calculate the lasso coefficients and identify which are zero.
Method. First calculate the ordinary least-squares coefficients from the cross-products: \(b_1=30/20=1.5\), \(b_2=-10/5=-2\), and \(b_3=4/10=0.4\). Next calculate the norm factors: \(q_1=20/10=2\), \(q_2=5/10=0.5\), and \(q_3=10/10=1\).
The thresholds at \(\lambda=0.5\) are \(0.5/2=0.25\), \(0.5/0.5=1\), and \(0.5/1=0.5\). For predictor 1, \(1.5>0.25\), giving \(\hat\beta_1=1.5-0.25=1.25\). For predictor 2, \(2>1\), giving \(\hat\beta_2=-2+1=-1\). For predictor 3, \(|0.4|\le0.5\), giving \(\hat\beta_3=0\). The fitted lasso coefficient vector is \((1.25,-1,0)\).
Check. The ordinary least-squares coefficient for predictor 3 has magnitude \(0.4\), below its threshold of \(0.5\), so it must be zero. The other two magnitudes exceed their respective thresholds. The common mistake avoided is treating \(X_j^\mathsf{T}X_j\) as \(q_j\): \(q_j\) is the cross-product divided by \(n\) under this definition.
Penalty Scaling and Common Traps
A formula that is correct for one objective can be wrong for another if its penalty parameter is carried over unchanged. For example, if the objective is written as \(\frac{1}{n}\lVert y-X\beta\rVert^2+\lambda\sum_j|\beta_j|\), its squared-error term is twice the one used here. Dividing that entire objective by two gives \(\frac{1}{2n}\lVert y-X\beta\rVert^2+\frac{\lambda}{2}\sum_j|\beta_j|\). Therefore, under that alternative convention, the threshold is \(\lambda/(2q_j)\), not \(\lambda/q_j\).
Another trap is to use the standardized threshold \(\lambda\) even when \(q_j\ne1\). A third is to shrink a negative estimate in the wrong direction: a negative surviving coefficient moves toward zero by adding \(\lambda/q_j\), not by subtracting it. In every calculation, state the objective, find \(q_j\), and compare \(|b_j|\) with \(\lambda/q_j\) before assigning a nonzero estimate.
Check Your Understanding
For each problem, use the objective \(\frac{1}{2n}\lVert y-X\beta\rVert^2+\lambda\sum_j|\beta_j|\), unless another convention is explicitly stated.
- Two orthogonal predictors have \(q_1=q_2=1\) and ordinary least-squares coefficients \(1.8\) and \(-0.7\). Find their lasso coefficients at \(\lambda=0.4\). (Answer: \(1.4\) and \(-0.3\))
- An orthogonal predictor has \(q=0.8\), ordinary least-squares coefficient \(b=1.2\), and penalty \(\lambda=0.4\). Find its lasso coefficient. (Answer: \(0.7\))
- For an orthogonal predictor with \(q=1.5\) and \(b=-0.8\), find the penalty value at which its lasso coefficient becomes zero. (Answer: \(\lambda=1.2\), with the coefficient zero at equality)
- Two orthogonal predictors have \((q_1,b_1)=(2,0.6)\) and \((q_2,b_2)=(0.5,-1.5)\). At \(\lambda=0.5\), calculate both lasso coefficients and identify the zero coefficient, if any. (Answer: \(\hat\beta_1=0.35\), \(\hat\beta_2=-0.5\); neither is zero)
- An orthogonal predictor has \(q=2\), \(b=1\), and the objective is instead \(\frac{1}{n}\lVert y-X\beta\rVert^2+0.6|\beta|\). Find the lasso coefficient under this stated convention. (Answer: \(0.85\))