Why Orthogonality Makes the Comparison Clear
Ridge and lasso both penalize large regression coefficients, but their penalties produce different behavior. Ridge pulls coefficients toward zero without usually removing them; lasso can set a coefficient exactly to zero. When predictor columns are orthogonal, each coefficient can be solved separately, so the distinction is especially transparent.
The comparison below uses the same least-squares fit and the same numerical penalty value for both methods. The penalty conventions matter: changing the scaling of the residual sum of squares changes the numerical meaning of the penalty parameter. We therefore state the objectives explicitly before calculating.
These definitions use the same residual-sum-of-squares multiplier, \(1/2\), and the same value of \(\lambda\) in the two penalty terms. This is a useful matched-penalty comparison, not a claim that the two methods must have the same prediction error or degree of shrinkage. If a problem instead divides the residual sum of squares by the sample size, its penalty parameter must be interpreted on that scale.
Coefficient Formulas and Their Derivation
Let \(\widehat{\beta}_{j,\mathrm{OLS}}\) denote the ordinary least-squares coefficient for predictor \(j\). Under orthogonality, it equals \(z_j/d_j\). The ridge and lasso coefficients can each be found from a one-variable minimization.
Here \((a)_+=\max(a,0)\), and \(\operatorname{sign}(z_j)\) is \(1\) for positive \(z_j\), \(-1\) for negative \(z_j\), and \(0\) when \(z_j=0\). The expression in the lasso formula is the soft-thresholding operation: it subtracts \(\lambda\) from the absolute cross-product if that cross-product exceeds \(\lambda\), and otherwise returns zero.
Orthogonality gives \(\lVert y-X\beta\rVert^2=y^\mathsf{T}y-\sum_j 2z_j\beta_j+\sum_j d_j\beta_j^2\). Each coefficient therefore contributes its own quadratic term, with no cross-products between coefficients.
For coefficient \(j\), the terms that depend on \(\beta_j\) are \(\frac{d_j}{2}\beta_j^2-z_j\beta_j+\frac{\lambda}{2}\beta_j^2\). Differentiating gives \((d_j+\lambda)\beta_j-z_j=0\), hence \(\widehat{\beta}_{j,\mathrm{ridge}}=z_j/(d_j+\lambda)\).
The relevant function is \(g(\beta_j)=\frac{d_j}{2}\beta_j^2-z_j\beta_j+\lambda|\beta_j|\). At a nonzero minimum, its derivative must be \(d_j\beta_j-z_j+\lambda\operatorname{sign}(\beta_j)=0\). A positive solution is \( (z_j-\lambda)/d_j\) when \(z_j>\lambda\); a negative solution is \( (z_j+\lambda)/d_j\) when \(z_j<-\lambda\). At zero, the subgradient condition is \(|z_j|\leq\lambda\). These cases give the soft-threshold formula.
The derivation also explains the practical difference. For finite \(\lambda\), ridge multiplies the least-squares coefficient by \(d_j/(d_j+\lambda)\), a positive factor when \(d_j>0\). Thus a nonzero least-squares coefficient remains nonzero. Lasso instead has an exact-zero condition: \(|z_j|\leq\lambda\). The threshold is on the cross-product \(z_j\), not directly on the least-squares coefficient.
Matched Penalties in Worked Examples
For each example, the predictors are centered and orthogonal, and the intercept is unpenalized. The supplied values \(d_j=x_j^\mathsf{T}x_j\) and \(z_j=x_j^\mathsf{T}y\) are sufficient for calculating the penalized coefficients. When all \(d_j\) are equal, both methods treat the predictors on the same scale.
Worked Example: Two Predictors and a Shared Penalty
A centered orthogonal design has two predictors with \(d_1=d_2=4\), and the cross-products are \(z_1=6\) and \(z_2=2\). Use \(\lambda=2\) for both stated objectives. Calculate the OLS, ridge, and lasso coefficients, then identify which predictor lasso removes.
Method and OLS calculation. For each predictor, \(\widehat{\beta}_{j,\mathrm{OLS}}=z_j/d_j\). Thus the OLS coefficients are \(6/4=1.5\) and \(2/4=0.5\).
Ridge calculation. The common denominator is \(d_j+\lambda=4+2=6\). Therefore \(\widehat{\beta}_{1,\mathrm{ridge}}=6/6=1\) and \(\widehat{\beta}_{2,\mathrm{ridge}}=2/6=1/3\). Both coefficients are reduced in magnitude from their OLS values, and neither is zero.
Lasso calculation. For predictor 1, \(|z_1|=6>2\), so its coefficient is \((6-2)/4=4/4=1\). For predictor 2, \(|z_2|=2\), so it is at the threshold and its coefficient is zero. The lasso estimates are \(1\) and \(0\), respectively.
Check and trap avoided. The lasso keeps predictor 1 and removes predictor 2 because the cross-products are respectively above and equal to the threshold. A tempting but incorrect approach is to threshold the OLS coefficient \(0.5\) at \(\lambda=2\); the penalty threshold in this objective is on \(z_2\), not on \(z_2/d_2\). The calculation \(|z_2|=2\leq2\) confirms the zero.
Worked Example: Mixed Signs and Unequal Signal Strengths
For three centered orthogonal predictors, suppose \(d_1=d_2=d_3=5\), with \(z_1=2\), \(z_2=-7\), and \(z_3=0.5\). With \(\lambda=2.5\), calculate both sets of penalized estimates and state which coefficients are zero.
Method and OLS calculation. Divide each cross-product by 5. The OLS estimates are \(2/5=0.4\), \(-7/5=-1.4\), and \(0.5/5=0.1\).
Ridge calculation. The ridge denominator is \(5+2.5=7.5\) for every predictor. The estimates are \(2/7.5=0.2667\), \(-7/7.5=-0.9333\), and \(0.5/7.5=0.0667\), rounded to four decimal places. Each retains its original sign and is smaller in magnitude than its OLS estimate.
Lasso calculation. For predictor 1, \(|2|<2.5\), so its estimate is zero. For predictor 2, the cross-product is negative and exceeds the threshold in absolute value: \((-7+2.5)/5=-4.5/5=-0.9\). For predictor 3, \(|0.5|<2.5\), so its estimate is zero. The lasso estimates are \(0,-0.9,0\).
Check and trap avoided. Only the second predictor survives lasso thresholding, and its estimate remains negative. Replacing \(z_2\) by \(|z_2|\) without restoring its sign would give the wrong direction. The numerator \(-7+2.5=-4.5\) preserves the sign, as required.
Worked Example: A Predictor Exactly at the Lasso Threshold
A centered orthogonal design has three predictors, each with squared norm \(d_j=9\). Their cross-products with the response are \(12,-3,\) and \(1.5\). Compare ridge and lasso at \(\lambda=3\), and verify the threshold decision for the second predictor.
Method and OLS calculation. The OLS estimates are \(12/9=4/3\), \(-3/9=-1/3\), and \(1.5/9=1/6\).
Ridge calculation. The denominator is \(9+3=12\). The ridge estimates are \(12/12=1\), \(-3/12=-0.25\), and \(1.5/12=0.125\). All three are nonzero because all three cross-products are nonzero and the ridge denominator is positive.
Lasso calculation. The first coefficient is \((12-3)/9=9/9=1\). For the second, \(|-3|=3=\lambda\), so the soft-thresholded numerator is \((-3+3)=0\), giving coefficient zero. For the third, \(|1.5|<3\), so its coefficient is also zero. Thus lasso estimates are \(1,0,0\).
Check and trap avoided. Equality at the threshold gives exactly zero; it does not leave a small coefficient. In particular, the second ridge estimate is \(-0.25\), while the second lasso estimate is zero. This contrast follows from the different penalties, not from a rounding choice.
How to Read the Comparison
The formulas make several comparisons immediate. With a common predictor norm \(d\), ridge shrinks every OLS coefficient by the same factor \(d/(d+\lambda)\). Lasso subtracts \(\lambda/d\) from the absolute OLS coefficient, stopping at zero. As a result, the lasso operation can create a sparse coefficient vector, while ridge generally retains all predictors with nonzero cross-products.
| Feature | Ridge under this objective | Lasso under this objective |
|---|---|---|
| Coordinate formula | \(z_j/(d_j+\lambda)\) | \(\operatorname{sign}(z_j)(|z_j|-\lambda)_+/d_j\) |
| Effect on a nonzero OLS coefficient | Multiplies it by \(d_j/(d_j+\lambda)\) | Reduces its magnitude by \(\lambda/d_j\), unless this reaches zero |
| Exact zero condition | Only when \(z_j=0\) for finite \(\lambda\) | \(|z_j|\leq\lambda\) |
| Sign when the estimate is nonzero | Same as \(z_j\) | Same as \(z_j\) |
If the design is orthogonal but predictor norms differ, the formulas still apply separately using each \(d_j\). The same \(\lambda\) can then have different effects across predictors because the scale \(d_j\) differs. In practice, predictors are often standardized before penalization to make comparisons more meaningful; coefficient formulas on the standardized scale should not be confused with coefficients on the original scale.
A common exam trap is to quote the unit-norm formula without checking the norm or the objective scaling. If \(d_j=1\), the lasso coefficient is \(\operatorname{sign}(z_j)(|z_j|-\lambda)_+\). If instead \(d_j=4\), the same objective gives \(\operatorname{sign}(z_j)(|z_j|-\lambda)_+/4\). A second trap is to silently switch from the objective with residual term \(\frac12\lVert y-X\beta\rVert^2\) to one with residual term \(\frac{1}{2n}\lVert y-X\beta\rVert^2\): that rescales the penalty comparison. Always read the full objective before applying a threshold formula.
Check Your Understanding
Use the objective conventions and formulas stated above. Unless specified otherwise, predictors are centered and orthogonal, and the intercept is unpenalized.
- A two-predictor design has \(d_1=d_2=4\), cross-products \(z_1=5,z_2=-1\), and \(\lambda=1\). Find the ridge and lasso estimates. (Answer: Ridge \((1,-0.2)\); lasso \((1,0)\).)
- Three orthogonal predictors each have \(d_j=6\), with cross-products \(3,-9,1\). At \(\lambda=3\), find both coefficient vectors and identify the lasso-selected predictor. (Answer: Ridge \((1/3,-1,1/9)\); lasso \((0,-1,0)\); only predictor 2 is selected.)
- For two orthogonal predictors with \(d_1=d_2=2\), let \(z_1=8,z_2=-4\) and \(\lambda=2\). Calculate the OLS and penalized estimates, and compare their magnitudes. (Answer: OLS \((4,-2)\); ridge \((2,-1)\); lasso \((3,-1)\).)
- The predictor norms are \(d_1=2,d_2=8\), and the cross-products are \(z_1=3,z_2=-12\). At \(\lambda=2\), calculate both methods and explain why a common threshold does not imply equal shrinkage. (Answer: Ridge \((0.75,-1.2)\); lasso \((0.5,-1.25)\); the predictors have different \(d_j\), so the division and shrinkage scales differ.)
- A single orthogonal predictor has \(d=9\), \(z=6\), and \(\lambda=4\). Find its OLS, ridge, and lasso estimates, and state whether lasso retains the predictor. (Answer: OLS \(2/3\); ridge \(6/13\); lasso \(2/9\); it is retained because \(|z|=6>4\).)