Why Small Details Matter in Penalized Regression
Ridge and lasso regression modify least squares by adding a penalty to the fitting objective. That modification can improve prediction or encourage a simpler model, but it also means that familiar least-squares interpretations need care. A coefficient can change because a predictor was recorded in different units; a zero lasso estimate does not establish that a predictor has no relationship with the response; and a penalized coefficient is generally not the ordinary least squares (OLS) estimate.
The first safeguard is to write down the objective actually being minimized. This tutorial uses a linear model with an intercept and \(p\) predictors, and the following penalty conventions. The intercept is unpenalized unless explicitly stated otherwise.
These objectives define the estimates; the optimization does not require normally distributed errors. Usual assumptions about the response and errors matter for model interpretation and inference, but they do not change what objective is being minimized. Also, other references may place a different constant in front of the penalty. A numerical value of \(\lambda\) is meaningful only together with the stated convention.
Trap 1: Applying the Penalty to Predictors on Incomparable Scales
Ridge and lasso penalize coefficient sizes, not the size of a predictor's contribution to fitted values directly. Changing a predictor's units changes its coefficient and therefore changes the penalty, even though the underlying information has not changed. For this reason, predictors are commonly centered and scaled before penalization. The scaling rule must be stated, and coefficients can be transformed back to the original units after fitting.
Worked Example: A Change of Units Changes Ridge Shrinkage
Consider four observations with centered, mutually orthogonal predictor vectors \(x_1=(-1,-1,1,1)\) and \(z=(-1,1,-1,1)\). Suppose the response vector is \(y=2x_1+5z\). Fit ridge regression with \(\lambda=4\), first using \(x_1\) and \(x_2=10z\), and then using \(x_1\) and \(z\). Compare the estimates.
Method. For centered, orthogonal predictors, the ridge slope for predictor \(x_j\) is \(x_j^\mathsf{T}y/(x_j^\mathsf{T}x_j+\lambda)\). This follows by differentiating the ridge objective with respect to that slope: the derivative is \(-x_j^\mathsf{T}y+(x_j^\mathsf{T}x_j+\lambda)\beta_j\), which is zero at the minimizer.
First, \(x_1^\mathsf{T}x_1=4\), \(z^\mathsf{T}z=4\), and \(x_1^\mathsf{T}z=0\). Thus \(x_1^\mathsf{T}y=2(4)+5(0)=8\). With \(x_2=10z\), \(x_2^\mathsf{T}x_2=100(4)=400\), and \(x_2^\mathsf{T}y=10z^\mathsf{T}(2x_1+5z)=10(20)=200\).
The fitted coefficient on \(z\), expressed in units of \(z\), is \(10\hat\beta_2=500/101\approx4.9505\). Now fit using \(z\) directly instead of \(10z\). The estimate on \(z\) is \(z^\mathsf{T}y/(z^\mathsf{T}z+4)=20/8=2.5\). The two ridge fits therefore differ despite the predictors representing the same underlying quantity in different units.
Check. The unpenalized slopes are \(2\) on \(x_1\) and \(0.5\) on \(x_2=10z\), or \(5\) on \(z\). Each ridge estimate is closer to zero on its own coefficient scale. The amount of shrinkage differs because the penalty is not invariant to a change of units. The mistake to avoid is comparing or penalizing raw coefficients when predictor scales are substantially different.
Centering also matters for the intercept: when each predictor has mean zero, the intercept represents the fitted response at the predictor means. Scaling by a standard deviation or by another stated scale is a modeling choice; the key is to apply the same documented transformation throughout and to interpret coefficients on the correct scale.
Trap 2: Penalizing the Intercept by Accident
The intercept ordinarily represents the response level when predictors are at their reference values. Penalizing it can pull the whole fitted response toward zero, which is not the intended effect of shrinking slope coefficients. With centered predictors, the intercept and slopes separate cleanly in the squared-error term, making this mistake especially easy to detect.
Worked Example: Keeping the Intercept Outside the Penalty
Let \(x=(-1,0,1)\) and \(y=(34,36,38)\). Use ridge regression with \(\lambda=2\). Find the estimates when the intercept is correctly unpenalized, then calculate what happens if it is mistakenly included in the ridge penalty.
Correct fit. The response mean is \(\bar y=(34+36+38)/3=36\), and \(\bar x=0\). The OLS slope is \(\sum_i x_i(y_i-\bar y)/\sum_i x_i^2=[(-1)(-2)+0+(1)(2)]/[1+0+1]=4/2=2\). For ridge, the slope estimate is \(4/(2+2)=1\). Because the predictors are centered and the intercept is not penalized, the intercept remains \(\hat\beta_0=36\). The fitted values are \(35,36,37\), so the residuals are \(-1,0,1\) and the residual sum of squares is \(2\).
Mistaken fit. If the intercept is also penalized by \(\lambda\beta_0^2/2\), its part of the objective is \(\frac12\sum_i(y_i-\beta_0)^2+\frac{2}{2}\beta_0^2\). Differentiating gives \(3\beta_0-108+2\beta_0=0\), so \(\hat\beta_0=108/5=21.6\), not \(36\). The centered predictor remains orthogonal to the intercept, so the slope is still \(1\). The resulting fitted values are \(20.6,21.6,22.6\), and the residuals are \(13.4,14.4,15.4\). Their squared sum is \(179.56+207.36+237.16=624.08\).
Check. The correct ridge fit has residual sum of squares \(2\); mistakenly penalizing the intercept shifts all fitted values far below the response level. The mistake to avoid is silently including \(\beta_0\) in a penalty that is intended for slopes only. Always inspect the objective before differentiating.
Trap 3: Confusing Shrinkage with Selection
Ridge shrinks slope estimates toward zero, but for a positive penalty it does not generally set them exactly to zero. Lasso can set some estimates exactly to zero, so it can produce a sparse fitted model. That distinction does not mean that a nonzero ridge coefficient is selected in the same way as a lasso coefficient, nor that a zero lasso coefficient proves the corresponding predictor is irrelevant in every model or population.
For orthogonal, centered predictors with squared lengths \(S_j=x_j^\mathsf{T}x_j\), the lasso estimate for one coefficient is \(\hat\beta_j=\operatorname{sign}(x_j^\mathsf{T}y)(|x_j^\mathsf{T}y|-\lambda)_+/S_j\), where \(a_+=\max(a,0)\). This is the soft-thresholding rule. To see why, hold the other coefficients fixed and write the part of the objective involving \(\beta_j\) as \(\frac12 S_j\beta_j^2-(x_j^\mathsf{T}r)\beta_j+\lambda|\beta_j|\), up to a constant, where \(r\) is the response after removing the other predictors. For a positive coefficient, setting the derivative to zero gives \(S_j\beta_j-x_j^\mathsf{T}r+\lambda=0\); for a negative coefficient it gives \(S_j\beta_j-x_j^\mathsf{T}r-\lambda=0\). At zero, the minimum condition is \(|x_j^\mathsf{T}r|\leq\lambda\). Together these conditions give soft-thresholding. With correlated predictors, the coordinates interact, so this separate formula cannot simply be applied to each raw predictor as if the others had no effect.
Worked Example: Interpreting a Zero Lasso Coefficient
Use centered orthogonal predictors \(x_1=(-1,-1,1,1)\) and \(x_2=(-1,1,-1,1)\), and response \(y=3x_1+0.75x_2\). Both predictor squared lengths are \(4\). Fit lasso with \(\lambda=4\), and compare the result with OLS.
Method. The OLS slopes are \(x_j^\mathsf{T}y/(x_j^\mathsf{T}x_j)\). Orthogonality gives \(x_1^\mathsf{T}y=3(4)=12\) and \(x_2^\mathsf{T}y=0.75(4)=3\). Therefore the OLS slopes are \(12/4=3\) and \(3/4=0.75\).
Apply the lasso soft-thresholding rule. For \(x_1\), the estimate is \((12-4)/4=2\). For \(x_2\), \((3-4)_+=0\), so the estimate is zero. The lasso fitted response is \(2x_1\). Its residual is \(y-2x_1=x_1+0.75x_2\); the residual sum of squares is \(4+0.75^2(4)=4+2.25=6.25\), since the predictors are orthogonal.
To check the zero-coordinate condition directly, \(x_2^\mathsf{T}(y-2x_1)=x_2^\mathsf{T}(x_1+0.75x_2)=3\), and \(|3|\leq4=\lambda\). The nonzero coordinate has \(x_1^\mathsf{T}(y-2x_1)=4=\lambda\), satisfying its positive-coefficient condition. Thus the lasso solution is consistent with the minimization conditions.
Interpretation and check. The OLS coefficient on \(x_2\) is \(0.75\), but the lasso coefficient is zero at this penalty value. The zero says that \(x_2\) is excluded from this particular penalized fit; it does not say its OLS association was zero. The mistake to avoid is translating “lasso coefficient equals zero” into “this predictor can never matter.”
Trap 4: Reporting Penalized Estimates as OLS Estimates
A penalized estimate is the minimizer of an objective that includes a penalty. OLS minimizes residual sum of squares alone. Except when the penalty is zero or in special cases where it does not alter the solution, these are different optimization problems and generally yield different coefficients. Penalized coefficients also do not automatically inherit the usual OLS standard errors or inferential interpretation.
A useful verification is to substitute the proposed coefficients into the stated objective or the corresponding optimality conditions. For ridge, the slope condition in matrix form is \((X^\mathsf{T}X+\lambda I)\hat\beta=X^\mathsf{T}y\) when the intercept has been handled separately and the predictors are centered. The added \(\lambda I\) is exactly what distinguishes the ridge normal equations from the OLS normal equations. For lasso, the absolute-value penalty is not differentiable at zero, so the zero-coefficient condition must be checked using the threshold or the corresponding subgradient condition rather than ordinary differentiation at zero.
Record the loss, penalty, penalty constant, and whether the intercept is excluded.
Confirm whether predictors were centered or scaled, and identify the scale on which each coefficient is penalized.
Describe ridge estimates as shrunk; describe zero lasso estimates as exclusions from that fitted linear predictor, not universal evidence of irrelevance.
Use the stated normal equations, soft-thresholding rule when its orthogonality conditions apply, or lasso optimality conditions.
Common Exam Traps
- Changing units without accounting for the penalty. A coefficient's numerical value changes when a predictor is rescaled, and so can its penalized estimate.
- Including the intercept by habit. Check the objective rather than assuming every coefficient is penalized.
- Calling every small coefficient a selected-out predictor. Ridge typically shrinks without exact zeros; lasso can produce exact zeros.
- Interpreting a lasso zero too broadly. It describes the fitted model at a particular penalty and data set, not all possible models or the underlying truth.
- Using an OLS formula without the penalty term. A penalized estimate must satisfy the penalized objective's conditions.
A reliable solution makes the modeling choices explicit before doing arithmetic: state how predictors are scaled, write whether the intercept is penalized, and name whether a coefficient is OLS, ridge, or lasso. These checks are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, identify the relevant trap and check the estimate against the stated objective.
- A centered predictor \(x\) has \(x^\mathsf{T}x=8\) and \(x^\mathsf{T}y=24\). Under ridge with \(\lambda=4\), find the slope and compare it with OLS. (Answer: ridge \(=24/(8+4)=2\); OLS \(=24/8=3\), so ridge is shrunk.)
- A predictor is replaced by twice its value, with no other change to the data. Does a ridge penalty on the raw coefficient necessarily produce the same fitted values at the same \(\lambda\)? (Answer: No; the coefficient scale and penalty change, so the fitted values generally change.)
- For an orthogonal predictor with \(x^\mathsf{T}x=5\), \(x^\mathsf{T}y=3\), and lasso \(\lambda=3\), find its lasso coefficient and state the zero condition. (Answer: \(0\); the threshold condition is \(|x^\mathsf{T}y|=3\leq\lambda=3\).)
- For three observations with responses \(8,10,12\) and no predictors, use the objective \(\frac12\sum_i(y_i-\beta_0)^2+\frac{2}{2}\beta_0^2\) to find the intercept. Compare it with the unpenalized intercept. (Answer: penalized intercept \(=30/(3+2)=6\); unpenalized intercept \(=30/3=10\).)
- For centered orthogonal predictors with \(x_1^\mathsf{T}x_1=x_2^\mathsf{T}x_2=4\), \(x_1^\mathsf{T}y=16\), and \(x_2^\mathsf{T}y=2\), find both lasso slopes at \(\lambda=4\). (Answer: \(\hat\beta_1=(16-4)/4=3\), \(\hat\beta_2=0\) because \(2\leq4\); the second predictor is excluded from this fit.)