Why Lasso Penalizes a Least-Squares Fit
Ordinary least squares chooses regression coefficients to make the fitted responses close to the observed responses. Lasso modifies that objective by adding a cost for nonzero coefficient size. The resulting fit balances two goals: matching the training data and keeping the slopes small. This tutorial focuses on the objective itself and on interpreting its penalty parameter, not on methods for solving the optimization problem.
The balance matters in predictive work because a coefficient vector that fits the observed data extremely closely may be less useful than a slightly less exact fit with smaller coefficients. The penalty makes that trade-off explicit. The objective and conventions below are aligned with the SOA Exam PA (Predictive Analytics) syllabus.
The response model used to form the predictions is \( \hat y_i=\beta_0+\sum_{j=1}^p x_{ij}\beta_j \). The residual for observation \(i\) is \(y_i-\hat y_i\), so the loss term is the residual sum of squares divided by \(2n\). This is an optimization criterion; by itself, it does not assert that errors follow a particular probability distribution. The factor \(1/2\) is a scaling convention, not a claim about the error variance.
The penalty parameter \(\lambda\) is nonnegative. When \(\lambda=0\), the criterion reduces to least squares. As \(\lambda\) increases, a nonzero slope adds more to the objective, so the fit is encouraged to use smaller slopes. The L1 penalty can also yield exactly zero slopes, which is why lasso can perform variable selection. The exact conditions under which a coefficient becomes zero are not needed to evaluate or interpret the objective.
What Each Term Contributes
The loss measures disagreement between observed and fitted responses. Because residuals are squared, a residual twice as large contributes four times as much to the residual sum of squares. The penalty instead measures the total absolute slope size. It does not square coefficients, and a negative slope contributes just as much as a positive slope of the same magnitude.
The intercept is ordinarily left out of the penalty so that the model can shift its overall level without treating that shift as a predictor effect to be shrunk. Penalizing the intercept would define a different objective. When an analysis has no intercept by design, the model and its penalty should be stated explicitly rather than silently using the usual intercept convention.
The objective is minimized over the coefficient values. For any proposed coefficient vector, its value is the sum of two calculated quantities: residual loss and penalty cost. A lower residual loss does not automatically mean a lower objective, because the coefficient penalty may outweigh the improvement in fit.
Worked Example: Evaluating Candidate Lasso Fits
Suppose there are three observations with predictor values \(x=(-1,0,1)\) and responses \(y=(-2,0,2)\). Use a model with an intercept and one slope, and set \(\lambda=0.5\). Evaluate the objective at slopes \(\beta_1=0\), \(0.5\), and \(1\). For each slope, use the least-squares intercept for that fixed slope.
Method. Compute the fitted values and residual sum of squares, divide that sum by \(2n=6\), then add \(0.5|\beta_1|\). The intercept is unpenalized. This calculation avoids the mistake of comparing residual loss alone.
For each fixed slope, the least-squares intercept is the mean of \(y_i-\beta_1x_i\). Both \(x\) and \(y\) have mean zero, so this intercept is \(0\) for all three candidates.
Candidate \(\beta_1=0\). The fitted values are \((0,0,0)\). The residuals are \((-2,0,2)\), giving \( \mathrm{RSS}=(-2)^2+0^2+2^2=4+0+4=8 \). The loss is \(8/6=4/3\), and the penalty is \(0.5|0|=0\). Thus \(Q_{0.5}=4/3\).
Candidate \(\beta_1=0.5\). The fitted values are \((-0.5,0,0.5)\). The residuals are \((-1.5,0,1.5)\), giving \( \mathrm{RSS}=(-1.5)^2+0^2+1.5^2=2.25+0+2.25=4.5 \). The loss is \(4.5/6=0.75\), and the penalty is \(0.5(0.5)=0.25\). The objective is \(0.75+0.25=1.00\).
Candidate \(\beta_1=1\). The fitted values are \((-1,0,1)\). The residuals are \((-1,0,1)\), giving \( \mathrm{RSS}=1+0+1=2 \). The loss is \(2/6=1/3\), and the penalty is \(0.5(1)=0.5\). The objective is \(1/3+0.5=5/6\), or approximately \(0.8333\).
Among these three candidates, the objective is lowest at \(\beta_1=1\), since \(5/6<1<4/3\). This is only a comparison of the stated candidates, not a claim that they are the only possible slopes. The calculation checks that the residual improvement from \(0.5\) to \(1\) is large enough to offset the additional penalty.
Why the Objective Has This Form
The two terms encode distinct preferences. Least squares alone favors a coefficient vector with the smallest residual sum of squares. Adding the penalty means a coefficient vector must earn its size by improving the fit enough to justify its additional cost. In particular, increasing \(\lambda\) changes the relative importance of these preferences; it does not change the observed responses or the definition of a residual.
The normalization by \(2n\) affects the numerical scale of \(\lambda\). To see why, write \(R(\boldsymbol{\beta},\beta_0)\) for the residual sum of squares and \(L_1(\boldsymbol{\beta})=\sum_{j=1}^p|\beta_j|\). The stated objective is \(R/(2n)+\lambda L_1\). Multiplication by the positive number \(n\) gives
Multiplying an objective by a positive constant does not change which coefficient vectors minimize it: if one objective value is smaller than another, multiplying both by the same positive number preserves that ordering. Therefore the convention \(R/2+\lambda_{\mathrm{alt}}L_1\) gives the same minimizer as \(R/(2n)+\lambda L_1\) when \(\lambda_{\mathrm{alt}}=n\lambda\). A numerical value of \(\lambda\) has no meaning apart from the objective scaling used with it.
Worked Example: Comparing the Trade-Off at Two Penalty Values
Consider three observations with \(x=(-1,0,1)\) and \(y=(-2,0,2)\), as in a hypothetical calibration exercise. Compare two candidate fits: A has intercept \(0\) and slope \(2\); B has intercept \(0\) and slope \(1.5\). Use \(Q_\lambda=\mathrm{RSS}/(2n)+\lambda|\beta_1|\). Compare the candidates at \(\lambda=0.1\) and \(\lambda=0.5\).
Method. Calculate each candidate's residual loss and L1 penalty separately, then add them. This avoids concluding that the exact-fitting candidate must always minimize a penalized objective.
For A, the fitted values are \((-2,0,2)\), so all residuals are zero and \(\mathrm{RSS}=0\). Since \(2n=6\), its loss is \(0/6=0\), and its penalty is \(2\lambda\). Thus its objective is \(0.2\) at \(\lambda=0.1\) and \(1.0\) at \(\lambda=0.5\).
For B, the fitted values are \((-1.5,0,1.5)\), so the residuals are \((-0.5,0,0.5)\). Hence \( \mathrm{RSS}=(-0.5)^2+0^2+0.5^2=0.25+0+0.25=0.5 \), and the loss is \(0.5/6=1/12\). Its penalty is \(1.5\lambda\). At \(\lambda=0.1\), its objective is \(1/12+0.15=0.2333\) to four decimal places. At \(\lambda=0.5\), its objective is \(1/12+0.75=5/6\), or approximately \(0.8333\).
At \(\lambda=0.1\), A has the lower objective: \(0.2<0.2333\). At \(\lambda=0.5\), B has the lower objective: \(0.8333<1.0\). The comparison shows how a larger penalty can favor the candidate with the smaller slope despite its larger residual loss. It compares only A and B; other coefficient values could have lower objectives.
A Useful Check on Objective Conventions
The number reported as \(\lambda\) depends on both the loss scaling and the penalty convention. When reading a formula or software output, identify the exact residual term and whether the intercept is included in the penalty before comparing values. In particular, changing \(R/(2n)\) to \(R/2\) without changing \(\lambda\) changes the balance between fit and penalty.
Worked Example: The Intercept Is Not Part of the Usual Penalty
A hypothetical two-observation data set has \(x=(0,1)\) and \(y=(5,7)\). Compare two candidate fits at \(\lambda=0.3\): an exact fit with \((\beta_0,\beta_1)=(5,2)\), and an intercept-only fit with \((\beta_0,\beta_1)=(6,0)\). Use the usual lasso objective with \(n=2\).
Method. Compute \( \mathrm{RSS}/(2n)=\mathrm{RSS}/4 \), then penalize only \(|\beta_1|\). This method avoids the common error of adding the intercept to the L1 penalty.
For the exact fit, the predictions are \((5,7)\), so the residuals are \((0,0)\) and \(\mathrm{RSS}=0^2+0^2=0\). The loss is \(0/4=0\), and the penalty is \(0.3|2|=0.6\). Its objective is \(0.6\).
For the intercept-only fit, the predictions are \((6,6)\), so the residuals are \((-1,1)\) and \(\mathrm{RSS}=(-1)^2+1^2=2\). The loss is \(2/4=0.5\), and the penalty is \(0.3|0|=0\). Its objective is \(0.5\). Among these two candidates, the intercept-only fit has the lower objective.
If the intercept were incorrectly penalized, the exact-fit objective would be calculated as \(0+0.3(|5|+|2|)=2.1\), and the intercept-only objective as \(0.5+0.3|6|=2.3\). That calculation would reverse the comparison, but it is not the stated lasso convention. The check is to write the penalty sum over slopes, \(j=1,\ldots,p\), not over the intercept and slopes together.
Interpreting Lambda Without Overclaiming
A larger \(\lambda\) assigns a greater cost to a given total absolute slope size. Consequently, the minimized objective tends to favor a model with less total coefficient size, potentially at the cost of greater residual loss. However, it is not correct to infer that every individual slope must decrease steadily as \(\lambda\) increases. In particular, changing relationships among predictors can affect how coefficient size is allocated. Interpret the objective as a trade-off, not as a guarantee about the path of each coefficient.
The numerical value of the penalty also depends on measurement scales. If predictor scales differ substantially, equal absolute slope magnitudes need not represent comparable effects. A fitted lasso model therefore needs a clearly stated scaling convention, and the value of \(\lambda\) should be interpreted in the context of that convention. This does not alter the definition of the L1 penalty; it affects what coefficient magnitudes represent.
Three checks make objective calculations more reliable: use the specified normalization of residual loss, apply absolute values to every penalized slope, and leave the intercept out when using the usual convention. Then compare the sum of loss and penalty, rather than ranking fits on either component alone.
Check Your Understanding
For each problem, compute the requested objective or comparison using the stated lasso convention.
- A two-observation data set has \(x=(0,1)\), \(y=(1,3)\), and candidate coefficients \((\beta_0,\beta_1)=(1,2)\). With \(\lambda=0.25\), calculate the objective using \(\mathrm{RSS}/(2n)+\lambda|\beta_1|\). (Answer: \(0.50\))
- For \(x=(0,1)\), \(y=(0,2)\), compare candidate A \((\beta_0,\beta_1)=(0,2)\) and candidate B \((1,0)\) at \(\lambda=0.2\). Which candidate has the lower objective, and what are both objective values? (Answer: A has \(0.40\); B has \(0.50\); A is lower.)
- An objective uses \(\mathrm{RSS}/(2n)+\lambda L_1\) with \(n=5\) and \(\lambda=0.04\). What penalty coefficient gives the same minimizer when the loss is written as \(\mathrm{RSS}/2\)? (Answer: \(\lambda_{\mathrm{alt}}=0.20\))
- For \(x=(0,1)\), \(y=(4,6)\), compare the exact-fit candidate \((4,2)\) with the intercept-only candidate \((5,0)\). Find the value of \(\lambda\) at which their objectives are equal, then identify the lower-objective candidate at \(\lambda=0.3\). (Answer: They are equal at \(\lambda=0.25\); at \(\lambda=0.3\), the intercept-only candidate is lower.)
- A candidate fit has \(n=4\), residual sum of squares \(2\), and slopes \((-1.2,0.5,-0.3)\). With \(\lambda=0.4\), calculate its objective, excluding the intercept from the penalty. (Answer: \(1.05\))