The Lasso Objective and Absolute Penalty

Why the Lasso Uses an Absolute Penalty

Penalized regression balances two aims: fitting the observed responses and keeping coefficient estimates under control. Ridge regression uses squared coefficients for its penalty. The lasso instead uses the sum of the coefficients’ absolute values. This change in shape matters: the lasso penalty has a sharp point at zero, whereas the ridge penalty is smooth there.

This tutorial focuses on the lasso objective itself, the behavior of its absolute-value term at zero, and numerical comparisons with the ridge penalty. The formulas use a common least-squares convention; other sources may scale the residual sum of squares differently, so always identify the convention before substituting a penalty parameter.

Model and assumptions: For observations \(i=1,\ldots,n\) and predictors \(j=1,\ldots,p\), consider \(y_i=\beta_0+\sum_{j=1}^p x_{ij}\beta_j+\varepsilon_i\). For the usual Gaussian regression interpretation, assume the predictors are fixed and the errors are independent with \(\varepsilon_i\sim N(0,\sigma^2)\), where \(\sigma^2>0\). The penalized least-squares objective can also be defined without a normality assumption; normality supplies a likelihood interpretation, not the algebraic definition of the penalty.

Let \(\operatorname{RSS}(\beta_0,\boldsymbol{\beta})\) denote the residual sum of squares, where \(\boldsymbol{\beta}=(\beta_1,\ldots,\beta_p)\). In the convention used here, the intercept is not penalized and the lasso objective is

J_{\text{lasso}}(\beta_0,\boldsymbol{\beta}) = \operatorname{RSS}(\beta_0,\boldsymbol{\beta}) +\lambda\sum_{j=1}^{p}|\beta_j|, \qquad \lambda\geq 0.

The tuning parameter \(\lambda\) controls the penalty’s weight. When \(\lambda=0\), the objective is ordinary residual sum of squares. For a positive \(\lambda\), a coefficient vector with a larger absolute-value sum incurs a larger penalty, all else equal. The objective combines the residual fit and the penalty; minimizing it is the estimation problem, not a calculation of the penalty term alone.

The Absolute-Value Penalty and Its Shape

For one coefficient \(\beta\), the lasso penalty contribution is \(\lambda|\beta|\). Its values on the two sides of zero are

\lambda|\beta|= \begin{cases} \lambda\beta, & \beta>0,\\ -\lambda\beta, & \beta<0,\\ 0, & \beta=0. \end{cases}

The formula is continuous at zero, but its slope changes abruptly there: the slope is \(-\lambda\) to the left and \(+\lambda\) to the right. Therefore, when \(\lambda>0\), \(\lambda|\beta|\) is not differentiable at zero. By contrast, the ridge penalty for this coefficient is \(\lambda\beta^2\), whose derivative is \(2\lambda\beta\); that derivative exists at zero and equals zero.

A useful comparison is between the penalty shapes for coefficients on the same numerical scale. If \(0<|\beta|<1\), then \(|\beta|>\beta^2\); if \(|\beta|>1\), then \(|\beta|<\beta^2\). At \(|\beta|=1\), they are equal. These comparisons describe the penalty terms when the same numerical \(\lambda\) is used. They do not mean the two methods have the same overall fit or that equal numerical values of \(\lambda\) are directly comparable across differently scaled objective conventions.

Key distinction: The lasso adds \(\lambda|\beta_j|\) for each penalized coefficient, not \(\lambda\beta_j\) and not \(\lambda\beta_j^2\). The absolute value makes a negative coefficient contribute a positive penalty, and creates a nondifferentiable point at zero.

The numerical scale of each predictor matters because the penalty acts on coefficient magnitudes. If a predictor is rescaled, its coefficient changes inversely to preserve the same fitted values, and so does that coefficient’s penalty contribution. Consequently, comparisons of coefficient penalties require a stated scale. Standardizing predictors before fitting is a common way to make coefficient magnitudes more comparable, but the precise preprocessing convention should be reported.

Worked Penalty Comparisons

Worked Example: Comparing Penalties for Small Coefficients

A hypothetical insurer is fitting a model for claim processing time. The predictors have been standardized, and a fitted candidate has coefficients \((0.20,-0.50,1.50)\). Use \(\lambda=0.6\) to compute its lasso penalty and ridge penalty, each defined as \(\lambda\) times the corresponding sum. Assume the response model has independent errors \(\varepsilon_i\sim N(0,9)\); no interest or mortality assumptions are involved.

Method. Calculate the absolute-value sum and the sum of squares separately, then multiply each by \(0.6\). The response fit, represented by RSS, is not supplied, so this example compares penalty contributions only.

For the lasso, the absolute values are \(0.20\), \(0.50\), and \(1.50\). Thus \(0.20+0.50+1.50=2.20\), and the penalty is \(0.6(2.20)=1.32\).

For ridge, the squares are \(0.20^2=0.04\), \((-0.50)^2=0.25\), and \(1.50^2=2.25\). Thus \(0.04+0.25+2.25=2.54\), and the penalty is \(0.6(2.54)=1.524\).

For the small coefficient \(0.20\), the lasso contribution is \(0.6(0.20)=0.12\), while the ridge contribution is \(0.6(0.04)=0.024\). For the coefficient \(1.50\), the contributions are \(0.6(1.50)=0.90\) and \(0.6(2.25)=1.35\), respectively. The totals are consistent: lasso gives \(1.32\), and ridge gives \(1.524\). This calculation compares penalties only; it does not establish which model has the smaller full objective because the RSS values are needed for that.

Common mistake avoided: Squaring the negative coefficient gives \(0.25\) for ridge, but its lasso contribution uses \(|-0.50|=0.50\), not \(-0.50\).

Worked Example: Comparing Full Objectives Under One Convention

A hypothetical risk analyst has two candidate coefficient vectors from a model with standardized predictors and an unpenalized intercept. Candidate A has RSS \(14\) and coefficients \((0.20,-0.50,1.00)\). Candidate B has RSS \(13.5\) and coefficients \((0.40,-0.20,0.80)\). With \(\lambda=2\), compare their lasso objectives and, separately, their ridge objectives. Assume independent Gaussian errors with variance \(\sigma^2=4\); no interest or mortality assumptions are involved.

Method. Use \(J_{\text{lasso}}=\operatorname{RSS}+2\sum_j|\beta_j|\) for the lasso comparison and \(J_{\text{ridge}}=\operatorname{RSS}+2\sum_j\beta_j^2\) for the ridge comparison. Do not carry the lasso penalty into the ridge objective.

For candidate A, the lasso sum is \(0.20+0.50+1.00=1.70\). Its lasso penalty is \(2(1.70)=3.40\), so its lasso objective is \(14+3.40=17.40\). For candidate B, the lasso sum is \(0.40+0.20+0.80=1.40\). Its penalty is \(2(1.40)=2.80\), so its lasso objective is \(13.5+2.80=16.30\). Under this lasso objective, candidate B has the smaller value.

For ridge, candidate A’s sum of squares is \(0.20^2+(-0.50)^2+1.00^2=0.04+0.25+1.00=1.29\). Its ridge penalty is \(2(1.29)=2.58\), giving objective \(14+2.58=16.58\). Candidate B’s sum of squares is \(0.40^2+(-0.20)^2+0.80^2=0.16+0.04+0.64=0.84\). Its ridge penalty is \(2(0.84)=1.68\), giving objective \(13.5+1.68=15.18\). Candidate B also has the smaller ridge objective.

The result is checked by adding each penalty to its own candidate’s RSS. Although candidate B ranks lower under both stated comparisons, the numerical lasso and ridge objectives should not be compared with each other as if they were the same criterion: their penalty definitions differ.

Common mistake avoided: A smaller RSS alone does not determine the penalized objective. The penalty must be calculated using the method being evaluated and added to that candidate’s RSS.

What Nondifferentiability Looks Like in an Objective

Worked Example: Checking the Lasso Objective at Zero

Consider two hypothetical observations with predictor values \(x_1=x_2=1\), response values \(y_1=2\) and \(y_2=0\), and a regression through the origin \(y_i=\beta x_i+\varepsilon_i\). Take independent errors \(\varepsilon_i\sim N(0,1)\), and use the lasso objective \(J(\beta)=\operatorname{RSS}(\beta)+2|\beta|\). There is no intercept, and no interest or mortality assumptions are involved. Find the objective at \(\beta=0\), \(0.10\), and \(-0.10\), and verify that the objective is not differentiable at zero.

Method. Compute residuals from \(y_i-\beta x_i\), square and add them to obtain RSS, then add \(2|\beta|\). To check differentiability, write the objective separately for positive and negative \(\beta\).

At \(\beta=0\), the residuals are \(2\) and \(0\), so \(\operatorname{RSS}(0)=2^2+0^2=4\). The penalty is \(2|0|=0\), and \(J(0)=4\).

At \(\beta=0.10\), the residuals are \(2-0.10=1.90\) and \(0-0.10=-0.10\). Thus \(\operatorname{RSS}(0.10)=1.90^2+(-0.10)^2=3.61+0.01=3.62\). The penalty is \(2(0.10)=0.20\), giving \(J(0.10)=3.82\).

At \(\beta=-0.10\), the residuals are \(2-(-0.10)=2.10\) and \(0-(-0.10)=0.10\). Thus \(\operatorname{RSS}(-0.10)=2.10^2+0.10^2=4.41+0.01=4.42\). The penalty is \(2(0.10)=0.20\), giving \(J(-0.10)=4.62\).

For any \(\beta\), the residual sum of squares is \((2-\beta)^2+(0-\beta)^2=2\beta^2-4\beta+4\). Therefore, for positive and negative \(\beta\), respectively,

J(\beta)= \begin{cases} 2\beta^2-2\beta+4, & \beta>0,\\ 2\beta^2-6\beta+4, & \beta<0. \end{cases}

The right-hand derivative at zero is \(-2\), obtained from the first expression; the left-hand derivative at zero is \(-6\), obtained from the second. They are unequal, so the full objective is not differentiable at zero. The directly computed objective values also check against the formulas: at \(0.10\), \(0.02-0.20+4=3.82\); at \(-0.10\), \(0.02+0.60+4=4.62\).

Common mistake avoided: The residual sum of squares is smooth at zero, but that does not make the full lasso objective differentiable there. The absolute-value term must also be considered.

Scaling Conventions and Common Traps

Some texts define the objective using an average or a half of the residual sum of squares. For example, one may encounter

\frac{1}{2n}\operatorname{RSS}+\lambda\sum_{j=1}^p|\beta_j|.

This is a valid convention, but its numerical \(\lambda\) is not automatically the same as the one in \(\operatorname{RSS}+\lambda\sum_j|\beta_j|\). Multiplying the entire objective by a positive constant does not change its minimizer; multiplying only the RSS term changes its relative weight against the penalty unless \(\lambda\) is adjusted accordingly. Thus, when calculating an objective or interpreting a stated tuning parameter, first copy the exact formula being used.

The intercept is usually excluded from the penalty. If a problem explicitly includes it, follow the stated objective instead. Penalizing an intercept would make a shift in the response level contribute to the penalty, which is not the usual convention. Also check whether the predictors were standardized: the same coefficient value can correspond to different changes in fitted response when predictor units differ.

Exam check: Before computing a lasso objective, confirm (1) whether the RSS is summed, averaged, or halved; (2) whether the intercept is penalized; (3) whether the penalty uses absolute values; and (4) the scale of the coefficients. These choices determine the arithmetic.

A frequent incorrect calculation illustrates why these checks matter. Suppose one coefficient is \(-0.30\) and \(\lambda=2\). The wrong lasso calculation \(2(-0.30)=-0.60\) assigns a negative penalty, which would reward a negative coefficient. The correction is \(2|-0.30|=0.60\). If the ridge penalty is being calculated instead, the correct contribution is \(2(-0.30)^2=2(0.09)=0.18\). Taking the absolute value and squaring are different operations, so the model’s penalty definition must decide which one to use.

The cusp at zero is the defining local feature of the lasso penalty. Away from zero, the absolute-value term has an ordinary slope; at zero, its left and right slopes differ. This explains why standard differentiation cannot be applied there as though the objective were smooth. The main practical lesson in this tutorial is to calculate the absolute-value penalty exactly as written and to keep the fit term, penalty, and scaling convention distinct.

Check Your Understanding

For each question, use the penalty definition stated in the problem and check the sign and scaling before calculating.

A standardized-predictor model has coefficients \((0.25,-0.40)\) and lasso tuning parameter \(\lambda=1.5\). What is the total lasso penalty? (Answer: \(0.975\))
For a single coefficient \(\beta=-0.20\) and \(\lambda=0.8\), calculate the lasso and ridge contributions using \(\lambda|\beta|\) and \(\lambda\beta^2\), respectively. (Answer: lasso \(0.16\); ridge \(0.032\))
A candidate has RSS \(18\), coefficients \((0.30,-0.60)\), and objective \(J=\operatorname{RSS}+3\sum_j|\beta_j|\). Calculate \(J\). (Answer: \(20.7\))
For \(\lambda=1\), compare the lasso and ridge penalty terms \(\lambda|\beta|\) and \(\lambda\beta^2\) at \(\beta=0.40\). Which is larger, and by how much? (Answer: lasso is larger; \(0.40-0.16=0.24\))
An objective is defined as \(\operatorname{RSS}/(2n)+\lambda\sum_j|\beta_j|\). For \(n=10\), RSS \(=24\), coefficients \((0.20,-0.30)\), and \(\lambda=0.5\), calculate the objective. (Answer: \(24/20+0.5(0.20+0.30)=1.45\))

What You'll Learn