Two Ways to Control the L1 Norm
Lasso can be written in two closely related ways. One form minimizes squared prediction error while requiring the coefficients to stay within an L1 radius. The other adds a penalty for the coefficients’ L1 norm to the squared-error objective. The first uses a constraint radius; the second uses a penalty strength. Both control coefficient size, but a particular radius does not correspond to a universal penalty value.
To make the comparison precise, let \(y_i\) be the response for case \(i\), let \(x_i\) be its predictor vector, let \(\beta_0\) be an intercept, and let \(\beta=(\beta_1,\ldots,\beta_p)\) be the slope coefficients. Use the least-squares criterion
In this tutorial the intercept is not penalized. To simplify expressions, write \(Q(\beta)\) for the criterion after the intercept has been handled in the same way in both optimization problems—for example, by including it as a variable in the minimization but excluding it from the L1 norm. Define \(\|\beta\|_1=\sum_{j=1}^p|\beta_j|\).
The constraint states how large the total absolute coefficient magnitude may be. The penalty instead charges \(\lambda\) units of objective value for each unit of L1 norm. A large \(\lambda\) makes that charge more influential; a small \(\lambda\) makes squared error relatively more influential. The conversion between \(t\) and \(\lambda\) depends on the least-squares surface produced by the data.
Why the Lagrangian Connects the Forms
The least-squares criterion is convex in the coefficients, and the set \(\{\beta:\|\beta\|_1\leq t\}\) is convex. For a positive radius, the standard convex optimization conditions give a nonnegative Lagrange multiplier for the constraint. The constrained problem’s Lagrangian is
For a fixed multiplier, the term \(-\lambda t\) is constant with respect to \(\beta\). Therefore minimizing the Lagrangian over \(\beta\) is equivalent to minimizing \(Q(\beta)+\lambda\|\beta\|_1\), the penalized lasso objective. This gives the core connection: a multiplier associated with the active constraint acts as the penalty strength.
Specify \(t\) and minimize \(Q(\beta)\) over coefficients satisfying \(\|\beta\|_1\leq t\).
At an optimum, the Lagrange multiplier \(\lambda\) is nonnegative. If the constraint is active, its value measures how much the best attainable squared error would improve if the radius were relaxed slightly.
For that multiplier, minimize \(Q(\beta)+\lambda\|\beta\|_1\). Under the convexity conditions, a constrained solution and a penalized solution can coincide.
The relationship can also be justified directly. Suppose \(\hat\beta_\lambda\) minimizes the penalized objective, and set \(t=\|\hat\beta_\lambda\|_1\). For any \(\beta\) with \(\|\beta\|_1\leq t\), penalized optimality gives
Since \(\|\beta\|_1\leq t=\|\hat\beta_\lambda\|_1\), subtracting the penalty terms yields \(Q(\hat\beta_\lambda)\leq Q(\beta)\). Thus \(\hat\beta_\lambda\) also minimizes squared error among coefficients within that radius. Conversely, under the convex optimization conditions, a constrained optimum has a multiplier satisfying the Lagrangian conditions, so it also minimizes the corresponding penalized objective.
This argument establishes a correspondence, not a data-independent conversion formula. It also allows nonunique coefficient solutions: if the design is rank-deficient, more than one coefficient vector may minimize an objective. The equivalence is about solutions to the stated optimization problems, not a claim that every coefficient vector is unique.
Worked Examples of the Correspondence
Worked Example: Finding the Penalty for a Given Radius
A hypothetical pricing analysis has two standardized predictors. After handling the intercept, suppose its coefficient-dependent squared-error criterion is \(Q(\beta)=(\beta_1-4)^2+(\beta_2-2)^2\). Find the constrained solution for radius \(t=3\), and identify a penalty strength that gives the same solution.
Method. Use the Lagrangian multiplier for the L1 constraint. The criterion is strictly convex, and the constraint is convex, so a solution satisfying the convex optimization conditions is a global minimum.
The unconstrained least-squares coefficients are \((4,2)\), whose L1 norm is \(4+2=6\), so the radius \(3\) is binding. At a candidate with both coefficients positive, the Lagrangian is
Because both coefficients are positive at the solution, differentiating with respect to each coefficient gives \(2(\beta_1-4)+\lambda=0\) and \(2(\beta_2-2)+\lambda=0\). Hence \(\beta_1=4-\lambda/2\) and \(\beta_2=2-\lambda/2\). The active constraint requires their sum to be 3:
Substitution gives \(\hat\beta=(2.5,0.5)\), with \(\|\hat\beta\|_1=2.5+0.5=3\). The squared-error criterion is \(Q(\hat\beta)=(2.5-4)^2+(0.5-2)^2=2.25+2.25=4.5\). The penalized objective is \(4.5+3(3)=13.5\). The coefficients are positive as assumed, and the constraint is satisfied exactly, so the calculation is internally consistent. The constrained solution is also the penalized solution at \(\lambda=3\).
Common mistake avoided: The radius is 3, but the penalty is also 3 here only because this particular criterion and data produce that multiplier. The equality of these two numbers is not a general rule.
Worked Example: Translating Between Squared-Error Scales
A hypothetical model has \(n=40\) observations. One implementation minimizes \(\operatorname{RSS}/(2n)+0.075\|\beta\|_1\); another minimizes \(\operatorname{RSS}+\lambda_{\text{RSS}}\|\beta\|_1\). Find the penalty in the second implementation that gives the same optimizer, assuming the same intercept treatment and predictors.
Method. Multiply the entire first objective by the positive constant \(2n\). Multiplying an objective by a positive constant does not change its minimizer.
Here \(2n=2(40)=80\). After multiplication, the first objective becomes
because \(80(0.075)=6\). Therefore \(\lambda_{\text{RSS}}=6\). The two objectives have the same minimizer, though their objective values have different scales. The coefficient units and L1 radius are unchanged; only the numerical penalty parameter changes with the scale of the squared-error term.
Common mistake avoided: Copying the value \(0.075\) into the RSS-scaled objective would make the penalty much weaker relative to squared error. The factor \(2n\) must multiply both terms.
Worked Example: Recognizing an Inactive Constraint
For another hypothetical analysis, let \(Q(\beta)=(\beta_1-1)^2+(\beta_2+2)^2\), and constrain the L1 norm to be at most \(t=4\). Determine the constrained solution and its corresponding multiplier.
Method. First find the unconstrained least-squares minimizer, then check whether it satisfies the bound. Setting each squared term to zero gives \(\hat\beta=(1,-2)\). Its L1 norm is \(|1|+|-2|=3\), and \(3\leq4\), so it is feasible.
The unconstrained minimum has \(Q(\hat\beta)=0\), and a sum of squares cannot be less than zero. It is therefore also the constrained minimum. Because the constraint has unused capacity, its multiplier is \(\lambda=0\); the penalized objective at \(\lambda=0\) is simply \(Q(\beta)\). The check is \(Q(1,-2)=0\) and \(\|(1,-2)\|_1=3\leq4\).
Common mistake avoided: A positive radius does not mean that a positive penalty must be applied. When the bound is inactive, the associated multiplier is zero.
What the Parameter Relationship Does—and Does Not—Say
For a fixed data set and a fixed objective convention, increasing \(\lambda\) generally makes large L1 norms less attractive. This monotonic direction can be established without assuming that the solution is unique. Let \(\hat\beta_1\) minimize the penalized objective at \(\lambda_1\), and let \(\hat\beta_2\) minimize it at \(\lambda_2\), where \(0\leq\lambda_1<\lambda_2\). Their optimality gives
Adding these inequalities and cancelling the squared-error terms gives \((\lambda_2-\lambda_1)(\|\hat\beta_2\|_1-\|\hat\beta_1\|_1)\leq0\). Since \(\lambda_2-\lambda_1>0\), it follows that \(\|\hat\beta_2\|_1\leq\|\hat\beta_1\|_1\). Thus a larger penalty cannot produce a larger L1 norm for chosen optimal solutions. The exact coefficient path and the penalty needed to attain a particular radius still depend on the data.
| Feature | Constrained form | Penalized form |
|---|---|---|
| Control parameter | Radius \(t\), an upper bound on \(\|\beta\|_1\) | Penalty \(\lambda\), the cost per unit of \(\|\beta\|_1\) |
| Optimization statement | Minimize squared error subject to the bound | Minimize squared error plus the L1 cost |
| Connection | A constraint multiplier supplies a corresponding penalty | A solution at \(\lambda\) is constrained-optimal at its own L1 norm |
| Numerical conversion | Depends on the fitted problem and objective scale | Depends on the fitted problem and objective scale |
At \(t=0\), the constrained problem requires all penalized coefficients to be zero. A sufficiently large penalty may produce that solution, but the value needed depends on the data. At the other extreme, if the radius includes an unconstrained least-squares minimizer, the constraint may be inactive. If several unconstrained minimizers exist, the radius must be large enough to include a minimizer for the constraint to leave the optimum unchanged.
These parameter interpretations matter when comparing software output, documenting a model, or translating a tuning value into a stated coefficient bound. The comparison is valid only when the response scale, predictor scale, intercept treatment, and squared-error normalization agree. If any of these differ, the numerical value of \(\lambda\) can change even when the underlying optimization problem is equivalent.
Common Exam Traps
- Assuming \(t=\lambda\). One is a bound and the other is an objective multiplier. Their relationship is determined by the optimization problem, not by equality of their numerical values.
- Ignoring objective normalization. Changing from \(\operatorname{RSS}/(2n)\) to \(\operatorname{RSS}\) multiplies the corresponding penalty by \(2n\) if the minimizer is to remain the same.
- Forgetting the unpenalized intercept. The intercept is excluded from \(\|\beta\|_1\) in the convention used here. Penalizing it changes the problem and the correspondence.
- Assuming the constraint is always active. If an unconstrained least-squares solution already meets the radius, the multiplier can be zero.
- Claiming a universal inverse conversion. The radius attained at a particular \(\lambda\) depends on the design matrix and response; it is not determined by \(\lambda\) alone.
Check Your Understanding
For each problem, use the stated objective convention and compare only problems with the same predictors and intercept treatment.
- A coefficient-dependent criterion is \(Q(\beta)=(\beta_1-4)^2+(\beta_2-2)^2\), with constraint \(\|\beta\|_1\leq4\). Using the positive-coefficient Lagrangian conditions, find the solution and its multiplier. (Answer: \(\hat\beta=(3,1)\), \(\lambda=2\).)
- An implementation minimizes \(\operatorname{RSS}/(2n)+0.08\|\beta\|_1\) with \(n=25\). What penalty in an equivalent \(\operatorname{RSS}+\lambda_{\text{RSS}}\|\beta\|_1\) objective gives the same optimizer? (Answer: \(\lambda_{\text{RSS}}=4\).)
- Let \(Q(\beta)=(\beta-2)^2\) for one coefficient, with constraint \(|\beta|\leq0.5\). Find the constrained solution and a corresponding multiplier. (Answer: \(\hat\beta=0.5\), \(\lambda=3\).)
- A least-squares minimizer is \((2,-1)\), and a proposed radius is \(t=3.5\). Is the constraint active at that minimizer, and what multiplier is associated with it? (Answer: It is inactive because \(2+1=3<3.5\); \(\lambda=0\).)
- For \(0\leq\lambda_1<\lambda_2\), penalized solutions have L1 norms \(r_1\) and \(r_2\). What inequality must their norms satisfy, even if solutions are not unique? (Answer: \(r_2\leq r_1\).)