Lasso Constraint Geometry and Sparse Solutions

Why Constraint Geometry Matters

Lasso and ridge both restrict the size of regression coefficients to reduce the fit’s sensitivity to the data. Yet their constraints have different shapes, and that difference helps explain why lasso can set a coefficient exactly to zero while ridge usually does not. The key is not simply that one method shrinks more than the other: it is that the lasso constraint has corners aligned with the coordinate axes.

A geometric picture uses contours of the loss function. Each contour joins coefficient vectors with the same loss. Imagine expanding these contours outward from the unconstrained least-squares estimate until one first touches the allowed coefficient region. At the point of contact lies the constrained solution. With a lasso constraint, that first contact can be a corner, where one or more coefficients equal zero. A smooth ridge boundary has no corresponding axis-aligned corners.

Loss Contours and Constraint Regions

For clarity, consider two predictors and write their coefficients as \(\beta_1\) and \(\beta_2\). Assume the response and predictors have been centered, so an intercept can be left out of the geometry; the intercept, when used, is not penalized. Let \(L(\beta_1,\beta_2)\) denote the least-squares loss. A loss contour is a set of coefficient pairs with the same value of \(L\). For a quadratic least-squares loss with a positive-definite quadratic form, these contours are ellipses centered at the unconstrained least-squares estimate. They are circles in the special case of equal, orthogonal directions in the loss.

The constrained lasso problem minimizes the least-squares loss over the \(L_1\) ball. Equivalently, one can minimize a penalized least-squares objective such as \(L(\boldsymbol{\beta})+\lambda\sum_j|\beta_j|\), with \(\lambda\geq0\). The constraint radius \(t\) and penalty weight \(\lambda\) express the strength of the restriction in different ways: increasing the penalty weight generally corresponds to allowing a smaller coefficient norm. Exact numerical conversions depend on the loss scaling and data. Ridge uses a squared-coefficient penalty, or equivalently an \(L_2\) constraint.

As the loss contours expand from the unconstrained estimate, the first contour that touches the constraint region identifies a constrained minimum. When the loss is strictly convex, that minimum is unique. This contour picture is useful because it shows how the shape of the region—not just the amount of shrinkage—affects which coefficients can become zero.

How a Lasso Corner Creates a Zero

In two dimensions, the lasso boundary has vertices at \((t,0)\), \((-t,0)\), \((0,t)\), and \((0,-t)\). At a vertex such as \((t,0)\), the second coefficient is exactly zero. If an expanding loss contour first touches the diamond at that vertex, the constrained solution therefore has one zero coefficient. In more dimensions, the \(L_1\) ball has faces and corners aligned with coordinate subspaces; contact with a face can correspond to several coefficients being zero.

The corner does not force every solution to have a zero. An ellipse can first touch the diamond along the middle of one of its edges, where both coefficients are nonzero. Nor does a constraint radius alone determine the solution: the position and shape of the loss contours matter. The geometry makes exact zeros possible and often natural, not inevitable in every problem.

A short optimality check makes the corner mechanism precise. Use the objective \(f(\boldsymbol{\beta})+\lambda\|\boldsymbol{\beta}\|_1\), where \(f\) is a differentiable convex loss and \(\|\boldsymbol{\beta}\|_1=\sum_j|\beta_j|\). For a nonzero coefficient, the derivative of its absolute-value penalty is its sign. At a zero coefficient, the allowable subgradient values range from \(-1\) to \(1\). Thus a zero coordinate can satisfy the optimum condition whenever the loss gradient in that coordinate is small enough to be balanced by a subgradient of the penalty. At a nonzero coordinate, the penalty can balance the gradient only in the direction fixed by that coefficient’s sign. The interval of allowable subgradients at zero is the algebraic counterpart of the lasso’s corner.

Contact on an Edge Need Not Produce a Zero

A lasso solution can also lie on an edge rather than at a vertex. An edge of the two-dimensional diamond has equation such as \(\beta_1+\beta_2=t\) when both coefficients are positive. Points in the interior of this edge have two nonzero coordinates. The next example verifies such a solution directly.

Why Ridge Usually Does Not Set Coefficients to Zero

A two-dimensional ridge constraint has a smooth circular boundary. A loss ellipse typically first touches that circle at a point that is not on either coordinate axis, so both coefficients are nonzero. In higher dimensions, the same idea applies: the \(L_2\) ball has no axis-aligned corners corresponding to setting selected coordinates exactly to zero. Exact zeros can still occur in special cases, for example when the loss and constraint geometry align with an axis, but the smooth boundary does not create the same broad opportunity for corner contact.

The qualifier “usually” matters. Geometry is an explanation of a tendency, not an unconditional theorem that ridge coefficients can never equal zero. A coefficient can be zero because of the particular data or model structure. The contrast is that the lasso’s nonsmooth boundary makes coordinate zeros compatible with ordinary boundary contact, whereas ridge’s smooth boundary generally produces continuous shrinkage instead.

A Useful Geometric Derivation

The corner condition can also be read from the penalized objective. For \(f(\boldsymbol{\beta})+\lambda\sum_j|\beta_j|\), a coordinate \(\beta_j\neq0\) must satisfy \(\partial f/\partial\beta_j+\lambda\operatorname{sign}(\beta_j)=0\). A coordinate \(\beta_j=0\) can satisfy the optimum condition when \(\left|\partial f/\partial\beta_j\right|\leq\lambda\). This follows because the subgradient of \(|\beta_j|\) at zero is any value \(z_j\in[-1,1]\), so the equation \(\partial f/\partial\beta_j+\lambda z_j=0\) has a solution exactly when the derivative’s magnitude is at most \(\lambda\).

These coordinate conditions are the first-order conditions for a convex objective; they are sufficient for a global minimum, and strict convexity makes the minimum unique. The range of permissible balancing values at zero helps explain why an interval of loss-gradient values can result in a zero coefficient. Geometrically, this corresponds to the range of directions for which a contour can touch a lasso corner rather than an edge interior.

Common Mistakes and Takeaways

• Wrong: “The lasso constraint is a diamond, so every lasso solution has a zero.” Correction: A solution may lie in the interior of an edge, where both two-predictor coefficients are nonzero, as in the edge example.
• Wrong: “Ridge can never produce a zero coefficient.” Correction: Ridge does not have lasso’s axis-aligned corners, so exact zeros are not generally produced by the same mechanism; special data or geometry can still yield a zero.
• Wrong: “A coefficient is zero because the corresponding predictor has no relationship with the response.” Correction: The zero is the result of the fitted loss together with the penalty and the other coefficients. It does not, by itself, establish absence of a relationship.
• Wrong: “The constraint radius alone determines which coefficients vanish.” Correction: The loss contours’ location and shape determine where first contact occurs; the radius controls the size of the allowed region.