Why L1 Geometry Can Produce Sparse Fits
Lasso does more than reduce coefficient magnitudes: it can set some coefficients exactly to zero. In two dimensions, this variable-selection behavior has a geometric explanation. The constraint \(|\beta_1|+|\beta_2|\le t\) forms a diamond with corners on the coordinate axes. Least-squares contours are ellipses, or circles in a special case. If the smallest residual-sum-of-squares contour that reaches the feasible diamond first touches a corner, one coefficient is zero at the solution.
The geometry explains why sparsity can occur, not why it must occur in every fit. A contour may instead touch the interior of a diamond edge, where both coefficients are nonzero. The examples below use two predictors so that this distinction can be seen directly.
Reading the Contours and the Constraint
Let \(y\) be the response vector and let \(X\) contain two predictor columns. For coefficient vector \(\boldsymbol{\beta}=(\beta_1,\beta_2)^\mathsf{T}\), the residual sum of squares is \(RSS(\boldsymbol{\beta})=(y-X\boldsymbol{\beta})^\mathsf{T}(y-X\boldsymbol{\beta})\). A least-squares contour is the set of coefficient pairs that give the same RSS. If \(X^\mathsf{T}X\) is positive definite, these contours are ellipses centered at the unconstrained least-squares estimate. If the predictors are orthogonal and have equal column lengths, the contours are circles. When the columns are not orthogonal, the ellipses are tilted or stretched.
The constrained problem can be pictured by expanding the L1 diamond from the origin. If the unconstrained estimate is already inside the diamond, it is feasible and remains the constrained minimizer. If it is outside, the constrained solution is the feasible point with the smallest RSS. Equivalently, imagine starting with the lowest-RSS contour and expanding it until it first touches the diamond. A touch at a corner gives an axis solution; a touch on an edge away from its endpoints generally gives two nonzero coefficients.
This picture assumes the two coefficients are on comparable scales, as is commonly arranged by scaling predictors before fitting a penalized model. The intercept is normally treated separately rather than included in the L1 constraint. The shape argument concerns the coefficient plane; it does not, by itself, determine how predictors should be scaled or how a tuning parameter should be selected.
Why the First Contact Is the Constrained Minimum
The geometric rule follows from the definition of a contour and the minimization objective. Each contour encloses coefficient pairs with RSS no greater than its contour level, because least squares is a convex quadratic with its minimum at the unconstrained estimate. If a contour has not yet touched the feasible diamond, no feasible coefficient pair has RSS as low as that contour level. At the first contact, a feasible point attains the lowest contour level available within the diamond, so it minimizes RSS subject to the constraint.
The local shape of the feasible region at the contact matters. At a diamond corner, two boundary edges meet, and the corner lies on an axis. If the contour reaches that point before any other feasible point, the constrained optimum has an exact zero coefficient. At an edge contact away from a corner, both coordinates are generally nonzero. This is the geometric source of lasso sparsity in two dimensions; the corresponding ridge constraint is circular and has no corners to favor coordinate axes.
Worked Examples
Worked Example: An Orthogonal Fit Reaches a Corner
A hypothetical insurer models a centered claim-cost response using two centered, orthogonal risk features. Suppose \(X^\mathsf{T}X=I\), \(X^\mathsf{T}y=(3,1)^\mathsf{T}\), and \(y^\mathsf{T}y=20\). Minimize RSS subject to \(|\beta_1|+|\beta_2|\le 2\). Find the solution and verify it.
Method. Expand RSS, identify the center of its contours, and check the candidate corner against every feasible coefficient pair. This avoids assuming that both coefficients must remain nonzero.
Step 1: Expand the objective. The general expansion is \(RSS=y^\mathsf{T}y-2\boldsymbol{\beta}^\mathsf{T}X^\mathsf{T}y+\boldsymbol{\beta}^\mathsf{T}X^\mathsf{T}X\boldsymbol{\beta}\). Substituting the supplied values gives
The unconstrained minimum is at \((3,1)\), which is outside the diamond because \(3+1=4>2\). The contours are circles centered at \((3,1)\). The candidate nearest corner is \((2,0)\), which satisfies \(|2|+|0|=2\).
Step 2: Verify the candidate globally. For any feasible \((\beta_1,\beta_2)\), define \(\delta_1=\beta_1-2\) and \(\delta_2=\beta_2\). The RSS difference from the candidate is
Feasibility implies \(\beta_1+\beta_2\le |\beta_1|+|\beta_2|\le 2\), so \(\delta_1+\delta_2=\beta_1+\beta_2-2\le 0\). Both \(\delta_1^2+\delta_2^2\) and \(-2(\delta_1+\delta_2)\) are therefore nonnegative. Thus no feasible point has smaller RSS; because the quadratic is strictly convex, \((2,0)\) is the unique minimizer.
Step 3: Check the RSS. At \((2,0)\), RSS is \(10+(2-3)^2+(0-1)^2=12\). The result is a corner solution with \(\beta_2=0\). The common mistake this calculation avoids is treating the L1 bound as if it were a bound on the sum of squared coefficients.
Worked Example: Correlation Changes the Contour Shape
For a hypothetical two-feature pricing model, suppose \(X^\mathsf{T}X=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}\), \(X^\mathsf{T}y=(3,2)^\mathsf{T}\), and \(y^\mathsf{T}y=20\). Minimize RSS subject to \(|\beta_1|+|\beta_2|\le 2\). Determine whether the solution has a zero coefficient.
Method. Complete the square to locate the ellipse center, then compare the candidate corner with an arbitrary feasible point. This explicitly accounts for correlation between the predictors.
Step 1: Find the contour center. The center solves \(X^\mathsf{T}X\boldsymbol{\beta}=X^\mathsf{T}y\). The determinant of \(X^\mathsf{T}X\) is \(1-(0.5)^2=0.75\), and solving gives the unconstrained estimate \((8/3,2/3)\). Expanding the quadratic gives
The matrix is positive definite because its leading diagonal entry is positive and its determinant is \(0.75>0\). Thus the contours are tilted ellipses. The unconstrained estimate is outside the diamond since \(8/3+2/3=10/3>2\).
Step 2: Verify the corner. Consider the feasible corner \((2,0)\), and write \(\boldsymbol{\delta}=(\beta_1-2,\beta_2)^\mathsf{T}\). Subtracting its RSS from that at any feasible point yields
Indeed, multiplying the matrix by \((2,0)^\mathsf{T}-(8/3,2/3)^\mathsf{T}=(-2/3,-2/3)^\mathsf{T}\) gives \((-1,-1)^\mathsf{T}\), which produces the linear term shown. Feasibility gives \(\delta_1+\delta_2=\beta_1+\beta_2-2\le 0\). The quadratic term is nonnegative because the matrix is positive definite, and the remaining term is also nonnegative. Therefore \((2,0)\) is the unique constrained minimum.
Step 3: Check the objective value. Direct substitution gives \(RSS(2,0)=20-12+4=12\). The correlated predictors tilt the ellipses, but the corner is still the first feasible contact. The common mistake avoided here is to assume that correlation rules out a zero coefficient; it changes contour orientation, not the diamond’s corners.
Worked Example: An Edge Contact Keeps Both Coefficients
A hypothetical exposure model has orthogonal predictors with \(X^\mathsf{T}X=I\), \(X^\mathsf{T}y=(3,1)^\mathsf{T}\), and \(y^\mathsf{T}y=15\). Minimize RSS subject to \(|\beta_1|+|\beta_2|\le 3\). Does the lasso set either coefficient to zero?
Method. Use the circular contours and verify the proposed edge contact by comparing RSS at an arbitrary feasible point. This guards against interpreting every active L1 constraint as a sparse solution.
Step 1: Locate the unconstrained center and feasible set. Completing the square gives \(RSS=5+(\beta_1-3)^2+(\beta_2-1)^2\). The center \((3,1)\) is outside the diamond because its L1 norm is 4, greater than 3. A natural edge candidate is \((2.5,0.5)\), which has L1 norm 3 and lies between the vertices \((3,0)\) and \((0,3)\).
Step 2: Verify the candidate. Set \(\delta_1=\beta_1-2.5\) and \(\delta_2=\beta_2-0.5\). For a feasible point,
Since \(\beta_1+\beta_2\le|\beta_1|+|\beta_2|\le3\), we have \(\delta_1+\delta_2=\beta_1+\beta_2-3\le0\). Both terms on the right are nonnegative, establishing that the candidate is the unique minimizer.
Step 3: Check the RSS and conclusion. At the candidate, \(RSS=5+(2.5-3)^2+(0.5-1)^2=5+0.25+0.25=5.5\). Both coefficients are nonzero. The L1 constraint is active, but the contour touches the interior of an edge rather than a corner. The common mistake avoided is equating “constraint active” with “a coefficient selected out.”
A Common Geometry Trap
The shape of the constraint is essential. For the first example, the L1 feasible region is \(|\beta_1|+|\beta_2|\le2\), and the verified lasso solution is \((2,0)\). If the same bound were incorrectly treated as the L2 constraint \(\beta_1^2+\beta_2^2\le4\), the feasible region would be a circle. The least-squares center is \((3,1)\), whose distance from the origin is \(\sqrt{10}\). The closest point on that circle to the center is the radial projection
Both coordinates are nonzero. This is the wrong calculation for lasso because a circle is not the L1 constraint region. It illustrates why ridge’s smooth circular geometry does not create the same corner-based preference for exact zeros.
Check Your Understanding
For each problem, use the stated L1 constraint and interpret the geometric contact. Answers are given to check the calculation.
- With orthogonal equal-length predictors and unconstrained least-squares estimate \((5,2)\), minimize squared distance to that estimate subject to \(|\beta_1|+|\beta_2|\le4\). State the constrained estimate. (Answer: \((3.5,0.5)\), an edge contact with both coefficients nonzero.)
- With orthogonal equal-length predictors and center \((4,1)\), use the constraint \(|\beta_1|+|\beta_2|\le3\). State the constrained estimate. (Answer: \((3,0)\), a corner contact with \(\beta_2=0\).)
- For an orthogonal fit centered at \((2,1)\), the constraint is \(|\beta_1|+|\beta_2|\le1.5\). Give the constrained estimate and say whether it is sparse. (Answer: \((1.25,0.25)\); no coefficient is zero.)
- Why is a circular constraint not an appropriate geometric representation of the lasso’s L1 bound? (Answer: The L1 bound is \(|\beta_1|+|\beta_2|\le t\), whose two-dimensional region is a diamond with axis corners; a circle represents an L2 bound.)
- Suppose the unconstrained least-squares estimate is \((1,0.5)\) and the L1 bound is \(t=2\). What is the constrained estimate, and why? (Answer: \((1,0.5)\), unchanged, because its L1 norm is \(1.5\le2\), so it is already feasible.)