The Geometric View of Lasso Sparsity

Introduction

Lasso regression can produce a fitted model with some coefficients exactly equal to zero. The geometric view explains how this can happen: the lasso searches within a diamond-shaped region of coefficient values, while residual-sum-of-squares contours show how well each value fits the data. If the best contour that reaches the allowed region touches a corner or face, one or more coefficients are zero.

This is a geometric explanation, not a guarantee that every lasso fit is sparse. The contact point can instead lie along an edge where both coefficients are nonzero. Understanding the distinction helps when interpreting a coefficient diagram or comparing lasso with a method whose constraint boundary is smooth.

Definition: For a regression model with an unpenalized intercept and coefficient vector \(\boldsymbol{\beta}=(\beta_1,\ldots,\beta_p)\), the constrained lasso problem minimizes residual sum of squares subject to \(\sum_{j=1}^{p}|\beta_j|\leq t\), where \(t\geq 0\) is the constraint size. The set satisfying this inequality is the L1 constraint region. In two coefficient dimensions it is a diamond; in higher dimensions it is a cross-polytope.

Contours and the Constraint Region

For a geometric explanation, suppose the response and predictors have been centered, so the intercept can be handled separately, and let \(X\) be the design matrix. The residual sum of squares at coefficients \(\boldsymbol{\beta}\) is \(\operatorname{RSS}(\boldsymbol{\beta})=\lVert y-X\boldsymbol{\beta}\rVert^2\). If \(X\) has full column rank, its ordinary least-squares estimate \(\hat{\boldsymbol{\beta}}_{\mathrm{OLS}}\) is unique. Completing the square gives

\operatorname{RSS}(\boldsymbol{\beta}) = \operatorname{RSS}(\hat{\boldsymbol{\beta}}_{\mathrm{OLS}}) + (\boldsymbol{\beta}-\hat{\boldsymbol{\beta}}_{\mathrm{OLS}})^\mathsf{T} X^\mathsf{T}X (\boldsymbol{\beta}-\hat{\boldsymbol{\beta}}_{\mathrm{OLS}}).

To see why, write \(y-X\boldsymbol{\beta}=(y-X\hat{\boldsymbol{\beta}}_{\mathrm{OLS}})-X(\boldsymbol{\beta}-\hat{\boldsymbol{\beta}}_{\mathrm{OLS}})\), square its length, and expand. The cross-product is zero because the ordinary least-squares residual is orthogonal to every column of \(X\): \(X^\mathsf{T}(y-X\hat{\boldsymbol{\beta}}_{\mathrm{OLS}})=0\). The remaining added term is the quadratic form displayed above.

When \(X^\mathsf{T}X\) is positive definite, the equal-RSS contours are ellipses centered at the ordinary least-squares estimate. If the predictors are orthogonal and have matching scales, those ellipses are circles; correlated predictors generally tilt the ellipses. The lasso solution is the feasible coefficient vector on the lowest-RSS contour that reaches the L1 region. Equivalently, it is the point in the region closest to the ordinary least-squares center when distance is measured by the quadratic form \(X^\mathsf{T}X\).

In two dimensions, the L1 region has vertices on the coefficient axes. At a vertex, one coefficient is zero. In more dimensions, a corner or lower-dimensional face can set several coefficients to zero. The boundary also includes edges and faces whose points need not have any additional zero coefficients. Thus, the geometry explains why zeros are possible, but the location of the contact matters.

Key takeaway: Lasso sparsity is associated with contour contact at a corner or face of the L1 region. A contact along an edge can leave multiple coefficients nonzero; the diamond does not force every solution onto a vertex.

Worked Geometric Calculations

Worked Example: A Circular Contour Reaches an Axis

Suppose two centered predictors produce the residual-sum-of-squares function \(Q(\beta_1,\beta_2)=(\beta_1-2)^2+(\beta_2-0.2)^2\). Minimize it subject to \(|\beta_1|+|\beta_2|\leq 1\). Identify the solution and its minimum value. The method is to compare squared distances from the contour center \((2,0.2)\); this example avoids assuming that both coefficients must remain nonzero.

The feasible set is the diamond \(|\beta_1|+|\beta_2|\leq 1\). Consider the axis point \((1,0)\), which is feasible because \(|1|+|0|=1\). Its objective value is

$$ Q(1,0)=(1-2)^2+(0-0.2)^2=1+0.04=1.04. $$

To verify it is the global minimum, take any feasible \((\beta_1,\beta_2)\) and subtract this value:

Q(\beta_1,\beta_2)-Q(1,0) = (\beta_1-1)^2+\beta_2^2+2(1-\beta_1)-0.4\beta_2.

Feasibility implies \(\beta_1\leq 1\). If \(\beta_2\geq 0\), it also implies \(\beta_1\leq 1-\beta_2\), so \(2(1-\beta_1)-0.4\beta_2\geq 1.6\beta_2\geq 0\). If \(\beta_2<0\), then \(2(1-\beta_1)\geq 0\) and \(-0.4\beta_2>0\), so this same expression is positive. The squared terms are nonnegative in either case. Therefore \(Q(\beta_1,\beta_2)\geq 1.04\) for every feasible point. The solution is \((1,0)\), and the second coefficient is exactly zero. The calculation confirms both feasibility and optimality.

Worked Example: A Tilted Ellipse Still Touches an Axis

Consider the quadratic objective \(Q(\beta_1,\beta_2)=d_1^2+d_1d_2+d_2^2\), where \(d_1=\beta_1-2\) and \(d_2=\beta_2-0.2\). Minimize it subject to \(|\beta_1|+|\beta_2|\leq 1\). The cross-product makes the contours tilted ellipses rather than circles. Use a quadratic expansion to check whether \((1,0)\) remains optimal.

The quadratic form has matrix \(H=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}\). Its eigenvalues are \(0.5\) and \(1.5\), both positive, so the contours are ellipses and the objective is strictly convex. At \((1,0)\), \(d_1=-1\) and \(d_2=-0.2\), giving

$$ Q(1,0)=(-1)^2+(-1)(-0.2)+(-0.2)^2=1+0.2+0.04=1.24. $$

For an arbitrary feasible point, set \(h_1=\beta_1-1\) and \(h_2=\beta_2\). Expanding around \((1,0)\) gives

Q(\beta_1,\beta_2)-Q(1,0) = h_1^2+h_1h_2+h_2^2 + 2.2(1-\beta_1)-1.4\beta_2.

The quadratic term is nonnegative: \(h_1^2+h_1h_2+h_2^2=\tfrac12(h_1^2+h_2^2)+\tfrac12(h_1+h_2)^2\). Also, feasibility gives \(\beta_1\leq 1\). If \(\beta_2\geq 0\), then \(1-\beta_1\geq\beta_2\), so \(2.2(1-\beta_1)-1.4\beta_2\geq 0.8\beta_2\geq 0\). If \(\beta_2<0\), both \(2.2(1-\beta_1)\geq0\) and \(-1.4\beta_2>0\). Thus every feasible point has objective at least \(1.24\). The minimizer is \((1,0)\). Even with tilted contours, the first contour to reach the diamond touches an axis.

Worked Example: Tightening the Constraint Changes the Contact

Let \(Q(\beta_1,\beta_2)=(\beta_1-1.6)^2+(\beta_2-0.6)^2\), with constraint \(|\beta_1|+|\beta_2|\leq t\). Compare the solutions for \(t=2\) and \(t=1\). This calculation shows why a diamond can yield a point with both coefficients nonzero at one constraint size and an axis point at another.

For \(t=2\), consider \((1.5,0.5)\), which is feasible because \(1.5+0.5=2\). Its objective is

$$ Q(1.5,0.5)=(-0.1)^2+(-0.1)^2=0.02. $$

For any feasible \((\beta_1,\beta_2)\), expansion around this point gives

Q(\beta_1,\beta_2)-Q(1.5,0.5) = (\beta_1-1.5)^2+(\beta_2-0.5)^2 + 0.2(2-\beta_1-\beta_2).

Since \(\beta_1+\beta_2\leq|\beta_1|+|\beta_2|\leq2\), every term on the right is nonnegative. Thus \((1.5,0.5)\) is the minimizer for \(t=2\), and both coefficients are nonzero.

For \(t=1\), the feasible axis point \((1,0)\) has objective \(Q(1,0)=(-0.6)^2+(-0.6)^2=0.72\). For any point in this smaller diamond,

Q(\beta_1,\beta_2)-Q(1,0) = (\beta_1-1)^2+\beta_2^2 + 1.2(1-\beta_1-\beta_2).

Feasibility implies \(\beta_1+\beta_2\leq|\beta_1|+|\beta_2|\leq1\), so the final term and both squared terms are nonnegative. Therefore \((1,0)\) minimizes the objective at \(t=1\), with \(\beta_2=0\). The verified solutions are \((1.5,0.5)\) at \(t=2\) and \((1,0)\) at \(t=1\).

From Constraint Size to Lasso Sparsity

A common penalized form of the lasso minimizes \(\operatorname{RSS}(\boldsymbol{\beta})+\lambda\sum_j|\beta_j|\), with \(\lambda\geq0\). Increasing \(\lambda\) makes large total absolute coefficient size more costly; the constrained form instead makes the allowed size \(t\) smaller. These are two ways to express the tradeoff between fit and coefficient size. For suitable choices, a penalized solution corresponds to a constrained solution at its attained L1 size.

The geometric reasoning does not require solving for that correspondence. It requires identifying the feasible region, locating the residual contours, and finding their first contact with the region. At an axis corner in two dimensions, one coefficient is zero. If the first contact is in the interior of an edge, neither need be zero. In higher dimensions, contact with a face can set the coefficients corresponding to the face’s fixed-zero coordinates to zero.

The intercept is ordinarily left outside the L1 constraint, so it does not contribute a coordinate to the diamond. Predictor scaling also matters: changing a predictor’s units changes the coefficient needed for the same fitted effect and therefore changes the geometry of the constraint in coefficient coordinates. Comparisons based on the diamond are meaningful only with the scaling convention understood.

Common Mistakes

Assuming every lasso fit must have a zero. Example 3 has a feasible optimum with two nonzero coefficients when \(t=2\). Sparsity depends on where the contour first touches the constraint boundary.
Confusing the contour center with the constrained solution. The ordinary least-squares estimate minimizes RSS without a coefficient constraint. If it lies outside the diamond, it is not feasible for the constrained lasso problem.
Treating tilted contours as circles. Correlation and predictor scales affect the shape and orientation of the RSS contours. The contact point must be assessed using the actual objective, not a presumed Euclidean distance.
Including the intercept in the penalty by default. In the usual setup, the intercept is unpenalized; the diamond describes the slope coefficients.

For the SOA Exam SRM (Statistics for Risk Modeling) syllabus, the useful interpretation is precise: an L1 boundary has corners and faces aligned with coefficient axes, and a contour that reaches one of these can produce exact zeros. The geometry explains the mechanism while leaving open whether a particular data set will yield such a contact.

Check Your Understanding

For each problem, use the L1 constraint and the objective’s contour geometry. Verify feasibility as well as the proposed optimum.

For \(Q(\beta_1,\beta_2)=(\beta_1-1.4)^2+(\beta_2-0.4)^2\) subject to \(|\beta_1|+|\beta_2|\leq1\), which point minimizes \(Q\), and what is the minimum value? (Answer: \((1,0)\), with minimum \(0.32\).)
For \(Q(\beta_1,\beta_2)=(\beta_1-1.5)^2+(\beta_2-0.5)^2\) subject to \(|\beta_1|+|\beta_2|\leq1.5\), identify the minimizer and state whether it is an axis point. (Answer: \((1.25,0.25)\); it is not an axis point.)
For \(Q(\beta_1,\beta_2)=(\beta_1-1.6)^2+(\beta_2-0.6)^2\) subject to \(|\beta_1|+|\beta_2|\leq2\), compare the objective at \((1.5,0.5)\) and \((2,0)\). Which is lower, and by how much? (Answer: \(0.02\) at \((1.5,0.5)\), which is lower by \(0.50\) than the value \(0.52\) at \((2,0)\).)
For \(Q(\beta_1,\beta_2)=(\beta_1-2)^2+(\beta_2-0.2)^2\) subject to \(|\beta_1|+|\beta_2|\leq1\), verify that \((1,0)\) has a lower objective than \((0,1)\). What are the two values, and which point is the constrained minimizer? (Answer: \(Q(1,0)=1.04\), \(Q(0,1)=4.64\), and the constrained minimizer is \((1,0)\).)
Suppose an L1 constraint in three coefficient dimensions has its optimum on a face where \(\beta_2=0\), while \(\beta_1\) and \(\beta_3\) are nonzero. How many coefficients are zero, and does this contact imply that all coefficients are zero? (Answer: One coefficient is zero, and no, the other two are nonzero.)

What You'll Learn