Why the Shape of the Constraint Matters
In genomic regression, the number of candidate predictors can exceed the number of samples. Many coefficient vectors may then fit the observed responses equally well, and a useful fit needs a rule for choosing among them. Lasso imposes a penalty on the sum of the absolute coefficient values. Its geometry helps explain a key feature of that choice: some fitted coefficients can be exactly zero, leaving a sparse model.
The essential picture is a set of least-squares loss contours meeting a constraint region. The least-squares contours describe how the fit changes as the coefficients move. The constraint region restricts how large their absolute values can be in total. Unlike the smooth circular or spherical regions associated with ridge, the lasso region has corners aligned with the coordinate axes. A contour can first touch the region at one of those corners, where one or more coefficients are zero.
Loss Contours and the Lasso Region
Assume a Gaussian-response least-squares model with design matrix \(X\), response vector \(y\), and coefficient vector \(\beta\). Here \(X\) has \(n\) rows and \(p\) columns. For clarity, suppose the response and predictor columns have been centered, and that any intercept is fitted separately and is not penalized. We use the objective
The least-squares part of the objective is \(RSS(\beta)/(2n)\), where \(RSS(\beta)=\|y-X\beta\|_2^2\). A loss contour is the set of coefficient vectors with the same least-squares loss. If \(X\) has full column rank and \(\hat\beta_{\mathrm{LS}}\) is its least-squares estimate, completing the square gives
Thus, contours are ellipses centered at the least-squares estimate (circles if \(X^\mathsf{T}X\) is proportional to the identity matrix). The lasso constraint formulation asks for the smallest least-squares loss subject to a limit \(t\) on the total absolute coefficient size:
For \(p=2\), the constraint region is a diamond with vertices on the two coordinate axes. For larger \(p\), it is a cross-polytope: its vertices are the signed coordinate vectors \(\pm t e_j\), where \(e_j\) has a 1 in coordinate \(j\) and zeros elsewhere. A point at such a vertex has only one nonzero coefficient. Edges and faces can contain more nonzero coefficients, but often still lie in coordinate subspaces where other coefficients are zero.
The two formulations express the same trade-off in different ways. Increasing \(\lambda\) gives more weight to the penalty; decreasing \(t\) makes the feasible region smaller. In a convex problem, a constrained solution can be supported by a penalty weight, with boundary and nonuniqueness details depending on the design and the chosen constraint. The geometry explains the trade-off without requiring the two parameterizations to have a simple one-to-one correspondence in every case.
Worked Examples: Reading the Geometry
Worked Example: A Contour Touches a Corner
Consider two orthogonal, standardized predictors for which the least-squares loss, apart from a constant, is \(L_0(\beta)=\tfrac12\|\beta-b\|_2^2\), with \(b=(2,0.5)\). Restrict the coefficients to \(\|\beta\|_1\leq1.5\). Find and verify the geometrically suggested solution.
The unconstrained least-squares center is \(b=(2,0.5)\). In the two-dimensional coefficient plane, its equal-loss contours are circles. The l1 constraint is a diamond with vertices \((1.5,0)\), \((-1.5,0)\), \((0,1.5)\), and \((0,-1.5)\). The circle centered at \(b\) first reaches the diamond at the positive horizontal vertex \(c=(1.5,0)\). Its squared distance from \(b\) is
To verify that no feasible point is closer, let \(q=b-c=(0.5,0.5)\). For any feasible \(\beta\), the l1 constraint implies \(q^\mathsf{T}\beta\leq0.5(|\beta_1|+|\beta_2|)\leq0.75=q^\mathsf{T}c\). Consequently,
So \(c=(1.5,0)\) is a constrained minimizer, with its second coefficient exactly zero. The zero arises because the first contact is at a coordinate-aligned corner, not because least squares itself favors a zero coefficient.
Worked Example: Contact Along an Edge Need Not Set a Coefficient to Zero
Keep the same orthogonal-predictor loss, but let \(b=(1.2,0.8)\) and constrain \(\|\beta\|_1\leq1\). The point \(c=(0.7,0.3)\) is on the positive edge of the diamond because \(0.7+0.3=1\). Its squared distance from the least-squares center is
To check that this is a closest feasible point, take \(q=b-c=(0.5,0.5)\). For every feasible \(\beta\), \(q^\mathsf{T}\beta\leq0.5\|\beta\|_1\leq0.5=q^\mathsf{T}c\). Expanding the squared distance as in the first example therefore shows \(\|b-\beta\|_2^2\geq\|b-c\|_2^2\). This solution lies along an edge, with both coefficients nonzero. The l1 boundary alone does not guarantee that any particular coefficient is zero; the location of contact matters.
Worked Example: A Corner in Three Coefficient Dimensions
For three orthogonal standardized predictors, suppose the least-squares center is \(b=(2,0.4,0.2)\), so the loss apart from a constant is \(\tfrac12\|\beta-b\|_2^2\). Impose \(\|\beta\|_1\leq1\). The candidate vertex is \(c=(1,0,0)\), with squared distance
For a verification, set \(q=b-c=(1,0.4,0.2)\). For every feasible \(\beta\), \(q^\mathsf{T}\beta\leq\max(1,0.4,0.2)\|\beta\|_1\leq1=q^\mathsf{T}c\). The squared-distance expansion then proves that \(c\) is a closest feasible point. The contact is at a vertex of the three-dimensional l1 region, leaving two of the three coefficients zero. In genomic applications the number of dimensions is much larger, but the same coordinate-alignment idea applies.
Why Sparsity Is Common, and What the Picture Does Not Prove
At a corner, moving away in a new coordinate direction uses part of the limited l1 budget immediately. A least-squares contour may reach the feasible region at the corner before it reaches an edge or face that allows that additional coefficient to be nonzero. This is the geometric source of lasso’s tendency to select a subset of predictors. It is not a guarantee that a particular gene is scientifically relevant, nor does a zero coefficient establish that its true effect is zero.
The geometry also requires care when \(p>n\). Then \(X^\mathsf{T}X\) cannot have full rank, so there is no unique least-squares center in coefficient space. Directions in the null space of \(X\) can change \(\beta\) without changing \(X\beta\), and loss contours are flat in those directions rather than ordinary bounded ellipses. The l1 constraint still limits the feasible coefficients, but geometric intuition alone does not establish that the minimizing coefficient vector is unique. Collinearity can also make the fitted response stable while individual coefficients are not.
The geometry is clearest after predictors are placed on comparable scales. If one predictor is measured in units that make its coefficient numerically small and another in units that make its coefficient large, the same l1 penalty treats those coefficient magnitudes differently. Predictor scaling before penalization is therefore part of interpreting which coordinates the constraint may suppress. The response scale and the chosen normalization of the loss also affect the numerical penalty value.
Comparison, Diagnostics, and Limitations
| Method | Constraint-region picture | Typical coefficient behavior |
|---|---|---|
| Lasso | l1 diamond or cross-polytope, with coordinate-aligned corners | Can produce exact zeros when contact is on an appropriate corner or face |
| Ridge | Smooth l2 circle or sphere | Shrinks coefficients toward zero, but generally does not set them exactly to zero |
| Unpenalized least squares | No coefficient-size constraint | Uses the least-squares minimizer; when \(p>n\), it need not be unique |
The l1 geometry is useful when a sparse coefficient representation is desired, but it is not a universal advantage. If correlated predictors carry similar information, different coefficient combinations may yield similar fits; which predictor is retained can depend on the data and design. Ridge instead uses a smooth constraint and is often useful for stabilizing coefficient estimates without selection. Neither geometric picture by itself resolves uncertainty about scientific interpretation or predictive performance.
Several checks help identify when a geometric explanation is being overextended:
- Check the scales. If predictors were not scaled comparably, apparent selection may reflect units as well as evidence in the data.
- Check the design rank and correlations. Flat or nearly flat loss contours indicate directions where coefficients are weakly distinguished by the observed responses.
- Do not infer sparsity from the penalty alone. A solution on an edge or higher-dimensional face can have several nonzero coefficients.
- Do not infer uniqueness from a diagram. In high dimensions or under collinearity, more than one coefficient vector may achieve the same minimum.
A frequent mistake is to say, “The lasso region has corners, so every lasso fit has many zero coefficients.” The correction is that corners make exact zeros geometrically possible and often encourage them, but the actual contour may touch an edge or face instead. The data, design, scaling, and constraint size determine the contact point. Geometry explains a mechanism; it does not replace checking the fitted solution.
Check Your Understanding
Answer each question, then compare with the solutions below.
- With two predictors, what shape is the l1 constraint region, and where are its vertices?
- In the second worked example, why does contact at \((0.7,0.3)\) not produce a zero coefficient?
- Why are least-squares contours not bounded ellipses in coefficient space when \(p>n\)?
- How does the usual ridge constraint shape differ from the lasso shape, and what is the usual implication for exact zeros?
- Why is it risky to interpret a lasso-selected gene as definitively more important than a correlated gene that was not selected?
Solutions. (1) The region is a diamond with vertices \((1,0)\), \((-1,0)\), \((0,1)\), and \((0,-1)\) when \(t=1\); in general the vertices are \((\pm t,0)\) and \((0,\pm t)\). (2) The point lies along an edge and both of its coordinates are nonzero; boundary contact does not necessarily mean corner contact. (3) When \(p>n\), \(X\) has a nontrivial null space, so some coefficient changes leave \(X\beta\) and the least-squares loss unchanged. (4) Ridge has a smooth l2 constraint region, while lasso has coordinate-aligned corners; ridge generally shrinks coefficients without making them exactly zero. (5) Correlated predictors can carry similar information, so selection among them may be unstable and reflect the observed design and data rather than a definitive difference in biological importance.