Why Elastic Net Geometry Matters
A regularization penalty does more than assign a numerical cost to large coefficients. Its shape affects where a model’s coefficients can be, and therefore which coefficients may be exactly zero. The L1 penalty has corners that can align with coordinate axes; the L2 penalty has a smooth circular shape. Elastic net combines these features: its contour retains axis corners when the L1 component is present, while its quadratic component adds curvature.
This geometry is useful for understanding a fitted model, not just for drawing a contour. In a constrained least-squares view, the best fit is sought among coefficient vectors inside a penalty contour. Where a contour meets a loss contour helps determine whether a coefficient is zero and how much the estimates are shrunk. The discussion below uses two coefficients so that the geometry can be seen directly.
This definition fixes the scaling used here. In the penalized least-squares objective, \(\lambda\geq0\) multiplies the penalty, so the penalty contribution is \(\lambda P_{\alpha}(\boldsymbol{\beta})\). The intercept is not penalized. Predictors are assumed to have been centered and scaled consistently before the coefficients are compared; changing their scales changes the meaning of the penalty on each coefficient.
From the L1 Diamond and L2 Circle to a Combined Contour
At \(\alpha=1\), the quadratic term disappears, and the constraint is \(|\beta_1|+|\beta_2|\leq t\). Its boundary is a diamond, with vertices on the coordinate axes. At \(\alpha=0\), the absolute-value term disappears, and the constraint is \((\beta_1^2+\beta_2^2)/2\leq t\), a circle of radius \(\sqrt{2t}\). For \(0<\alpha<1\), the boundary combines a diamond-like axis feature with curved arcs between the axes.
The intermediate contour still has a corner where it meets each axis if \(\alpha>0\), because the absolute value function is not differentiable at zero. It is less sharply diamond-shaped than a pure L1 contour because the quadratic term contributes curvature. When \(\alpha\) is reduced toward zero, the L1 contribution and the strength of the axis corner decrease. When \(\alpha\) is increased toward one, the contour becomes more dominated by the diamond.
| Penalty | Two-coefficient constraint | Geometric feature | Implication for estimates |
|---|---|---|---|
| L1, \(\alpha=1\) | \(|\beta_1|+|\beta_2|\leq t\) | Diamond with axis corners | Can yield exact zero coefficients |
| L2, \(\alpha=0\) | \((\beta_1^2+\beta_2^2)/2\leq t\) | Circle with smooth boundary | Shrinks coefficients but generally does not set them exactly to zero |
| Elastic net, \(0<\alpha<1\) | \(\alpha(|\beta_1|+|\beta_2|)+(1-\alpha)(\beta_1^2+\beta_2^2)/2\leq t\) | Curved contour with axis corners | Combines the possibility of exact zeros with additional quadratic shrinkage |
In the constrained view, imagine contours of the loss function expanding outward from its unconstrained minimum. The fitted coefficient vector is the point with the smallest loss that is feasible under the penalty constraint. If the contact point is on an axis corner, one coefficient is zero. If contact occurs on a smooth part of the boundary, both coefficients may be nonzero. A corner makes a zero coefficient possible, not inevitable: the location and orientation of the loss contours matter too.
Worked Example: Comparing Axis and Diagonal Points
Worked Example: Comparing Axis and Diagonal Points
A hypothetical two-predictor model uses \(\alpha=0.5\) and the constraint level \(t=1.5\). Determine whether the coefficient vectors \((2,0)\) and \((1,1)\) are inside, on, or outside the constraint contour. Then find the positive axis intercept and the positive diagonal point on the contour.
Method. Substitute each point into the stated elastic net penalty. For the intercepts, set the penalty equal to \(t\) and solve. This avoids treating the intermediate contour as either a pure diamond or a pure circle.
Here \(\alpha=0.5\) and \((1-\alpha)/2=0.25\), so \(P_{0.5}(\boldsymbol{\beta})=0.5(|\beta_1|+|\beta_2|)+0.25(\beta_1^2+\beta_2^2)\).
At \((2,0)\), the L1 contribution is \(0.5(2+0)=1\), and the quadratic contribution is \(0.25(2^2+0^2)=1\). Thus \(P_{0.5}(2,0)=1+1=2\). Since \(2>1.5\), this point is outside the constraint set.
At \((1,1)\), the L1 contribution is \(0.5(1+1)=1\), and the quadratic contribution is \(0.25(1^2+1^2)=0.5\). Thus \(P_{0.5}(1,1)=1+0.5=1.5\), so this point is on the contour.
For a positive axis intercept \((b,0)\), set \(0.5b+0.25b^2=1.5\). Multiplying by four gives \(b^2+2b-6=0\), with roots \(b=-1\pm\sqrt{7}\). The positive root is \(b=\sqrt{7}-1\), approximately \(1.646\).
For a positive diagonal point \((c,c)\), set \(0.5(2c)+0.25(2c^2)=1.5\). This simplifies to \(c+0.5c^2=1.5\), or \(c^2+2c-3=0\). The roots are \(1\) and \(-3\); the positive point is \((1,1)\), matching the direct check.
Check and common mistake. The direct substitution confirms that \((1,1)\) lies on the contour. A common mistake is to use only the L1 term and conclude that the axis intercept is \(3\), because \(0.5b=1.5\). That ignores the positive quadratic contribution; the correct intercept is \(\sqrt{7}-1\), which is smaller.
Why Corners Can Produce Zeros, and Curvature Shrinks Nonzero Coefficients
A short derivation in an orthogonal setting makes the roles of the two penalty components precise. Suppose the centered, scaled predictors are orthogonal, with \(X^\mathsf{T}X/n=I\), and define \(z=X^\mathsf{T}y/n\). Ignoring a constant that does not depend on the coefficients, the least-squares objective is then \(\frac{1}{2}\sum_j(\beta_j-z_j)^2\). With elastic net regularization, each coordinate minimizes \(\frac{1}{2}(\beta_j-z_j)^2+\lambda\alpha|\beta_j|+\frac{\lambda(1-\alpha)}{2}\beta_j^2\). The argument below applies coordinate by coordinate.
For a positive coefficient, differentiate the objective with respect to \(\beta_j\). Setting the derivative to zero gives \(\beta_j-z_j+\lambda\alpha+\lambda(1-\alpha)\beta_j=0\), so \(\bigl(1+\lambda(1-\alpha)\bigr)\beta_j=z_j-\lambda\alpha\). For a negative coefficient, the sign of the L1 derivative is reversed. Combining the two nonzero cases with the condition at zero gives the coordinate solution
At zero, the absolute-value term has subgradients between \(-1\) and \(1\). Therefore zero is optimal for coordinate \(j\) exactly when \(|z_j|\leq\lambda\alpha\). This threshold comes from the L1 component. For a coefficient that survives the threshold, the denominator \(1+\lambda(1-\alpha)\) reduces its magnitude when \(0\leq\alpha<1\). That additional shrinkage comes from the quadratic component. This derivation relies on the stated orthogonality and scaling assumptions; with correlated predictors, the coordinates do not separate in this way.
Worked Example: Separating Thresholding from Additional Shrinkage
In a hypothetical orthogonal two-predictor analysis, suppose \(z=(3,0.4)\), \(\lambda=1\), and \(\alpha=0.5\). Calculate the elastic net coefficients and compare them with the pure L1 and pure L2 estimates at \(\lambda=1\).
Method. Apply the coordinate solution just derived. The L1 threshold is \(\lambda\alpha=1(0.5)=0.5\), and the elastic net denominator is \(1+\lambda(1-\alpha)=1+0.5=1.5\).
For the first coordinate, \(|z_1|=3>0.5\), so it survives. Its estimate is \((3-0.5)/1.5=2.5/1.5=5/3\). For the second coordinate, \(|z_2|=0.4\leq0.5\), so its estimate is exactly zero. The elastic net result is \((5/3,0)\).
For pure L1 at \(\lambda=1\), the threshold is \(1\), and the denominator is \(1\). The estimates are \((3-1,0)=(2,0)\). For pure L2 at \(\lambda=1\), the threshold is zero and the denominator is \(2\), so the estimates are \((3/2,0.4/2)=(1.5,0.2)\).
Check and common mistake. The elastic net estimate has a nonzero first coefficient smaller than \(3\), and a second coefficient set to zero because \(0.4\leq0.5\). A common mistake is to apply the threshold but omit the denominator, reporting \((2.5,0)\). That would include the L1 adjustment but fail to include the L2 shrinkage.
Checking a Sparse Candidate at an Axis
The contour picture can also be expressed as a coordinate check. For a differentiable convex loss \(L(\boldsymbol{\beta})\), the objective is \(L(\boldsymbol{\beta})+\lambda P_\alpha(\boldsymbol{\beta})\). If a candidate has \(\beta_1=0\), the L1 part allows the loss gradient in that coordinate to be balanced by any value from \(-\lambda\alpha\) to \(+\lambda\alpha\). Thus the zero coordinate is compatible with optimality when \(|\partial L/\partial\beta_1|\leq\lambda\alpha\), evaluated at the candidate. For a nonzero coordinate, ordinary differentiation applies, including the ridge derivative \(\lambda(1-\alpha)\beta_j\).
When the loss is convex, satisfying these coordinate conditions in every coordinate is sufficient for a global minimum. If the objective is strictly convex, that minimum is unique. The following example checks both an axis coordinate and a nonzero coordinate, rather than inferring a solution from a sketch alone.
Worked Example: Verifying an Axis Solution
For a hypothetical orthogonal model, the loss up to an irrelevant constant is \(L(\boldsymbol{\beta})=\frac{1}{2}\{(\beta_1-0.6)^2+(\beta_2-3)^2\}\). Let \(\lambda=2\) and \(\alpha=0.4\). Test whether \((0,1)\) is the minimizer.
Method. Check the nonzero coordinate by differentiating there, and check the zero coordinate using the allowed L1 subgradient range. Here \(\lambda\alpha=2(0.4)=0.8\), and \(\lambda(1-\alpha)=2(0.6)=1.2\).
At \((0,1)\), the loss gradient in coordinate 2 is \(1-3=-2\). The L1 derivative is \(+0.8\), since \(\beta_2=1>0\), and the quadratic penalty derivative is \(1.2(1)=1.2\). The total derivative is \(-2+0.8+1.2=0\), as required.
In coordinate 1, the loss gradient is \(0-0.6=-0.6\). Because \(\beta_1=0\), it is sufficient that the magnitude of this gradient be no more than \(\lambda\alpha\): \(|-0.6|=0.6\leq0.8\). The zero coordinate therefore satisfies the optimality condition.
The loss is convex, and its quadratic term has identity Hessian; adding the nonnegative elastic net penalty preserves convexity. The verified conditions make \((0,1)\) a global minimizer. The loss is strictly convex, so the minimizer is unique.
Check and common mistake. The first coefficient is zero because its loss gradient falls within the L1 subgradient range, while the second balances all three derivative terms. A common mistake is to demand that the loss gradient for the zero coefficient equal zero. It need not: the L1 corner allows any balancing value in the interval from \(-0.8\) to \(0.8\).
Interpreting the Geometry Carefully
The combined contour explains the basic trade-off without implying that geometry alone determines the fitted model. The L1 component can create exact zeros because its contour has axis corners. The L2 component adds curvature and shrinks nonzero coefficients through a smooth quadratic contribution. The fitted point still depends on the loss surface, the data, predictor scaling, \(\lambda\), and \(\alpha\).
- A corner permits sparsity; it does not guarantee it. A loss contour may touch an elastic net contour away from the axes, producing nonzero coefficients.
- Shrinkage is not the same as selection. The quadratic term reduces coefficient magnitudes, but by itself does not create the L1 threshold that sets coefficients to zero.
- Do not compare contour levels without checking the penalty definition. The factor \(1/2\) on the quadratic term and the weights \(\alpha\) and \(1-\alpha\) determine the contour dimensions.
- Do not treat a two-dimensional drawing as a full solution method. With more predictors or correlated predictors, the loss contours and coefficient interactions affect where the optimum lies.
Check Your Understanding
Work through each problem by identifying the L1 and quadratic contributions separately.
- For \(\alpha=0.25\) and \(t=1.25\), determine whether \((2,0)\) is inside, on, or outside the contour, and whether \((1,1)\) is inside, on, or outside. (Answer: \(P(2,0)=2\), so \((2,0)\) is outside; \(P(1,1)=1.25\), so \((1,1)\) is on the contour.)
- An orthogonal model has \(z=(1.2,0.5)\), \(\lambda=1\), and \(\alpha=0.6\). Calculate both elastic net coefficients. (Answer: \((3/7,0)\), approximately \((0.4286,0)\).)
- For \(\alpha=0.3\) and \(t=0.9\), find the positive axis intercept \(b\) and the positive diagonal coordinate \(c\), where the points are \((b,0)\) and \((c,c)\). (Answer: \(b=3(\sqrt{15}-1)/7\), approximately \(1.2313\); \(c=(-3+6\sqrt{2})/7\), approximately \(0.7836\).)
- In an orthogonal model with \(z=(0.7,2)\), \(\lambda=2\), and \(\alpha=0.5\), calculate the elastic net estimates and state which coefficient is zero. (Answer: \((0,0.5)\); the first coefficient is zero.)
- For the loss \(L(\boldsymbol{\beta})=\frac{1}{2}\{(\beta_1-0.6)^2+(\beta_2-3)^2\}\), \(\lambda=2\), and \(\alpha=0.4\), check whether \((0,1)\) satisfies the coordinate optimality conditions. (Answer: Yes. The nonzero-coordinate derivative is \(-2+0.8+1.2=0\), and the zero-coordinate loss-gradient magnitude is \(0.6\leq0.8\).)