Why Add an \(L_2\) Penalty to the Lasso?
When predictors are strongly correlated, several variables may carry nearly the same information about the outcome. The lasso can select one of them and set the others to zero, even when the data do not clearly distinguish which variable deserves the coefficient. This can make the selected model sensitive to small changes in the sample. The elastic net combines the lasso’s \(L_1\) penalty, which encourages sparse coefficients, with an \(L_2\) penalty, which discourages large or uneven coefficients.
The combination changes both the optimization problem and how the fitted coefficients behave across correlated predictors. It does not guarantee that every correlated variable will be retained, nor does it by itself establish that a selected predictor has a causal effect. Its value is as a regularized prediction and variable-selection method when correlated predictors make a purely lasso fit unstable or difficult to interpret.
Objective and Conventions
Consider a linear model with response vector \(y\in\mathbb{R}^n\), design matrix \(X\in\mathbb{R}^{n\times p}\), and coefficient vector \(\beta\in\mathbb{R}^p\). Center the response and each predictor, and standardize each predictor so that \(x_j'x_j/n=1\). The intercept is then handled separately (or absorbed by centering) and is not penalized. Standardization matters because otherwise the penalty can shrink coefficients differently simply because predictors use different measurement scales.
The factor \(1/2\) on the squared-error term and on the \(L_2\) penalty is a convention that makes the derivatives simple. Other presentations may rescale the tuning parameters; when comparing formulas, check the objective rather than comparing parameter labels alone. For \(\lambda>0\) and \(\gamma<1\), the \(L_2\) term makes the objective strictly convex in \(\beta\), so the minimizer is unique even if the columns of \(X\) are linearly dependent.
Deriving the Coordinate Update
Coordinate descent updates one coefficient at a time while holding the others fixed. For coordinate \(j\), define the partial residual \(r_j=y-\sum_{k\ne j}x_k\beta_k\). As \(\beta_j\) varies, the objective terms involving it are $$ \frac{1}{2n}\|r_j-x_j\beta_j\|_2^2+ \lambda\gamma|\beta_j|+\frac{\lambda(1-\gamma)}{2}\beta_j^2. $$ Expanding the squared norm and dropping terms constant in \(\beta_j\) gives a one-dimensional problem: $$ \frac{1}{2}\left(\frac{x_j'x_j}{n}+\lambda(1-\gamma)\right)\beta_j^2 -\left(\frac{x_j'r_j}{n}\right)\beta_j+\lambda\gamma|\beta_j|. $$ The \(L_1\) term has a kink at zero, so the minimizer is obtained by soft-thresholding the partial-residual correlation.
The formula makes the two roles of the penalty visible. The \(L_1\) component supplies a threshold: small partial-residual correlations are set exactly to zero. The \(L_2\) component adds to the denominator, shrinking nonzero updates toward zero. At \(\gamma=1\), the denominator is just the predictor’s scaled sum of squares and the update is the lasso soft-thresholding step. At \(\gamma=0\), there is no threshold, but the denominator includes the ridge adjustment.
Worked Example: One Coordinate Update
Suppose a standardized predictor has partial-residual correlation \(x_j'r_j/n=1.20\). Let \(\lambda=0.80\) and \(\gamma=0.50\). Assume the other coordinates are held fixed while this update is made.
The threshold is \(\lambda\gamma=0.80(0.50)=0.40\), and the denominator is \(1+\lambda(1-\gamma)=1+0.80(0.50)=1.40\). Since \(1.20>0.40\), $$ \beta_j\leftarrow\frac{S(1.20,0.40)}{1.40} =\frac{1.20-0.40}{1.40} =\frac{0.80}{1.40} \approx 0.5714. $$
The positive partial correlation survives the threshold, but the \(L_2\) term reduces its update through the denominator. Had the absolute correlation been at most \(0.40\), the updated coefficient would have been zero.
Why Correlated Predictors Tend to Move Together
The grouping effect refers to a tendency for elastic net to assign similar coefficients to highly correlated predictors, rather than selecting just one arbitrarily. A useful precise version applies when the predictors are standardized and the fitted coefficients for a pair are nonzero and have the same sign. It is not a universal guarantee that every correlated pair receives equal coefficients.
Let \(r=y-X\hat\beta\) be the residual at a minimizer. For a nonzero coefficient, the first-order condition for coordinate \(j\) is $$ \frac{x_j'r}{n}=\lambda\gamma\,\operatorname{sign}(\hat\beta_j) +\lambda(1-\gamma)\hat\beta_j. $$ For two nonzero coefficients with the same sign, subtract their conditions. The sign terms cancel: $$ \lambda(1-\gamma)(\hat\beta_j-\hat\beta_k) =\frac{(x_j-x_k)'r}{n}. $$ If both columns are standardized and their sample correlation is \(\rho_{jk}=x_j'x_k/n\), then \(\|x_j-x_k\|_2^2=2n(1-\rho_{jk})\). Applying the Cauchy–Schwarz inequality gives $$ |\hat\beta_j-\hat\beta_k| \leq \frac{\sqrt{2(1-\rho_{jk})}\,\|r\|_2} {\sqrt{n}\,\lambda(1-\gamma)}. $$ Thus, holding the residual scale and penalty fixed, the bound becomes smaller as the correlation approaches one. The argument requires \(\gamma<1\); for the lasso limit, this particular bound has no positive \(L_2\) term to control the coefficient difference.
Worked Example: Identical Predictors and Equal Sharing
Consider two identical standardized predictors, \(x_1=x_2=x\), with a positive total fitted contribution \(b=\beta_1+\beta_2\). For fixed \(b\), the fitted values depend only on the sum of the coefficients. Suppose also that the two coefficients are nonnegative, so their \(L_1\) contribution is \(|\beta_1|+|\beta_2|=b\), regardless of how \(b\) is split.
For any fixed sum \(b\), the elastic-net \(L_2\) contribution is minimized by splitting the sum equally. To see this, write \(\beta_2=b-\beta_1\): $$ \beta_1^2+\beta_2^2 =\beta_1^2+(b-\beta_1)^2 =2\left(\beta_1-\frac b2\right)^2+\frac{b^2}{2}. $$ The expression is smallest at \(\beta_1=b/2\), which also gives \(\beta_2=b/2\). When \(\lambda>0\) and \(\gamma<1\), the \(L_2\) part therefore favors equal sharing between these identical predictors.
With the lasso alone, the \(L_1\) contribution is unchanged by any nonnegative split summing to \(b\), so the objective does not choose between those splits. This example isolates the grouping mechanism: the \(L_2\) term resolves the lasso’s ambiguity by preferring a balanced allocation. With highly, rather than perfectly, correlated predictors, the fitted values are not identical under every split, but the grouping argument explains why elastic net can still favor more similar coefficients.
Comparing Elastic Net, Lasso, and Ridge
The following hypothetical example compares coefficients under the three objectives. It is constructed from standardized predictors with \(x_1'x_1/n=x_2'x_2/n=1\), correlation \(x_1'x_2/n=0.90\), and response correlations \(x_1'y/n=x_2'y/n=2.00\). The equal response correlations make the coefficient solutions symmetric. The predictor Gram matrix is positive definite because its eigenvalues are \(1.90\) and \(0.10\), so these summary quantities describe a nondegenerate two-predictor least-squares problem.
Worked Example: Coefficients Under Three Penalties
For elastic net, take \(\lambda=1.00\) and \(\gamma=0.50\). By symmetry, try \(\beta_1=\beta_2=b\). For either coordinate, the fitted correlation with \(x_j\) is \(b+0.90b=1.90b\). The positive-coefficient first-order condition is therefore $$ 2.00=1.90b+\lambda\gamma+\lambda(1-\gamma)b =1.90b+0.50+0.50b. $$ It follows that \(2.40b=1.50\), so \(b=0.6250\). The fitted elastic-net coefficients are \((0.6250,0.6250)\).
For comparison, use a lasso penalty \(\lambda=0.50\). If both coefficients are positive, its first-order condition is \(2.00=1.90b+0.50\), giving \(b=1.50/1.90\approx0.7895\). This solution is valid: both coefficients are positive, and the strictly convex least-squares term for this design makes it the unique minimizer.
For ridge regression, use an \(L_2\) penalty coefficient of \(0.50\) in the convention \(\|y-X\beta\|_2^2/(2n)+0.50\|\beta\|_2^2/2\). The first-order condition is \(2.00=1.90b+0.50b=2.40b\), so \(b=0.8333\). The coefficient comparison is:
| Method and penalty | Coefficient for \(x_1\) | Coefficient for \(x_2\) |
|---|---|---|
| Elastic net: \(\lambda=1.00,\ \gamma=0.50\) | 0.6250 | 0.6250 |
| Lasso: \(\lambda=0.50\) | 0.7895 | 0.7895 |
| Ridge: \(L_2\) coefficient \(0.50\) | 0.8333 | 0.8333 |
These values illustrate the specified objectives, not a claim that one method always produces smaller coefficients than another: the penalty settings differ. Elastic net applies both thresholding and additional shrinkage. In this symmetric example, all three methods retain both predictors, so it would be incorrect to say that lasso must select only one member of a correlated pair. The instability concern is clearest when correlations are high but the response correlations are not exactly symmetric.
Practical Interpretation and a Common Pitfall
Elastic net is useful when a sparse fit is desired but predictors occur in correlated groups. The \(L_1\) component can still set coefficients to zero, while the \(L_2\) component encourages stability and sharing among predictors that carry similar information. The grouping effect can make a selected set easier to interpret as a group of related variables, but it does not show that every selected predictor is independently informative, nor that any predictor causes the outcome.
The values of \(\lambda\) and \(\gamma\) jointly determine the fit. Choosing them is a model-selection task; for example, cross-validation can be used to compare predictive performance over candidate settings. Coefficients should be interpreted on the scale induced by the predictor standardization, and reported in original units only after accounting for any scaling transformation.
- Pitfall: “The predictors have correlation close to one, so elastic net must give them exactly equal coefficients.”
- Correction: The grouping result is a tendency, not a blanket equality claim. The bound applies to nonzero coefficients with the same sign and depends on the residual, sample size, penalty, and mixing parameter. Exact equality follows in special symmetric cases, such as identical predictors with an equal-sharing solution.
Check Your Understanding
Use the objective and conditions in this tutorial to answer the following.
- For \(\lambda=0.60\), what objective results when \(\gamma=1\), and what objective results when \(\gamma=0\)?
- A standardized predictor has partial-residual correlation \(0.25\). With \(\lambda=0.50\) and \(\gamma=0.60\), what is its coordinate update if the current partial residual is used?
- Why does the \(L_2\) term make the elastic-net minimizer unique when \(\lambda>0\) and \(\gamma<1\), even if predictors are linearly dependent?
- State the conditions used in the coefficient-difference bound for two predictors.
- Why does the identical-predictor example show a grouping tendency without proving that every highly correlated pair receives equal coefficients?