Elastic-Net Penalties for Correlated Predictors

Why Combine Lasso and Ridge?

When predictors carry overlapping information, a lasso fit can select one and discard another, even when both are scientifically relevant. Ridge regularization, by contrast, tends to shrink coefficients continuously rather than set them exactly to zero. The elastic net combines these two behaviors: its absolute-value component supports sparse fits, while its quadratic component discourages a fit from assigning very different coefficients to predictors whose contributions to the likelihood are similar.

The key is not that correlation automatically forces two predictors to be selected together. Rather, when two active coefficients have the same sign, the quadratic part of the penalty makes a large difference between them costly. The strength of that effect depends on the mixing parameter, the penalty level, and how similar the predictors’ contributions are to the likelihood.

Definition and Objective

Let \(\ell(\beta)\) be the log-likelihood of a GLM as a function of the \(p\)-vector of penalized coefficients \(\beta=(\beta_1,\ldots,\beta_p)^\mathsf{T}\). For a tuning parameter \(\lambda\geq 0\) and mixing parameter \(\alpha\in[0,1]\), define the elastic-net estimate as a minimizer of the following objective:

Here the negative log-likelihood is divided by \(n\), so the penalty is expressed on the same objective scale. This is one common parameterization. The equivalent form with separate nonnegative coefficients on the \(L_1\) and squared \(L_2\) terms uses \(\lambda_1=\lambda\alpha\) and \(\lambda_2=\lambda(1-\alpha)\). In this tutorial, coefficients included in \(\beta\) are the ones being penalized; any unpenalized terms are kept outside that penalty.

Penalty Geometry

A useful way to compare penalties is to examine their constraint sets: the set of coefficient vectors whose penalty is no greater than a fixed number \(t\). For the lasso, the two-dimensional set \(|\beta_1|+|\beta_2|\leq t\) is a diamond with corners on the coordinate axes. Those corners matter because a likelihood contour can touch the constraint at a point where one coefficient is exactly zero. The ridge constraint \((\beta_1^2+\beta_2^2)/2\leq t\) is a circle, with no corners favoring exact zeros.

For \(0<\alpha<1\), the elastic-net constraint is \(\alpha(|\beta_1|+|\beta_2|)+(1-\alpha)(\beta_1^2+\beta_2^2)/2\leq t\). It combines the axis-related corners from the absolute values with curvature from the quadratic term. Thus it can still encourage zero coefficients, while also penalizing the overall size and unevenness of the coefficient vector.

Why the Quadratic Component Can Group Predictors

The ridge component favors distributing a fixed total coefficient across two predictors rather than concentrating it in just one. The likelihood determines whether those allocations fit equally well; when predictors are identical, it cannot distinguish allocations that give the same linear predictor. The following calculation separates the penalty comparison from the fit comparison.

A Grouping Bound for Same-Sign Coefficients

The next result makes the grouping idea precise for a convex differentiable loss. It is conditional: it applies to two active coefficients with the same sign, and it controls their difference through the difference between the loss gradients for those predictors.

To prove the claim, the first-order optimality condition for a convex objective says that zero belongs to its subgradient at a minimizer. At a nonzero coefficient, the absolute-value term is differentiable, with derivative equal to the coefficient’s sign. Therefore, for each of \(j\) and \(k\), \(\partial_j L(\hat\beta)+\lambda(1-\alpha)\hat\beta_j+\lambda\alpha\operatorname{sign}(\hat\beta_j)=0\). Subtract the corresponding equation for \(k\). Since the coefficients have the same sign, their sign terms cancel, giving \(\lambda(1-\alpha)(\hat\beta_j-\hat\beta_k)=-(\partial_jL(\hat\beta)-\partial_kL(\hat\beta))\). The factor \(\lambda(1-\alpha)\) is positive because \(\lambda>0\) and \(\alpha<1\); taking absolute values and dividing proves the result. \(\square\)

This identity explains the role of the quadratic component: for a fixed gradient difference, increasing \(\lambda(1-\alpha)\) decreases the permitted difference between same-sign active coefficients. To connect this to predictor similarity, consider a canonical GLM with loss \(L(\beta)=n^{-1}\sum_{i=1}^n\{b(\eta_i)-y_i\eta_i\}\), where \(\eta_i\) is the linear predictor and \(b'(\eta_i)\) is the fitted mean. For design columns \(x_j\) and \(x_k\), \(\partial_jL-\partial_kL=n^{-1}\sum_i(x_{ij}-x_{ik})r_i\), where \(r_i=b'(\eta_i)-y_i\). By the Cauchy–Schwarz inequality, \[ |\partial_jL-\partial_kL| \leq \frac{\|x_j-x_k\|_2\|r\|_2}{n}. \] Thus columns that are close in Euclidean distance have similar gradients at the fitted parameter, provided the residual norm is controlled.

Uniqueness and Limits of the Grouping Interpretation

The quadratic term also has a useful consequence for uniqueness. If \(L\) is convex, \(\lambda>0\), and \(\alpha<1\), the elastic-net objective is strictly convex and therefore has at most one minimizer. Indeed, the term \(\lambda(1-\alpha)\|\beta\|_2^2/2\) is strictly convex, and adding a convex function to a strictly convex function preserves strict convexity. A strictly convex function cannot have two distinct minimizers: at their midpoint its value would be strictly smaller than the common minimum, a contradiction.

This uniqueness statement concerns the penalized coefficient vector under the stated objective and assumptions. It does not mean that every correlated predictor will receive a nonzero coefficient. The lasso component can still set coefficients to zero, and the grouping bound says nothing about a pair with opposite signs or with either coefficient equal to zero. When \(\alpha=1\), the quadratic component vanishes, so the bound’s denominator is zero and the argument provides no lasso grouping guarantee.

For exactly identical columns, there is an especially direct interpretation. Swapping their coefficients leaves the likelihood unchanged. If the minimizer is unique, swapping must leave the minimizing vector unchanged, so the two coefficients are equal. More generally, highly correlated columns are not identical: the likelihood can favor unequal coefficients, and the ridge component only moderates that difference. The elastic net is therefore a compromise, not a rule that correlated predictors must always enter or leave the model as a block.

Common Mistake

A tempting but incorrect argument is: “The lasso chooses one of two correlated predictors, so the elastic net must always choose both.” The penalty comparison does not establish that conclusion. The elastic net makes splitting a fixed coefficient total less costly in its quadratic component, but the likelihood can still favor one predictor, the estimated coefficients can have opposite signs, or a coefficient can remain zero. The appropriate correction is to treat grouping as a tendency supported by the same-sign gradient bound, not as a guarantee of joint selection.