Lasso Versus Elastic Net on Correlated Feature Blocks

Why Correlated Features Create Different Coefficient Patterns

When several predictors carry similar information, a model can often make similar predictions using different allocations of their coefficients. Lasso tends to produce sparse solutions: it may retain one feature and assign zero coefficients to other features in the same correlated block. Elastic net adds a squared-coefficient penalty to the absolute-value penalty. That additional penalty tends to discourage a solution in which one correlated feature receives nearly all the coefficient while its companions receive none.

This is a tendency, not a guarantee. The lasso solution can depend on small differences in the predictors’ relationships with the response, and elastic net can still exclude features. The useful comparison is between coefficient patterns under stated data and penalty assumptions—not a rule that one method always selects one variable and the other always selects all of them.

Definition: A correlated feature block is a set of predictors whose values are strongly associated with one another. In the calculations below, predictors and the response are centered, each predictor has squared length $n$, and the intercept is not penalized. For a coefficient vector $\boldsymbol{\beta}$, the lasso minimizes $$ \frac{1}{2n}\lVert y-X\boldsymbol{\beta}\rVert_2^2+\lambda\sum_{j=1}^{p}|\beta_j|, $$ while elastic net minimizes $$ \frac{1}{2n}\lVert y-X\boldsymbol{\beta}\rVert_2^2+ \lambda\left(\alpha\sum_{j=1}^{p}|\beta_j|+ \frac{1-\alpha}{2}\sum_{j=1}^{p}\beta_j^2\right). $$ Here $\lambda\geq 0$ controls the overall penalty and $0\leq\alpha\leq1$ sets the balance between the absolute-value and squared-coefficient terms. Lasso is the case $\alpha=1$; for $0\leq\alpha<1$, elastic net includes a squared-coefficient term. This convention assumes squared-error loss and the stated predictor scaling; a change in loss scaling or predictor scaling changes the numerical penalty equations.

The absolute-value penalty can make coefficients exactly zero, which is the source of lasso’s sparsity. With correlated features, however, the data may not strongly distinguish which feature should carry the signal. The squared-coefficient term in elastic net makes unequal coefficient allocations more costly, all else equal. Its influence depends on the strength of the correlations, the response associations, and the penalty values.

Checking a Sparse Lasso Fit

For centered data, define $c_j=x_j^\mathsf{T}y/n$ and $G_{jk}=x_j^\mathsf{T}x_k/n$. For lasso, if a proposed solution has $\beta_j\ne0$, its coordinate condition is $c_j-\sum_kG_{jk}\beta_k=\lambda\,\operatorname{sign}(\beta_j)$. If $\beta_j=0$, the necessary condition is $\left|c_j-\sum_kG_{jk}\beta_k\right|\leq\lambda$. The latter condition checks whether an excluded feature’s residual association is small enough for zero to remain optimal. These conditions assume the squared-error objective above.

Worked Example: Lasso Selects One Feature From a Correlated Pair

Two centered, standardized predictors have correlation $0.9$, and their correlations with a centered response, expressed as $x_j^\mathsf{T}y/n$, are $0.80$ and $0.70$. Fit lasso with $\lambda=0.20$. Find and verify the solution. Then compare it with elastic net using $\lambda=0.20$ and $\alpha=0.50$. The method is to solve the active-coordinate equations and check the excluded-coordinate condition; this avoids assuming that both correlated features must be selected.

For lasso, first propose that feature 1 is active and feature 2 is zero. The active-coordinate equation gives $\beta_1=0.80-0.20=0.60$, so the candidate is $(0.60,0)$. The residual association for feature 2 is $0.70-0.90(0.60)=0.16$. Since $|0.16|\leq0.20$, the zero-coefficient condition holds. The active coefficient is positive and satisfies its equation, so this is a valid lasso solution. Its correlated companion is excluded.

For elastic net, $\lambda\alpha=0.10$ and $\lambda(1-\alpha)=0.10$. If both coefficients are positive, the coordinate equations are $1.10\beta_1+0.90\beta_2=0.80-0.10=0.70$ and $0.90\beta_1+1.10\beta_2=0.70-0.10=0.60$. Adding gives $2(\beta_1+\beta_2)=1.30$, so the sum is $0.65$. Subtracting gives $0.20(\beta_1-\beta_2)=0.10$, so the difference is $0.50$. Therefore $\beta_1=0.575$ and $\beta_2=0.075$. Both are positive, as assumed, and satisfy the two equations.

The lasso solution uses only the stronger feature, whereas elastic net retains both, giving the second feature a much smaller coefficient. A common mistake is to use the elastic-net equations with lasso’s penalty, or vice versa; checking the penalty contributions separately prevents that error.

Why Elastic Net Tends to Group Correlated Features

A short derivation makes the grouping tendency precise for active features with coefficients of the same sign. Let $r=y-X\boldsymbol{\beta}$ be the residual at an elastic-net solution. For two nonzero coefficients with the same sign, the absolute-value penalty has the same derivative for both coordinates. Subtract their two coordinate equations. The absolute-value terms cancel, leaving

\lambda(1-\alpha)(\beta_j-\beta_k) =\frac{(x_j-x_k)^\mathsf{T}r}{n}.

Taking absolute values and applying the Cauchy–Schwarz inequality gives $\lambda(1-\alpha)|\beta_j-\beta_k|\leq \lVert x_j-x_k\rVert_2\lVert r\rVert_2/n$. Because the predictors have squared length $n$ and correlation $\rho_{jk}$, $\lVert x_j-x_k\rVert_2^2=2n(1-\rho_{jk})$. Thus, when $\alpha<1$,

|\beta_j-\beta_k| \leq \frac{\sqrt{2(1-\rho_{jk})}\,\lVert r\rVert_2} {\sqrt{n}\lambda(1-\alpha)}.

As the correlation approaches one, this upper bound decreases, all else equal. The derivation explains why elastic net can encourage similar coefficients for highly correlated features. It does not say the coefficients must be equal: the bound also depends on the residual size and the penalty, applies to nonzero coefficients with the same sign, and becomes uninformative when $\alpha=1$. It is not a guarantee that every member of a correlated block will be retained.

Worked Example: Lasso Excludes a Third Feature That Elastic Net Retains

Three centered, standardized predictors have pairwise correlation $0.80$. Their response correlations are $0.80$, $0.78$, and $0.72$. Compare lasso with $\lambda=0.20$ to elastic net with $\lambda=0.20$ and $\alpha=0.50$. Calculate the coefficients and verify the proposed active sets. This method uses the full correlated-feature block rather than judging each feature by its response correlation alone.

For lasso, propose that features 1 and 2 are active and feature 3 is zero. Their equations are $\beta_1+0.80\beta_2=0.60$ and $0.80\beta_1+\beta_2=0.58$, after subtracting $\lambda=0.20$ from each positive response correlation. Adding the equations gives $1.80(\beta_1+\beta_2)=1.18$, so the sum is $59/90$. Subtracting gives $0.20(\beta_1-\beta_2)=0.02$, so the difference is $0.10$. Therefore $\beta_1=17/45\approx0.3778$ and $\beta_2=5/18\approx0.2778$.

The residual association for the excluded third feature is $0.72-0.80(59/90)=0.1956$ to four decimal places. Its absolute value is below $0.20$, so exclusion satisfies the lasso condition. Both proposed active coefficients are positive, verifying this lasso solution.

For elastic net, the absolute-value contribution is $0.10$ and the squared-coefficient contribution to each coordinate equation is $0.10\beta_j$. With all three coefficients proposed positive, the equations have diagonal entries $1.10$, off-diagonal entries $0.80$, and right-hand side $(0.70,0.68,0.62)$. The sum of the right-hand side is $2.00$; the common component of each coefficient is $2.00/2.70=20/27$. Subtracting the shared component gives $\beta_j=(c_j-0.10-16/27)/0.30$. Thus the coefficients are $29/81\approx0.3580$, $118/405\approx0.2914$, and $37/405\approx0.0914$. All are positive and sum to $20/27$, so the active-set assumption and coordinate equations are satisfied.

Here elastic net retains all three features, while lasso excludes the third even though its response correlation is $0.72$. Correlation with the other predictors matters because each coordinate condition is based on residual association after accounting for the fitted coefficients. The common mistake is to treat a large marginal response correlation as proof that a lasso coefficient must be nonzero.

Interpreting the Comparison Carefully

A coefficient pattern describes how a particular penalized fit allocates signal among the available predictors. It does not establish that a retained feature is causally important, uniquely informative, or more important than a feature with a smaller coefficient. For highly correlated predictors, multiple allocations can yield similar fitted values. The elastic-net grouping tendency is useful when retaining a set of related predictors is preferable to choosing a single representative, but the resulting coefficients still depend on the data and penalty.

The grouping argument also has boundaries. The displayed bound concerns two nonzero coefficients of the same sign. If a feature has zero coefficient, the active-coordinate equation used in the derivation does not apply to it. If two features are weakly correlated, the bound need not make their coefficients close. If $\alpha$ is near one, the squared-coefficient contribution is small, so the bound can be loose. Finally, retaining more features is not itself evidence of better predictive performance; that question requires an appropriate evaluation of predictions.

Key takeaway: Lasso’s absolute-value penalty can select a sparse representative from a correlated block. Elastic net’s additional squared-coefficient penalty often spreads weight across related predictors. Verify the pattern using the coordinate conditions, and describe it as a tendency rather than a guarantee. This comparison is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each problem, use the stated centered, standardized predictor setup and the squared-error objectives defined above.

Two predictors have correlation $0.80$, and response correlations $0.70$ and $0.58$. Under lasso with $\lambda=0.20$, check whether $(\beta_1,\beta_2)=(0.50,0)$ satisfies the coordinate conditions. (Answer: Yes; the excluded feature’s residual association is $0.58-0.80(0.50)=0.18$, which is at most $0.20$ in absolute value.)
For two predictors with correlation $0.90$ and response correlations $0.75$ and $0.65$, find the lasso solution at $\lambda=0.15$, assuming the second coefficient is zero, and verify that assumption. (Answer: $(0.60,0)$; the second feature’s residual association is $0.65-0.90(0.60)=0.11$, within the $0.15$ threshold.)
Two predictors have correlation $0.80$ and response correlations $0.70$ and $0.65$. For elastic net with $\lambda=0.20$ and $\alpha=0.50$, solve assuming both coefficients are positive. (Answer: $(\beta_1,\beta_2)=(0.35,0.25)$; the coordinate equations have diagonal $1.10$, off-diagonal $0.80$, and right-hand side $(0.60,0.55)$.)
For the elastic-net setup in question 3, calculate the upper bound on the coefficient difference if the residual norm is $8$ and $n=64$. (Answer: $\sqrt{2(1-0.80)}(8)/(\sqrt{64}(0.20)(0.50))=2.8284$ to four decimals; this bound is valid for active coefficients with the same sign.)
Three predictors have pairwise correlation $0.80$ and response correlations $(0.75,0.70,0.50)$. Under lasso with $\lambda=0.15$, propose that the first two are active and the third is zero. Find the active coefficients and check the excluded-feature condition. (Answer: $(\beta_1,\beta_2,\beta_3)=(13/45,1/9,0)$; the third residual association is $0.50-0.80(13/45+1/9)=0.2222$, which exceeds $0.15$, so this proposed solution is not valid and the third feature cannot remain excluded.)

===

What You'll Learn