Ridge And Lasso As Elastic Net Limits

Introduction

Elastic net combines two familiar penalties: ridge regression’s squared-coefficient penalty and lasso’s absolute-value penalty. Its mixing parameter, $\alpha$, determines the relative contribution of those penalties. At the endpoints, the objective becomes ridge or lasso; between the endpoints, both penalties contribute. Recognizing these limits is useful when interpreting a fitted model and when checking a coefficient calculation.

This tutorial uses a common scaled least-squares objective and then examines a setting in which the coefficients can be calculated directly. The worked examples show what changes—and what does not—when $\alpha$ moves from zero to one.

Definition: For a response vector $y$, design matrix $X$, intercept $\beta_0$, and coefficient vector $\beta=(\beta_1,\ldots,\beta_p)$, the elastic-net estimate minimizes $$ \frac{1}{2n}\sum_{i=1}^n\left(y_i-\beta_0-\sum_{j=1}^p x_{ij}\beta_j\right)^2 +\lambda\left\{\alpha\sum_{j=1}^p|\beta_j|+\frac{1-\alpha}{2}\sum_{j=1}^p\beta_j^2\right\}, $$ where $\lambda\geq 0$ is the penalty strength and $0\leq\alpha\leq1$ is the mixing parameter. The intercept is not penalized. The displayed scaling is part of the convention: changing the scaling of either penalty changes how a numerical value of $\lambda$ corresponds to a fit.

The Two Endpoints

When $\alpha=0$, the absolute-value term has coefficient zero, while the squared-coefficient term remains. The objective reduces to

\frac{1}{2n}\sum_{i=1}^n\left(y_i-\beta_0-\sum_{j=1}^p x_{ij}\beta_j\right)^2 +\frac{\lambda}{2}\sum_{j=1}^p\beta_j^2.

This is ridge regression under the stated scaling. When $\alpha=1$, the squared-coefficient term has coefficient zero, leaving the absolute-value penalty:

\frac{1}{2n}\sum_{i=1}^n\left(y_i-\beta_0-\sum_{j=1}^p x_{ij}\beta_j\right)^2 +\lambda\sum_{j=1}^p|\beta_j|.

This is lasso regression. In between, $0<\alpha<1$, the objective includes both penalties. The value of $\alpha$ describes the mix, not the overall penalty strength: $\lambda$ still multiplies the combined penalty. The endpoints identify the penalty forms, but coefficient values also depend on the data, predictor scaling, and chosen $\lambda$.

Mixing parameter	Penalty that remains	Resulting method
$\alpha=0$	$\frac{\lambda}{2}\sum_j\beta_j^2$	Ridge regression
$0<\alpha<1$	$\lambda\{\alpha\sum_j\|\beta_j\|+\frac{1-\alpha}{2}\sum_j\beta_j^2\}$	Elastic net with both penalties
$\alpha=1$	$\lambda\sum_j\|\beta_j\|$	Lasso regression

A Direct Formula for Orthogonal Predictors

A particularly clear coefficient comparison is available when the predictors are centered and orthogonal, with each column scaled so that $X^\mathsf{T}X/n=I$. The response is centered as well, so the fitted intercept is zero. Define $z_j=x_j^\mathsf{T}y/n$, the unpenalized least-squares coefficient for predictor $j$ in this design. No distributional assumption is needed to minimize the stated least-squares objective. If the errors are additionally modeled as independent, mean-zero, constant-variance normal errors, that assumption gives the usual normal-error regression interpretation; it is not needed for the calculation below.

With orthogonal columns, the part of the objective that depends on coefficient $\beta_j$ can be written, up to a constant independent of $\beta_j$, as

\frac{1}{2}(\beta_j-z_j)^2 +\lambda\left\{\alpha|\beta_j|+\frac{1-\alpha}{2}\beta_j^2\right\}.

This reduces the calculation to minimizing one function for each coefficient. The absolute-value term has a corner at zero, so the minimizer may be exactly zero. Define the soft-thresholding function $S(z,t)=\operatorname{sign}(z)\max(|z|-t,0)$, where $t\geq0$. Then the coefficient is

\widehat{\beta}_j= \frac{S(z_j,\lambda\alpha)}{1+\lambda(1-\alpha)}.

Why the formula holds: For a positive candidate coefficient, differentiating the one-coordinate objective gives $(\beta_j-z_j)+\lambda\alpha+\lambda(1-\alpha)\beta_j=0$. Thus $\beta_j=(z_j-\lambda\alpha)/[1+\lambda(1-\alpha)]$, which is positive only if $z_j>\lambda\alpha$. For a negative candidate, the absolute-value derivative is $-1$, giving $\beta_j=(z_j+\lambda\alpha)/[1+\lambda(1-\alpha)]$, which is negative only if $z_j<-\lambda\alpha$. If $-\lambda\alpha\leq z_j\leq\lambda\alpha$, zero satisfies the minimization condition at the corner of the absolute-value term. These three cases give the soft-thresholding formula. The denominator is positive because $\lambda\geq0$ and $0\leq\alpha\leq1$.

At $\alpha=0$, the threshold is zero and the denominator is $1+\lambda$, so $\widehat{\beta}_j=z_j/(1+\lambda)$: ridge shrinks each orthogonal-design coefficient toward zero without setting a nonzero $z_j$ exactly to zero. At $\alpha=1$, the denominator is one and the threshold is $\lambda$, so $\widehat{\beta}_j=S(z_j,\lambda)$: lasso can set coefficients to zero. At an intermediate value, the threshold is $\lambda\alpha$, and the denominator is $1+\lambda(1-\alpha)$. Both features matter.

Worked Example: Calculating the Ridge and Lasso Endpoints

Suppose three centered, scaled, orthogonal predictors have least-squares coefficients $z=(1.2,-0.6,0.2)$. Use $\lambda=0.5$ to find the ridge and lasso coefficients. Round final coefficients to four decimal places.

Method and setup. Apply the orthogonal-design formula with the same $\lambda$ at each endpoint. The inputs are $z_1=1.2$, $z_2=-0.6$, and $z_3=0.2$.

Ridge, $\alpha=0$. The threshold is $\lambda\alpha=0.5(0)=0$, and the denominator is $1+0.5(1-0)=1.5$. Therefore,

\widehat{\beta}=\left(\frac{1.2}{1.5},\frac{-0.6}{1.5},\frac{0.2}{1.5}\right) =(0.8000,-0.4000,0.1333).

Lasso, $\alpha=1$. The threshold is $\lambda\alpha=0.5$, and the denominator is $1+0.5(1-1)=1$. The thresholded numerators are $1.2-0.5=0.7$, $-0.6+0.5=-0.1$, and zero because $|0.2|\leq0.5$. Thus $\widehat{\beta}=(0.7000,-0.1000,0.0000)$.

Check and trap avoided. The third ridge coefficient is nonzero because ridge only divides $z_3$ by $1.5$. The third lasso coefficient is zero because its absolute unpenalized coefficient is below the lasso threshold $0.5$. Applying that threshold to the ridge calculation would incorrectly turn the ridge coefficient into zero.

What an Intermediate Mixing Value Does

The orthogonal formula separates the effects of the two parts of the penalty. Increasing $\alpha$ increases the threshold $\lambda\alpha$, which can remove a coefficient. At the same time, it decreases the ridge component in the denominator. Consequently, an intermediate coefficient is not generally a simple weighted average of the ridge and lasso estimates. It must be calculated from both the threshold and the denominator.

Worked Example: An Intermediate Elastic-Net Fit

Use the same orthogonal predictors and $z=(1.2,-0.6,0.2)$, but now take $\lambda=0.5$ and $\alpha=0.4$. Calculate all three coefficients and check which are active. Round final coefficients to four decimal places.

Method. The threshold is $\lambda\alpha=0.5(0.4)=0.2$. The denominator is $1+\lambda(1-\alpha)=1+0.5(0.6)=1.3$.

For $z_1=1.2$, the thresholded numerator is $1.2-0.2=1.0$, so $\widehat{\beta}_1=1.0/1.3=0.7692$. For $z_2=-0.6$, it is $-0.6+0.2=-0.4$, so $\widehat{\beta}_2=-0.4/1.3=-0.3077$. For $z_3=0.2$, $|z_3|=0.2$ is exactly the threshold, so $\widehat{\beta}_3=0$.

The result is $\widehat{\beta}=(0.7692,-0.3077,0.0000)$. The signs of the two nonzero estimates match their corresponding $z_j$ values. The third coefficient is at the threshold boundary and is zero. As a check, the first two values satisfy the positive- and negative-coefficient equations from the derivation: $1.3(0.7692)$ is approximately $1.0=1.2-0.2$, and $1.3(-0.3077)$ is approximately $-0.4=-0.6+0.2$.

Trap avoided. Keeping the lasso threshold but forgetting the denominator would give $1.0$ and $-0.4$, not the elastic-net estimates. The squared-coefficient penalty still contributes at $\alpha=0.4$, so the denominator cannot be dropped.

Worked Example: Finding When a Coefficient Becomes Zero

A standardized orthogonal predictor has $z=0.30$, and $\lambda=0.50$. Determine the values of $\alpha$ for which its elastic-net coefficient is zero. Then calculate the coefficient at $\alpha=0.40$. Round the nonzero final coefficient to four decimal places.

Method. The coefficient is zero when $|z|\leq\lambda\alpha$. Substituting the given values gives $0.30\leq0.50\alpha$, or $\alpha\geq0.60$. Since the mixing parameter is restricted to $0\leq\alpha\leq1$, the coefficient is zero for $0.60\leq\alpha\leq1$, including the boundary.

At $\alpha=0.40$, the threshold is $0.50(0.40)=0.20$, so the thresholded numerator is $0.30-0.20=0.10$. The denominator is $1+0.50(1-0.40)=1+0.30=1.30$. Therefore, $\widehat{\beta}=0.10/1.30=0.0769$.

Check and trap avoided. At $\alpha=0.60$, the threshold equals $0.30$, so the coefficient is exactly zero; it does not require a strict inequality. At $\alpha=0.40$, the coefficient is positive and smaller than $z=0.30$, as expected after thresholding and division.

Common Endpoint Mistakes

The endpoint calculations are simple, but the penalty parameterization makes several errors easy to make:

Reversing the endpoints. $\alpha=0$ is ridge, and $\alpha=1$ is lasso for the objective stated here.
Using the wrong lasso threshold. In the orthogonal formula, the threshold is $\lambda\alpha$; at the lasso endpoint it is $\lambda$, not $\lambda/2$.
Omitting the elastic-net denominator. For an intermediate $\alpha$, the denominator $1+\lambda(1-\alpha)$ accounts for the squared-coefficient penalty.
Assuming every penalized coefficient is nonzero. Lasso can give exact zeros, and an intermediate elastic-net coefficient can also be zero if $|z_j|\leq\lambda\alpha$ in the orthogonal setting.
Comparing numerical penalties across conventions without checking the objective. A formula that uses a different factor on the squared penalty may use a different numerical $\lambda$ for the same ridge fit.

The orthogonal coefficient formula is a special case, not a general coordinatewise solution for correlated predictors. With correlated columns, the coefficients are coupled through the design matrix, so each coefficient cannot in general be obtained from its $z_j$ alone. The endpoint identification remains valid because setting $\alpha$ to zero or one still reduces the stated objective to ridge or lasso, respectively.

Key takeaway: Under the stated elastic-net objective, $\alpha=0$ gives ridge and $\alpha=1$ gives lasso. For orthogonal, centered, unit-scaled predictors, $\widehat{\beta}_j=S(z_j,\lambda\alpha)/[1+\lambda(1-\alpha)]$, which shows exactly how the threshold and squared-penalty shrinkage combine.

Check Your Understanding

For each problem, assume centered, unit-scaled orthogonal predictors and use the elastic-net objective stated above. Round nonzero final coefficients to four decimal places.

For $z=(1.2,-0.3)$, $\lambda=0.4$, and $\alpha=0$, calculate the ridge coefficients. (Answer: $(0.8571,-0.2143)$)
For the same $z$ and $\lambda$, set $\alpha=1$. Calculate the lasso coefficients, applying the threshold to both predictors. (Answer: $(0.8000,0.0000)$)
For $z=(1.5,-0.9,0.2)$, $\lambda=0.5$, and $\alpha=0.4$, calculate the elastic-net coefficients. (Answer: $(1.0000,-0.5385,0.0000)$)
A predictor has $z=0.45$, with $\lambda=0.6$ and $\alpha=0.75$. Determine whether its coefficient is zero, including how equality with the threshold is handled. (Answer: zero; $\lambda\alpha=0.45=|z|$, so the boundary coefficient is zero.)
For $z=(0.9,-0.7)$, $\lambda=0.8$, and $\alpha=0.5$, calculate both coefficients and state whether either is removed. (Answer: $(0.3571,-0.2143)$; neither is zero.)

Mixing parameter	Penalty that remains	Resulting method
\(\alpha=0\)	\(\frac{\lambda}{2}\sum_j\beta_j^2\)	Ridge regression
\(0<\alpha<1\)	\(\lambda\{\alpha\sum_j\|\beta_j\|+\frac{1-\alpha}{2}\sum_j\beta_j^2\}\)	Elastic net with both penalties
\(\alpha=1\)	\(\lambda\sum_j\|\beta_j\|\)	Lasso regression

What You'll Learn