Comparing Lasso and Elastic Net at a Fixed Penalty

Why Compare Fits at Fixed Penalties?

A lasso fit and an elastic net fit can use the same data and the same value of $\lambda$, yet produce different coefficients, selected predictors, and fitted values. Comparing them at fixed settings isolates how the penalty form affects the fit. It does not establish which model will predict better on new data, and it does not determine which settings should be chosen. This comparison is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

To make the arithmetic transparent, this tutorial uses centered, mutually orthogonal predictor columns with equal scale. In applications, predictors may not have these properties, so there is generally no one-step coefficient formula. The objective still defines the fit, but its coefficients must be found by optimization.

Definition: For a quantitative response $y_i$, predictors $x_{ij}$, and $n$ observations, consider the penalized least-squares objective $$ \frac{1}{2n}\sum_{i=1}^{n}\left(y_i-\beta_0-\sum_{j=1}^{p}x_{ij}\beta_j\right)^2 +\lambda\left[\alpha\sum_{j=1}^{p}|\beta_j|+\frac{1-\alpha}{2}\sum_{j=1}^{p}\beta_j^2\right]. $$ Here $\beta_0$ is an unpenalized intercept, $\beta_j$ are predictor coefficients, $\lambda\geq 0$ controls the overall penalty, and $0\leq\alpha\leq1$ controls its mix. At $\alpha=1$, the penalty is lasso; at $0\leq\alpha<1$, it combines an absolute-value penalty with a squared-coefficient penalty. The intercept is not included in either penalty sum.

The factor $1/(2n)$ in the loss is part of the stated convention for these calculations. If an implementation uses a differently scaled loss or penalty, the numerical meaning of a reported $\lambda$ can differ. Therefore, an equal numerical $\lambda$ only makes a valid comparison when the objective conventions and preprocessing are held fixed.

A Useful Formula for Orthogonal Predictors

Suppose the response and predictor columns are centered, and the predictors are mutually orthogonal and scaled so that $\sum_i x_{ij}^2=n$ for every $j$. Define $z_j=(1/n)\sum_i x_{ij}y_i$. Under these conditions, the coefficient for predictor $j$ can be calculated separately from the other coefficients.

The lasso coefficient is obtained by soft-thresholding $z_j$: values whose absolute size does not exceed $\lambda$ become zero, while the others are reduced in absolute size by $\lambda$. For the elastic net, the same threshold is $\alpha\lambda$, and the remaining coefficient is also divided by $1+\lambda(1-\alpha)$.

\hat{\beta}_{j,\mathrm{lasso}} =\operatorname{sign}(z_j)(|z_j|-\lambda)_+, \qquad \hat{\beta}_{j,\mathrm{EN}} =\frac{\operatorname{sign}(z_j)(|z_j|-\alpha\lambda)_+} {1+\lambda(1-\alpha)},

where $(a)_+=\max(a,0)$. A coefficient is selected by the fit when it is nonzero. These formulas require the stated centered, orthogonal, equally scaled design and the stated objective. They do not apply as separate one-variable calculations when predictors are correlated.

Why the Orthogonal-Design Formula Holds

With a centered response and centered predictors, the intercept estimate is the response mean. After subtracting that mean, orthogonality lets the squared-error term separate into a sum of terms, one for each coefficient, plus a constant that does not depend on the coefficients. For one predictor, the coefficient-dependent part of the objective is

\frac{1}{2}(\beta_j-z_j)^2 +\lambda\alpha|\beta_j| +\frac{\lambda(1-\alpha)}{2}\beta_j^2.

For $\beta_j>0$, differentiating this expression gives $(\beta_j-z_j)+\lambda\alpha+\lambda(1-\alpha)\beta_j=0$, or $(1+\lambda(1-\alpha))\beta_j=z_j-\lambda\alpha$. This solution is positive only if $z_j>\lambda\alpha$. For $\beta_j<0$, the absolute-value term has derivative $-\lambda\alpha$, giving a negative solution only if $z_j<-\lambda\alpha$. When $|z_j|\leq\lambda\alpha$, the minimum occurs at zero. Combining these cases gives the elastic net formula. Setting $\alpha=1$ removes the squared-coefficient term and gives the lasso formula.

Fix what is being compared.
Use the same observations, response, predictor scaling, intercept convention, loss scaling, and specified penalty settings for each fit.

Calculate coefficients.
For an orthogonal design, calculate each $z_j$ and apply the appropriate threshold and, for elastic net, the additional denominator.

Compare the fits in context.
Identify zero and nonzero coefficients, compare coefficient magnitudes, and calculate fitted values using the intercept and predictor values.

Worked Comparisons

Worked Example: Comparing Selection and Fitted Values

A hypothetical insurer models a centered claim-cost outcome using three centered, standardized rating variables. There are four observations and the predictor columns are $x_1=(1,1,-1,-1)$, $x_2=(1,-1,1,-1)$, and $x_3=(1,-1,-1,1)$. The response values are $15,11,8,6$, so the response mean is 10 and the centered values are $5,1,-2,-4$. Compare lasso with $\lambda=0.8$ and elastic net with $\lambda=0.8,\alpha=0.5$. Calculate the selected predictors and fitted values for all four observations.

Method and design check. Each predictor has squared length 4, equal to $n$, and the columns are pairwise orthogonal. For example, $x_1^\mathsf{T}x_2=1-1-1+1=0$; the other two cross-products are also zero. Thus the orthogonal-design formulas apply. The intercept for both fits is 10.

Calculate the predictor-response quantities. Dividing each predictor’s inner product with the centered response by $n=4$ gives $z_1=(5+1+2+4)/4=3$, $z_2=(5-1-2+4)/4=1.5$, and $z_3=(5-1+2-4)/4=0.5$.

Lasso coefficients. The threshold is $0.8$. Therefore, $\hat{\beta}_1=3-0.8=2.2$, $\hat{\beta}_2=1.5-0.8=0.7$, and $\hat{\beta}_3=0$, since $0.5\leq0.8$. Lasso selects $x_1$ and $x_2$.

Elastic net coefficients. The threshold is $\alpha\lambda=0.5(0.8)=0.4$, and the denominator is $1+\lambda(1-\alpha)=1+0.8(0.5)=1.4$. Hence $\hat{\beta}_1=(3-0.4)/1.4=13/7$, $\hat{\beta}_2=(1.5-0.4)/1.4=11/14$, and $\hat{\beta}_3=(0.5-0.4)/1.4=1/14$. All three predictors are selected.

Calculate fitted values. Add the intercept 10 to the coefficient-weighted predictor values. For lasso, the four fitted values are $10+2.2+0.7=12.90$, $10+2.2-0.7=11.50$, $10-2.2+0.7=8.50$, and $10-2.2-0.7=7.10$. For elastic net, they are $10+13/7+11/14+1/14=12.71$, $10+13/7-11/14-1/14=11.00$, $10-13/7+11/14-1/14=8.86$, and $10-13/7-11/14+1/14=7.43$. Values are rounded to two decimals only in the final fitted values.

Check and interpretation. The lasso coefficients are smaller than the corresponding unpenalized quantities $3,1.5,0.5$, and its third coefficient is zero. Elastic net also shrinks the coefficients, but its smaller threshold allows the third predictor to remain nonzero. The fitted values are not identical because the coefficient vectors differ. A common mistake is to use the lasso threshold $0.8$ for elastic net; the correct elastic net threshold here is $\alpha\lambda=0.4$, followed by division by 1.4.

Worked Example: A Weak Signal Near the Selection Threshold

In a hypothetical service-duration model, two centered, orthogonal, equally scaled predictors have $z_1=2.4$ and $z_2=0.9$. Compare lasso at $\lambda=0.7$ with elastic net at $\lambda=0.7,\alpha=0.4$. The centered response is exactly $y=2.4x_1+0.9x_2$, and $n=4$. Find the coefficients and the fitted values when the predictor rows are $(1,1),(1,-1),(-1,1),(-1,-1)$.

Lasso. Its threshold is $0.7$, giving $\hat{\beta}_1=2.4-0.7=1.7$ and $\hat{\beta}_2=0.9-0.7=0.2$. Both predictors are selected. The fitted values, in the stated row order, are $1.7+0.2=1.90$, $1.7-0.2=1.50$, $-1.7+0.2=-1.50$, and $-1.7-0.2=-1.90$.

Elastic net. Its threshold is $0.4(0.7)=0.28$, and its denominator is $1+0.7(0.6)=1.42$. The coefficients are $\hat{\beta}_1=(2.4-0.28)/1.42=106/71\approx1.4930$ and $\hat{\beta}_2=(0.9-0.28)/1.42=31/71\approx0.4366$. Both are selected. The fitted values are $137/71\approx1.9296$, $75/71\approx1.0563$, $-75/71\approx-1.0563$, and $-137/71\approx-1.9296$.

Check and interpretation. The lasso coefficient for $x_2$ is positive because $0.9>0.7$; it is not zero merely because its signal is smaller than $z_1$. Elastic net also selects it, with a different coefficient magnitude. The method avoids a ranking-based error: selection is determined by comparing each predictor’s own quantity with the penalty threshold, not by whether its coefficient is the largest.

Worked Example: Comparing Objective Values and a New Fitted Value

A hypothetical maintenance-cost model has four centered observations and two centered, orthogonal, equally scaled predictors. Its response is $y=1.5x_1+x_2$, so $z_1=1.5$ and $z_2=1.0$. Compare lasso at $\lambda=0.5$ with elastic net at $\lambda=0.5,\alpha=0.6$. Calculate the training residual sum of squares and objective for each fit, then predict for a new record with $x_1=0.5,x_2=-1$. The intercept is zero.

Coefficients and predictions. Lasso gives $(1.5-0.5,1.0-0.5)=(1,0.5)$. Elastic net has threshold $0.6(0.5)=0.3$ and denominator $1+0.5(0.4)=1.2$, so its coefficients are $(1.2/1.2,0.7/1.2)=(1,7/12)$. The new-record predictions are $1(0.5)+0.5(-1)=0$ for lasso and $1(0.5)+(7/12)(-1)=-1/12\approx-0.0833$ for elastic net.

Residual sums of squares. In this orthogonal design, the residual sum of squares is $n\sum_j(z_j-\hat{\beta}_j)^2$. For lasso, this is $4[(1.5-1)^2+(1-0.5)^2]=4(0.25+0.25)=2$. For elastic net, it is $4[(0.5)^2+(1-7/12)^2]=4[1/4+25/144]=61/36\approx1.6944$.

Objectives. The lasso objective is $2/(2\cdot4)+0.5(|1|+|0.5|)=0.25+0.75=1.00$. For elastic net, the loss term is $(61/36)/8=61/288$. Its penalty term is $0.5[0.6(1+7/12)+(0.4/2)(1^2+(7/12)^2)]$ $=0.5[19/20+193/720]=877/1440$. The total is $61/288+877/1440=197/240\approx0.8208$.

Check and interpretation. The elastic net objective is smaller for its own stated objective, but that does not by itself prove it predicts better: the lasso and elastic net objectives use different penalty mixtures. The residual sums of squares describe these particular training fits, while the new-record predictions differ. A common mistake is to treat the objective values as directly comparable evidence of predictive superiority when the objectives themselves differ.

How to Read the Comparison

A fixed-penalty comparison supports specific conclusions about the fits on the stated data. It can show which coefficients are zero, how much the nonzero coefficients have shrunk, and how the fitted values change. It does not show that a selected predictor is causally important, that a nonzero coefficient will be stable under new samples, or that one fit has lower prediction error on unseen cases.

In particular, $\lambda$ is not a universal measure of penalty strength independent of the other choices. In the stated objective, changing $\alpha$ changes both the lasso threshold $\alpha\lambda$ and the elastic net shrinkage denominator $1+\lambda(1-\alpha)$. Changing predictor scales or the loss convention can also change the numerical meaning of the penalty. Any conclusion about the comparison should therefore name the objective, settings, and preprocessing used.

Key takeaway: At fixed settings, calculate each model from its own penalty. Lasso can set coefficients exactly to zero; elastic net uses both thresholding and additional shrinkage. Compare selected predictors, coefficient sizes, and fitted values separately, and do not treat objective values from different penalty mixtures as a predictive-performance contest.

Check Your Understanding

For the first three questions, assume centered, orthogonal predictors scaled so that each has squared length $n$, and use the objective defined above.

Two predictors have $z_1=1.8,z_2=0.6$. Find the lasso coefficients at $\lambda=0.5$. (Answer: $(1.3,0.1)$.)
For $z_1=1.2,z_2=0.3,\lambda=0.4,\alpha=0.25$, find the elastic net coefficients to four decimal places and state which predictors are selected. (Answer: $(0.8462,0.1538)$; both predictors are selected.)
For $z_1=2,z_2=0.7,\lambda=0.8,\alpha=0.5$, compare lasso and elastic net coefficients. (Answer: lasso $(1.2,0)$; elastic net $(1.1429,0.2143)$, so elastic net selects both predictors.)
A fitted model has intercept 5, coefficients $(0.8,0.25)$, and a new record has predictor values $(1,-2)$. Calculate its fitted value. (Answer: $5.30$.)
At $\alpha=1$, does the elastic net coefficient formula reduce to the lasso formula at the same $\lambda$ under the stated objective? (Answer: Yes; the denominator is 1 and the threshold is $\lambda$.)

What You'll Learn