Elastic Net Penalized Objective

Why the Elastic Net Uses Two Penalty Components

Penalized regression trades fit to the observed responses against restrictions on the coefficient vector. The lasso uses an L1 penalty, which can set fitted coefficients exactly to zero. A squared-L2 penalty instead adds a smooth cost that grows quadratically with coefficient size. The elastic net combines these two penalties, allowing both effects to contribute to the same objective.

The distinction matters in practice. A coefficient that is merely made smaller still contributes to the fitted model; a coefficient set to zero does not. The elastic net’s mixing parameter controls the relative contributions of the L1 and squared-L2 terms, while its overall penalty parameter controls their combined strength. These parameters have meaning only relative to the stated loss scaling and predictor scaling.

Definition: For a response vector $y$, an $n \times p$ design matrix $X$, an unpenalized intercept $\beta_0$, and coefficients $\beta_1,\ldots,\beta_p$, the elastic net penalized least-squares objective is $$ \frac{1}{2n}\sum_{i=1}^{n}\left(y_i-\beta_0-\sum_{j=1}^{p}x_{ij}\beta_j\right)^2 +\lambda\left[\alpha\sum_{j=1}^{p}|\beta_j|+\frac{1-\alpha}{2}\sum_{j=1}^{p}\beta_j^2\right]. $$ Here $\lambda \geq 0$ is the overall penalty parameter and $0 \leq \alpha \leq 1$ is the mixing parameter. The fitted coefficients minimize this objective. The intercept is not penalized in this stated formulation.

The first term is the mean squared error with a factor of $1/2$, a scaling that simplifies derivatives. The penalty combines the sum of absolute coefficient values with half the sum of squared coefficient values. The factor $1/2$ in the squared-L2 term is part of the convention used here; omitting it changes the corresponding derivative and coefficient formula.

For a fixed $\lambda$, a larger $\alpha$ gives the L1 term more weight relative to the squared-L2 term. For a fixed $\alpha$, a larger $\lambda$ strengthens both terms. These parameters should not be interpreted independently of how the response loss is scaled. In particular, changing the loss from a sum of squared errors to a mean squared error changes the numerical penalty strength corresponding to a given $\lambda$.

A Coordinate Formula and Its Derivation

A coordinate update considers one coefficient while holding the intercept and all other coefficients fixed. Let $r$ be the current residual after excluding the predictor whose coefficient is being updated. Write $x$ for that predictor column and $b$ for its coefficient. Define $q=x^Tx/n$ and $z=x^Tr/n$. Up to a constant that does not depend on $b$, the objective for this coordinate is

\frac{q}{2}b^2-zb+\lambda\alpha|b|+\frac{\lambda(1-\alpha)}{2}b^2.

The quantity $q$ records the predictor’s squared scale, while $z$ records its residual association with the response under the same $1/n$ scaling. Define the soft-threshold function $S(t,c)=\operatorname{sign}(t)(|t|-c)_+$, where $(a)_+=\max(a,0)$. Minimizing this one-coordinate objective gives

\widehat b=\frac{S(z,\lambda\alpha)}{q+\lambda(1-\alpha)}.

To justify the formula, first suppose $b>0$. The derivative of the objective is $qb-z+\lambda\alpha+\lambda(1-\alpha)b$. Setting it to zero gives $(q+\lambda(1-\alpha))b=z-\lambda\alpha$, which has a positive solution only when $z>\lambda\alpha$. If $b<0$, the derivative is $qb-z-\lambda\alpha+\lambda(1-\alpha)b$, giving a negative solution only when $z<-\lambda\alpha$. At $b=0$, the absolute-value term has subgradient in $[-1,1]$; zero minimizes the coordinate objective when $|z|\leq\lambda\alpha$. These three cases together give the soft-threshold formula above. The denominator includes the predictor scale $q$, so the formula must be adjusted if the predictors are scaled differently.

Key result: If the predictor has squared norm $x^Tx=n$, then $q=1$, and its coordinate update is $$ \widehat b=\frac{\operatorname{sign}(z)(|z|-\lambda\alpha)_+}{1+\lambda(1-\alpha)}, \qquad z=\frac{x^Tr}{n}. $$ If instead $x^Tx=1$ and $z=x^Tr$, then the same objective gives $$ \widehat b=\frac{\operatorname{sign}(z)(|z|-n\lambda\alpha)_+}{1+n\lambda(1-\alpha)}. $$

The second form is not obtained by simply substituting a unit-norm column into the first formula: both the residual association and the curvature of the loss have changed scale. This distinction is essential when checking a coordinate calculation.

Worked Example: Updating Several Orthogonal Coefficients

Suppose the response is centered, the intercept is therefore zero, and three centered predictor columns are mutually orthogonal, each with squared norm $n$. In a coordinate step, their residual associations $z_j=x_j^Tr_j/n$ are $2.0$, $-0.8$, and $0.25$. Use $\lambda=0.4$ and $\alpha=0.5$ to calculate the three coefficient updates. This method applies the coordinate formula to each orthogonal predictor; it avoids treating a small coefficient as zero without checking the threshold.

First calculate the L1 threshold and the denominator shared by the three updates:

\lambda\alpha=0.4(0.5)=0.2,\qquad 1+\lambda(1-\alpha)=1+0.4(0.5)=1.2.

For the first predictor, $z_1=2.0$, so $|z_1|=2.0>0.2$. Its coefficient is $\widehat b_1=(2.0-0.2)/1.2=1.8/1.2=1.5$.

For the second predictor, $z_2=-0.8$, so $|z_2|=0.8>0.2$. Its coefficient is $\widehat b_2=-(0.8-0.2)/1.2=-0.6/1.2=-0.5$.

For the third predictor, $z_3=0.25$, so $|z_3|=0.25>0.2$. Its coefficient is $\widehat b_3=(0.25-0.2)/1.2=0.05/1.2=0.0416667$.

The coordinate updates are $1.5$, $-0.5$, and $0.0416667$. Each association exceeds the threshold, so none is set to zero. As a check, the first two estimates have the same signs as their associations, and all three magnitudes are smaller than their unpenalized associations. The small third estimate is nonzero because its association is above, not below, the threshold.

Reading the Parameters in a Numerical Calculation

The next example uses a unit-norm predictor rather than a predictor with squared norm $n$. It illustrates why one must carry the loss scaling through the derivation. The objective remains scaled by $1/(2n)$, so the resulting threshold and denominator differ from the update for a predictor with squared norm $n$.

Worked Example: A Unit-Norm Predictor

For one coefficient update, let $n=4$, let the predictor column satisfy $x^Tx=1$, and let the residual excluding this predictor satisfy $x^Tr=2.4$. Use $\lambda=0.1$ and $\alpha=0.5$. Calculate the coordinate update using the stated $1/(2n)$ least-squares loss. This method uses the unit-norm version of the formula; it avoids applying the standardized-predictor formula with the wrong threshold.

With $z=x^Tr=2.4$, the unit-norm formula gives a threshold of $n\lambda\alpha=4(0.1)(0.5)=0.2$. Its denominator is $1+n\lambda(1-\alpha)=1+4(0.1)(0.5)=1.2$.

\widehat b=\frac{2.4-0.2}{1.2}=\frac{2.2}{1.2}=1.833333\ldots

A direct check comes from the coordinate objective. Since $x^Tx=1$, its loss part, up to a constant, is $b^2/(2n)-(x^Tr)b/n=b^2/8-0.6b$. For a positive $b$, add the penalties $0.05b+0.025b^2$. The derivative is $b/4-0.6+0.05+0.05b=0.30b-0.55$. Setting it equal to zero gives $b=0.55/0.30=1.833333\ldots$, agreeing with the formula. The positive solution is consistent with $x^Tr>n\lambda\alpha$, since $2.4>0.2$. Rounding only at the final step, the coefficient is $1.8333$ to four decimal places.

For a predictor with squared norm $n$, the corresponding residual association is normally written $x^Tr/n$, and the threshold is $\lambda\alpha$. For a unit-norm predictor, the data term has less curvature in its coefficient, while the penalty has not changed. Consequently, the threshold and the squared-L2 contribution appear multiplied by $n$ in the formula written using $x^Tr$. That is a scaling consequence of the objective, not a different elastic net model.

Worked Example: Separating the Effects of the Two Parameters

Suppose two mutually orthogonal predictor columns each have squared norm $n$, and their coordinate residual associations are $z_1=1.0$ and $z_2=0.35$. Compare the updates for $\lambda=0.4,\alpha=0.5$ with those for $\lambda=0.4,\alpha=0.8$. Then compare the first setting with $\lambda=0.2,\alpha=0.5$ and $\lambda=0.6,\alpha=0.5$. Use the coordinate formula throughout. This comparison isolates the mixing parameter from the overall penalty parameter and avoids assuming that increasing either parameter affects only one part of the objective.

For $\lambda=0.4,\alpha=0.5$, the threshold is $0.4(0.5)=0.2$, and the denominator is $1+0.4(0.5)=1.2$. Therefore, $\widehat b_1=(1.0-0.2)/1.2=0.6667$ and $\widehat b_2=(0.35-0.2)/1.2=0.1250$.

For $\lambda=0.4,\alpha=0.8$, the threshold is $0.4(0.8)=0.32$, and the denominator is $1+0.4(0.2)=1.08$. Thus, $\widehat b_1=(1.0-0.32)/1.08=0.6296$ and $\widehat b_2=(0.35-0.32)/1.08=0.0278$. The second coefficient remains positive because $0.35>0.32$, but it is much closer to the selection threshold.

Now hold $\alpha=0.5$ and change $\lambda$. With $\lambda=0.2$, the threshold is $0.1$, the denominator is $1.1$, and the updates are $0.9/1.1=0.8182$ and $0.25/1.1=0.2273$. With $\lambda=0.6$, the threshold is $0.3$, the denominator is $1.3$, and the updates are $0.7/1.3=0.5385$ and $0.05/1.3=0.0385$. The second coefficient is still nonzero because $0.35>0.3$.

The comparison shows that increasing $\alpha$ at fixed $\lambda$ changes both the L1 threshold and the squared-L2 contribution. Increasing $\lambda$ at fixed $\alpha$ strengthens both penalties. All four calculations were rounded to four decimal places; each coefficient remains nonzero because its association exceeds its setting’s threshold.

Shrinkage, Selection, and Interpretation

The two components have distinct roles in the coordinate formula. The L1 term creates a threshold: if $|z|\leq\lambda\alpha$, the coefficient update is exactly zero. If $|z|>\lambda\alpha$, the magnitude is first reduced by the threshold and then divided by $q+\lambda(1-\alpha)$. The squared-L2 term increases this denominator, reducing the magnitude of a nonzero update. Thus, the objective can produce both selection (exact zeros) and shrinkage (smaller nonzero coefficients).

The formula is a coordinate update, not generally a complete solution obtained by updating every coefficient once. When predictors are not orthogonal, changing one coefficient changes the residual associations used for the others. The formula still describes the minimizer of a single coordinate with the others held fixed, but the full model must be optimized over all coefficients. The orthogonal examples above avoid that interaction so that each update can be calculated directly.

Coefficient scale also matters. If one predictor is measured in dollars and another in thousands of dollars, their columns have different scales, and the same penalty can affect their coefficients differently. Predictor scaling should therefore be stated when interpreting the numerical value of $\lambda$ or comparing coefficient updates. Likewise, the intercept is excluded from the penalty in the objective defined here; penalizing it would be a different specification.

Common Exam Trap: Mixing Predictor Norms

A frequent error is to use the update for $x^Tx=n$ while using $x^Tx=1$. In the unit-norm example, incorrectly using $x^Tr=2.4$ in the standardized-predictor formula would give $(2.4-0.05)/(1+0.05)=2.2381$, because that mistaken calculation uses $\lambda\alpha=0.05$ and $1+\lambda(1-\alpha)=1.05$. Those are not the correct threshold and denominator for the stated unit-norm design and $1/(2n)$ loss.

The correction is to retain the unit-norm scaling: the threshold is $n\lambda\alpha=0.2$, and the denominator is $1+n\lambda(1-\alpha)=1.2$. The correct coefficient is therefore $1.8333$, as verified by differentiating the original coordinate objective. Before using any update formula, check both the loss divisor and the predictor’s squared norm.

Key takeaway: The elastic net objective adds an L1 penalty weighted by $\lambda\alpha$ and a squared-L2 penalty weighted by $\lambda(1-\alpha)/2$. The L1 component creates a zero threshold; the squared-L2 component contributes additional shrinkage. Correct calculations require matching the coordinate formula to the objective’s loss scaling and the predictor’s scale.

Check Your Understanding

For each problem, use the stated objective scaling and predictor norm. Round coefficients to four decimal places where needed.

A predictor has squared norm $n$, and its coordinate residual association $x^Tr/n$ is $-0.9$. For $\lambda=0.25$ and $\alpha=0.6$, calculate the coordinate update and state whether it is zero. (Answer: $-0.6818$; nonzero.)
A unit-norm predictor is used with $n=9$, $x^Tr=1.8$, $\lambda=0.08$, and $\alpha=0.5$. Using the $1/(2n)$ loss, calculate the update. (Answer: $1.0588$.)
A predictor has squared norm $n$, with $x^Tr/n=0.12$, $\lambda=0.2$, and $\alpha=0.5$. Calculate the update and explain why it is nonzero even though it is small. (Answer: $0.0182$; the association $0.12$ is above the threshold $0.10$.)
A predictor has squared norm $n$, with $x^Tr/n=-0.08$, $\lambda=0.2$, and $\alpha=0.6$. Calculate the update. (Answer: $0$, because $0.08\leq 0.12$.)
For a predictor with squared norm $n$, $z=0.7$, $\lambda=0.3$, and $\alpha=0.4$, calculate the threshold, denominator, and coefficient update. (Answer: threshold $0.12$, denominator $1.18$, and update $0.4915$.)

What You'll Learn