Elastic-Net Coordinate Updates for GLMs

From a GLM Approximation to a Coordinate Update

Penalized GLM fitting often replaces a difficult likelihood optimization step with a weighted least-squares subproblem. Coordinate descent solves that subproblem by updating one coefficient at a time while holding the others fixed. For the elastic net, the coordinate calculation combines two effects: an absolute-value penalty can set an update exactly to zero, while a squared penalty reduces its magnitude through an enlarged denominator.

The formula is simple, but its parts depend on the precise scaling of the quadratic loss and penalty. This tutorial fixes those conventions, derives the update from the one-coordinate objective, and checks the lasso, ridge, and unpenalized limits.

Definition: Let $z_i$ and $w_i$, with $w_i\geq 0$, be the working response and weight for observation $i$ in a GLM quadratic approximation. With predictor vector $x_i=(x_{i1},\ldots,x_{ip})$, offset $o_i$, and intercept $\beta_0$, consider the subproblem $$ Q(\beta_0,\beta)=\frac{1}{2n}\sum_{i=1}^n w_i\left(z_i-o_i-\beta_0-\sum_{k=1}^p x_{ik}\beta_k\right)^2 +\lambda\alpha\sum_{k=1}^p|\beta_k| +\frac{\lambda(1-\alpha)}{2}\sum_{k=1}^p\beta_k^2, $$ where $\beta=(\beta_1,\ldots,\beta_p)$, $\lambda\geq 0$, and $0\leq\alpha\leq 1$. The intercept and offset are not penalized.

The factor $1/(2n)$ in the loss and the factor $1/2$ in the squared penalty are part of this convention. Different scalings produce different-looking formulas, so they must be kept consistent. The working responses and weights are held fixed during this quadratic subproblem; the resulting update is not, by itself, an exact minimization of the original GLM penalized likelihood.

Deriving the Elastic-Net Coordinate Formula

Fix a predictor index $j$. Hold the intercept and all coefficients other than $\beta_j$ fixed, and form the partial residual excluding predictor $j$: $$ r_{ij}^{(-j)}=z_i-o_i-\beta_0-\sum_{k\ne j}x_{ik}\beta_k. $$ If the candidate value of the coordinate is $b$, the fitted residual for observation $i$ is $r_{ij}^{(-j)}-x_{ij}b$. Define the weighted curvature and the weighted partial-residual association by $$ A_j=\frac{1}{n}\sum_{i=1}^n w_i x_{ij}^2, \qquad c_j=\frac{1}{n}\sum_{i=1}^n w_i x_{ij}r_{ij}^{(-j)}. $$ Both quantities are computed with the other coordinates fixed.

Theorem: If $D_j=A_j+\lambda(1-\alpha)>0$, the unique minimizer of $Q$ over coordinate $\beta_j$, with all other coordinates held fixed, is $$ \beta_j^{\mathrm{new}}=\frac{S(c_j,\lambda\alpha)}{A_j+\lambda(1-\alpha)}, \qquad S(c,t)=\operatorname{sign}(c)(|c|-t)_+, $$ where $(u)_+=\max(u,0)$. Equivalently, $S(c,t)=0$ for $|c|\leq t$, $S(c,t)=c-t$ for $c>t$, and $S(c,t)=c+t$ for $c<-t$.

Proof. Terms of $Q$ that do not depend on $b$ can be discarded. Expanding the weighted residual square gives $$ \frac{1}{2n}\sum_i w_i(r_{ij}^{(-j)}-x_{ij}b)^2 =\text{constant}+\frac{A_j}{2}b^2-c_jb. $$ After adding the two penalties involving this coordinate, its objective, up to an additive constant, is $$ q_j(b)=\frac{D_j}{2}b^2-c_jb+\lambda\alpha|b|. $$ Since $D_j>0$, this is a strictly convex function, so any point satisfying its convex subgradient condition is its unique minimizer. The subgradient of $|b|$ is $\{1\}$ if $b>0$, $\{-1\}$ if $b<0$, and the interval $[-1,1]$ if $b=0$. Thus the condition is $$ 0\in D_jb-c_j+\lambda\alpha\,\partial|b|. $$ If $b>0$, the condition becomes $D_jb-c_j+\lambda\alpha=0$, which gives $b=(c_j-\lambda\alpha)/D_j$; this is positive exactly when $c_j>\lambda\alpha$. If $b<0$, it becomes $D_jb-c_j-\lambda\alpha=0$, which gives $b=(c_j+\lambda\alpha)/D_j$; this is negative exactly when $c_j<-\lambda\alpha$. At $b=0$, the condition is $c_j\in\lambda\alpha[-1,1]$, or $|c_j|\leq\lambda\alpha$. These three cases are exactly the stated soft-threshold formula. Strict convexity makes the minimizer unique. $\square$

The derivation identifies the separate roles of the two penalties. The lasso part sets the numerator to zero when the weighted association is small enough. The ridge part is added to the curvature in the denominator, so it shrinks a nonzero update. The denominator increases from $A_j$ when $\lambda>0$ and $\alpha<1$. For positive $\lambda$, changing $\alpha$ changes both the threshold $\lambda\alpha$ and the ridge addition $\lambda(1-\alpha)$; when $\lambda=0$, neither penalty contribution depends on $\alpha$.

Worked Coordinate Updates

Worked Example: A Nonzero Elastic-Net Update

Suppose $A_j=4$, $c_j=2.4$, $\lambda=0.5$, and $\alpha=0.6$. The threshold is $$ \lambda\alpha=0.5(0.6)=0.3, $$ and the denominator is $$ A_j+\lambda(1-\alpha)=4+0.5(0.4)=4.2. $$ Since $c_j=2.4>0.3$, the soft-thresholded numerator is $2.4-0.3=2.1$. Therefore $$ \beta_j^{\mathrm{new}}=\frac{2.1}{4.2}=0.5. $$ Without the ridge contribution to the denominator, the lasso-only update would be $2.1/4=0.525$. Here the ridge component makes the nonzero update smaller.

Worked Example: A Coordinate Thresholded to Zero

Let $A_j=2$, $c_j=-0.18$, $\lambda=0.3$, and $\alpha=0.8$. The threshold is $$ \lambda\alpha=0.3(0.8)=0.24, $$ so $|c_j|=0.18\leq 0.24$. Thus $S(c_j,\lambda\alpha)=0$. The denominator is $2+0.3(0.2)=2.06>0$, and the update is $$ \beta_j^{\mathrm{new}}=\frac{0}{2.06}=0. $$ This is not a rounding of a small nonzero value: the one-coordinate minimizer is exactly zero because the lasso threshold covers the association.

Worked Example: Ridge and Zero-Penalty Limits

First take a ridge-only penalty with $A_j=1.5$, $c_j=-0.9$, $\lambda=0.4$, and $\alpha=0$. The threshold is zero, and the denominator is $1.5+0.4=1.9$, so $$ \beta_j^{\mathrm{new}}=\frac{-0.9}{1.9}\approx -0.474. $$ The update is nonzero because there is no lasso threshold, but its magnitude is smaller than the unpenalized coordinate value $-0.9/1.5=-0.6$.

Now use $A_j=2$, $c_j=1.2$, and $\lambda=0$. For either $\alpha=0.25$ or $\alpha=0.9$, the threshold $\lambda\alpha$ and ridge addition $\lambda(1-\alpha)$ are both zero. In both cases, $$ \beta_j^{\mathrm{new}}=\frac{1.2}{2}=0.6. $$ Thus the mixing parameter has no effect when the overall penalty parameter is zero.

Special Cases and Conditions

The coordinate formula includes familiar penalties as endpoints. When $\alpha=1$, the ridge addition vanishes and the update is the lasso soft-thresholded value divided by $A_j$, provided $A_j>0$. When $\alpha=0$, the threshold vanishes and the update is $c_j/[A_j+\lambda]$. When $\lambda=0$, it is $c_j/A_j$, independent of $\alpha$, provided $A_j>0$.

The condition $D_j>0$ matters. It holds if the weighted predictor column has positive curvature $A_j>0$, or if $\lambda>0$ and $\alpha<1$. If $D_j=0$, then $A_j=0$ and $\lambda(1-\alpha)=0$, so the displayed quotient is not defined. Since $w_i\geq0$, $A_j=0$ implies $w_ix_{ij}^2=0$ for every $i$, and hence $c_j=0$. If $\lambda\alpha>0$, the coordinate-dependent objective then reduces to $\lambda\alpha|b|$, uniquely minimized at zero. If also $\lambda\alpha=0$, the objective is constant in $b$, so the coordinate is unidentified by this subproblem and there is no unique update. These degenerate cases should not be forced into a formula with a zero denominator.

Result: An exact coordinate minimization cannot increase the quadratic-subproblem objective. If $\beta_j^{\mathrm{new}}$ minimizes $Q$ over coordinate $j$ with all other coordinates fixed, then $$ Q(\beta_0,\beta_1,\ldots,\beta_j^{\mathrm{new}},\ldots,\beta_p) \leq Q(\beta_0,\beta_1,\ldots,\beta_j,\ldots,\beta_p). $$ Proof. The current value $\beta_j$ is one of the candidate values in the one-dimensional minimization. A minimizer has objective no greater than the objective at any candidate, including the current coordinate value. All other terms and coordinates are unchanged, which gives the inequality. $\square$

This guarantee applies to the fixed weighted least-squares subproblem. In an iterative GLM method, working responses and weights may be recomputed at a later outer iteration, producing a different subproblem. The coordinatewise inequality alone does not assert that every such change decreases the original penalized likelihood. It does explain why each exact coordinate update is appropriate within the current quadratic approximation.

Why the Two Parts of the Update Matter

A lasso-only description misses continuous shrinkage from the ridge component, while a ridge-only description misses the exact zeros induced by thresholding. For a positive penalty parameter, the mixing value $\alpha$ allocates penalty weight between those mechanisms: increasing $\alpha$ raises the threshold and reduces the ridge addition. The net change in a coefficient need not be summarized as simply “more shrinkage,” because both numerator and denominator can change.

A common mistake is to write the denominator as $A_j+\lambda$, or to say that it always increases whenever $\alpha<1$. Under the stated elastic-net scaling, the denominator is $A_j+\lambda(1-\alpha)$; it increases beyond $A_j$ only when both $\lambda>0$ and $\alpha<1$. Likewise, $\alpha$ affects neither the threshold nor denominator when $\lambda=0$. Keeping the loss scaling, penalty scaling, and parameter edge cases explicit prevents these errors.

Compute the partial residual.
Remove the offset, intercept, and all fitted predictor contributions except the coordinate being updated.

Calculate curvature and association.
Use the current weights to obtain $A_j$ and $c_j$.

Threshold, then divide.
Apply $S(c_j,\lambda\alpha)$, and divide by $A_j+\lambda(1-\alpha)$ when that denominator is positive.

Check Your Understanding

Use the stated objective scaling and assume the coordinate denominator is positive unless a question asks about a degenerate case.

For $A_j=3$, $c_j=0.5$, $\lambda=0.4$, and $\alpha=0.5$, calculate the threshold, denominator, and coordinate update.
Why does the update become exactly zero when $|c_j|\leq\lambda\alpha$?
With $\lambda>0$, what happens to the ridge addition when $\alpha=1$? What happens to the lasso threshold when $\alpha=0$?
Explain why changing $\alpha$ has no effect on the coordinate formula when $\lambda=0$.
What does $A_j=0$ imply about $c_j$ under nonnegative weights, and why can the formula then have a zero denominator?