Coordinate Descent Updates for Lasso and Elastic Net

Why Update One Coefficient at a Time?

In genomic regression, the number of predictors can exceed the number of samples, so fitting a model by inverting the full predictor cross-product matrix is generally not an option. Lasso and elastic net avoid that requirement: with all coefficients but one held fixed, each coefficient has a simple one-dimensional minimization. Repeatedly minimizing over coordinates gives cyclic coordinate descent, a practical way to fit these penalized models even when \(p>n\).

The update depends on the predictor's association with the response after accounting for the other current coefficients. It is not, in general, the same as applying the orthogonal-predictor lasso formula to each predictor separately. Correlation between predictors enters through the partial residual.

Model, Scaling, and Objective

Let \(y\) be a response vector of length \(n\), and let \(X\) contain \(p\) predictor columns \(x_1,\ldots,x_p\). Assume the predictors have been centered and standardized so that \(\sum_{i=1}^{n}x_{ij}=0\) and \(x_j^\top x_j/n=1\) for each \(j\). An intercept is left unpenalized; with centered predictors, it is estimated by \(\bar y\). Equivalently, the formulas below apply to a centered response after the intercept has been removed.

Use the following parameterization, with \(\lambda\geq 0\) and \(0\leq\alpha\leq 1\):

Here \(\beta=(\beta_1,\ldots,\beta_p)^\top\) is the coefficient vector. The lasso is the special case \(\alpha=1\). For \(0\leq\alpha<1\), the objective includes an elastic-net ridge component as well as the lasso component. It is convenient to write \(\lambda_1=\lambda\alpha\) and \(\lambda_2=\lambda(1-\alpha)\), so the penalty is \(\lambda_1\sum_j|\beta_j|+\lambda_2\sum_j\beta_j^2/2\). This convention fixes the factors of two in the updates.

The updates assume squared-error loss, a common set of observed responses, and the stated predictor scaling. The penalty is separable across coefficients, which is essential to the simple coordinatewise minimization. In practice, if predictors were standardized using a different convention, such as a sum of squares of \(n-1\), the coordinate denominator must reflect the actual column sum of squares. The earlier material on predictor scaling before penalization explains why scaling matters when penalties are applied.

Deriving the Coordinate Update

Hold every coefficient except \(\beta_j\) fixed. Define the partial residual that excludes predictor \(j\) as \(r_{-j}=y-\sum_{k\ne j}x_k\beta_k\). The parts of the objective that depend on a candidate value \(b\) for \(\beta_j\) are

Expanding the squared norm and dropping terms that do not depend on \(b\) gives a scalar objective. Define \(z_j=x_j^\top r_{-j}/n\) and \(a_j=x_j^\top x_j/n+\lambda_2\). Then the scalar objective is \(a_jb^2/2-z_jb+\lambda_1|b|\). For standardized predictors, \(a_j=1+\lambda_2\).

For \(b>0\), differentiating the scalar objective gives \(a_jb-z_j+\lambda_1=0\), hence \(b=(z_j-\lambda_1)/a_j\), which is positive only if \(z_j>\lambda_1\). For \(b<0\), the derivative is \(a_jb-z_j-\lambda_1=0\), giving \(b=(z_j+\lambda_1)/a_j\), which is negative only if \(z_j<-\lambda_1\). At \(b=0\), the subgradient condition is \(|z_j|\leq\lambda_1\). These three cases yield the update:

For standardized predictors, this becomes \(S(z_j,\lambda\alpha)/(1+\lambda(1-\alpha))\). The lasso update has denominator 1; the elastic-net ridge component adds to the denominator and further shrinks nonzero updates. If the coefficient is set to zero, the reason is that the partial association \(z_j\) does not exceed the lasso threshold in magnitude.

A Cyclic Sweep on a Small Design

In a cyclic sweep, visit the predictors in a fixed order and update each using the current values of all the other coefficients. The residual form makes this efficient. If the current residual is \(r=y-X\beta\), then after changing \(\beta_j\) by \(\Delta=\beta_j^{\mathrm{new}}-\beta_j^{\mathrm{old}}\), update it as \(r\leftarrow r-x_j\Delta\). This keeps the residual consistent without recomputing the full matrix product after every coordinate.

Optimality Checks and Failure Modes

A completed coordinate update minimizes the objective over that coordinate, but a single sweep does not usually minimize the full objective. At a solution, every coordinate must satisfy its one-dimensional optimality condition. Define the residual \(r=y-X\beta\) and \(g_j=x_j^\top r/n\). For the objective above, the conditions are

These conditions follow from the derivative of the squared-error term, which is \(-x_j^\top r/n\), together with the derivative or subgradient of the penalty. They provide a practical diagnostic: a nonzero coefficient should match the signed penalty-adjusted partial association, while a zero coefficient should remain within the lasso threshold. A sizeable violation indicates that the coordinates have not yet reached a solution or that the residual, scaling, or penalty has been implemented incorrectly.

For a convex objective, each exact coordinate update cannot increase \(Q\), because it minimizes \(Q\) over the coordinate being changed while holding the others fixed. With standardized columns, each coordinate subproblem has positive quadratic curvature, so its update is well-defined. Under the usual finite-dimensional convex coordinate-descent conditions, cycling toward coordinatewise optimality reaches a minimizer. If \(\lambda_2>0\), the squared-error term plus the ridge penalty makes the objective strictly convex in \(\beta\), giving a unique coefficient minimizer. With pure lasso, minimizers need not be unique when predictors are dependent; the fitted values can be unique even when multiple coefficient vectors attain the minimum.

This distinction matters when predictors are highly correlated. Coordinate descent can still decrease the objective, but coefficients may be sensitive to the order of updates or to small perturbations when the lasso solution is nonunique. The elastic-net ridge component can stabilize the coefficient solution, but does not make a poorly scaled or incorrectly constructed design matrix appropriate.

How Coordinate Descent Compares With Alternatives

The main advantage of coordinate descent is computational simplicity, especially when residuals are updated efficiently and many coefficients remain zero. Its limitations include sensitivity to scaling, potentially slow progress with strongly correlated predictors, and the possibility of nonunique lasso coefficients. It also depends on the correct loss and penalty: these squared-error updates are not logistic-regression updates and should not be transferred unchanged to a different response model.

Common Mistakes

• Ignoring the current coefficients. Using \(x_j^\top y/n\) at every update treats the predictors as if they were orthogonal. Use \(x_j^\top r_{-j}/n\), which accounts for the other current coefficients.
• Dropping the elastic-net denominator. For the stated objective, the denominator is \(1+\lambda(1-\alpha)\) with standardized predictors. The ridge component changes more than the threshold.
• Mixing penalty conventions. If the objective is written with different factors, its update must be derived from that objective. Do not combine a threshold or denominator from one parameterization with another.
• Assuming a zero update proves a true effect is absent. A zero is a penalized-model result conditional on the other coefficients, the response, scaling, and chosen penalty.
• Stopping after one pass. A sweep updates every coordinate once; it does not guarantee that all coordinates satisfy their optimality conditions.

A common mistaken argument is: “The predictor is standardized, so its lasso coefficient is just the soft-thresholded correlation with the response.” That is correct for an orthogonal design, not generally for correlated predictors. The correction is to form the partial residual excluding that predictor, compute \(x_j^\top r_{-j}/n\), and repeat the coordinate updates until the objective and coordinatewise conditions have settled.