The Lasso Objective, Standardization, and Coordinate Descent

Why the Objective and Scaling Matter

In a high-dimensional regression, many predictors may be irrelevant or redundant, so fitting ordinary least squares can be unstable or may not yield a unique coefficient vector. The lasso addresses this by adding a penalty on coefficient magnitudes to the least-squares criterion. Its solution depends not only on the data but also on how the objective is scaled and how predictors are measured. Those choices must be explicit before interpreting a penalty value or carrying out an optimization.

This tutorial uses coordinate descent: at each step, it minimizes the objective with respect to one coefficient while holding the others fixed. The resulting update has a simple thresholding form. The calculation is useful in its own right and also makes the consequences of standardization and penalty scaling concrete.

Definition: For observations $i=1,\ldots,n$, let $y_i$ be the outcome and $x_{ij}$ the value of predictor $j$, for $j=1,\ldots,p$. The lasso least-squares estimator under the convention used here minimizes $$ \frac{1}{2n}\sum_{i=1}^{n}\left(y_i-\alpha-\sum_{j=1}^{p}x_{ij}\beta_j\right)^2 +\lambda\sum_{j=1}^{p}|\beta_j|, \qquad \lambda\geq 0. $$ The intercept $\alpha$ is not penalized. The coefficients $\beta_j$ are penalized by their absolute values, and $\lambda$ controls the strength of that penalty.

The factor $1/(2n)$ is a convention, not a universal definition. It makes the least-squares part an average and simplifies the coordinate update. A package that instead minimizes the residual sum of squares plus a penalty uses a correspondingly different numerical penalty value. The same symbol $\lambda$ should therefore be compared across calculations only after the objective convention is checked.

Centering and Standardizing Predictors

The intercept can be handled separately by centering. For each predictor, let $\bar{x}_j=n^{-1}\sum_i x_{ij}$, and let $\bar{y}=n^{-1}\sum_i y_i$. Define $x_{ij}^{c}=x_{ij}-\bar{x}_j$ and $y_i^{c}=y_i-\bar{y}$. For fixed slopes, minimizing squared error over the unpenalized intercept gives $\alpha=\bar{y}-\sum_j\bar{x}_j\beta_j$. Thus, fitting the slopes with centered variables is equivalent to fitting the original model with an intercept, and the intercept can be recovered afterward.

For this tutorial, a centered predictor is standardized by dividing by its root mean square, so its squared values average to one. This convention uses $n$, not $n-1$, in the scaling:

\frac{1}{n}\sum_{i=1}^{n}x_{ij}^{c\,2}=1.

A standardized coefficient measures the change in the outcome associated with a one-unit increase in the standardized predictor. Standardization is important because the lasso penalizes coefficient magnitudes: without it, changing a predictor from dollars to thousands of dollars changes its coefficient scale and therefore changes the penalty it experiences. Centering also ensures that the intercept is not forced to compensate for arbitrary predictor means.

Worked Example: Standardizing Two Predictors

Suppose a hypothetical data set has $n=4$ observations. Predictor $a$, measured in dollars, has values $10,20,30,40$; predictor $b$, measured in hours, has values $1,1,3,3$. Compute their centered, standardized values using the convention above.

For $a$, the mean is $\bar{x}_a=(10+20+30+40)/4=25$, so the centered values are $(-15,-5,5,15)$. Their mean squared value is $(225+25+25+225)/4=125$. The root mean square is $\sqrt{125}\approx 11.18$, giving standardized values approximately $(-1.342,-0.4472,0.4472,1.342)$.

For $b$, the mean is $\bar{x}_b=(1+1+3+3)/4=2$, so its centered values are $(-1,-1,1,1)$. The mean squared value is $1$, and the root mean square is $1$. Its standardized values are therefore $(-1,-1,1,1)$. Both standardized columns have mean zero and mean square one, so their coefficients are penalized on a comparable predictor scale.

Some conventions divide centered columns by the sample standard deviation, whose squared denominator uses $n-1$. Under that convention the standardized column has sum of squares $n-1$, rather than $n$. The coordinate formula still applies, but its denominator must be the column's actual mean square. Standardization is not a reason to silently replace the denominator in the update.

Deriving the Coordinate-Descent Update

Work with centered $y$ and predictors, and consider updating coefficient $j$, holding all other coefficients fixed. Define the partial residual $r_{ij}=y_i-\sum_{k\ne j}x_{ik}\beta_k$. The terms of the objective that depend on $\beta_j$ are

Q_j(\beta_j)=\frac{1}{2n}\sum_{i=1}^{n}(r_{ij}-x_{ij}\beta_j)^2+\lambda|\beta_j|.

Write $d_j=n^{-1}\sum_i x_{ij}^2$ and $c_j=n^{-1}\sum_i x_{ij}r_{ij}$. Expanding the square and dropping terms that do not involve $\beta_j$ gives $Q_j(\beta_j)=\tfrac12 d_j\beta_j^2-c_j\beta_j+\lambda|\beta_j|+\text{constant}$. For positive $\beta_j$, the derivative of the terms that vary with $\beta_j$ is $d_j\beta_j-c_j+\lambda$; for negative $\beta_j$, it is $d_j\beta_j-c_j-\lambda$. If the minimum is at zero, the objective cannot be reduced by moving a small distance in either direction, which occurs when $|c_j|\leq\lambda$. If $c_j>\lambda$, minimizing on the positive side gives $\beta_j=(c_j-\lambda)/d_j$; if $c_j<-\lambda$, minimizing on the negative side gives $\beta_j=(c_j+\lambda)/d_j$. Combining these cases yields the update.

Coordinate update: Define the soft-thresholding function $S(c,\lambda)=\operatorname{sign}(c)\max(|c|-\lambda,0)$. If $d_j>0$, the exact minimizer in coordinate $j$, with all other coefficients held fixed, is $$ \beta_j\leftarrow\frac{S(c_j,\lambda)}{d_j} = \frac{S\!\left(n^{-1}\sum_i x_{ij}r_{ij},\lambda\right)} {n^{-1}\sum_i x_{ij}^2}. $$ For a standardized column with mean square one, $d_j=1$, so the update is simply $S(c_j,\lambda)$.

The thresholding action is the key feature: if the partial association $c_j$ is no larger than $\lambda$ in absolute value, the updated coefficient is zero. Otherwise the value is pulled toward zero by $\lambda$, with the amount also adjusted for the column's mean square. When updating one coefficient, the partial residual excludes that coefficient's current contribution. This makes the formula an exact minimization along that coordinate, not an update based on the full residual while treating the other coefficients as absent.

Worked Coordinate Updates

Worked Example: One Standardized Predictor

Consider four hypothetical observations with centered predictor $x=(-1,1,-1,1)$ and centered outcome $y=(-2,2,-2,2)$. The predictor has mean square $d=1$. Use $\lambda=0.5$, and start from $\beta=0$. With one predictor, the partial residual is $r=y$.

The score is $c=n^{-1}\sum_i x_i y_i$. Each product is $2$, so $c=(2+2+2+2)/4=2$. The update is $\beta=S(2,0.5)/1=(2-0.5)=1.5$. The fitted values are $1.5x=(-1.5,1.5,-1.5,1.5)$, and the residuals are $(-0.5,0.5,-0.5,0.5)$. The objective at this coefficient is $\{4(0.5^2)\}/(2\cdot4)+0.5|1.5|=0.125+0.75=0.875$.

For comparison, the unpenalized fit has coefficient $2$, zero residual sum of squares, and objective $0.5(2)=1$. The penalized solution has a smaller objective: it accepts some residual error in exchange for a smaller coefficient penalty.

Worked Example: A Correlated-Predictor Coordinate Cycle

Use the hypothetical centered columns $x_1=(-1,0,1)$, $x_2=(-1,1,0)$, and $y=(-2,1,1)$, with $n=3$ and $\lambda=0.2$. Both predictor means are zero, so an unpenalized intercept is handled by centering. Here each column has mean square $d_1=d_2=2/3$, and their cross-product mean is $n^{-1}x_1'x_2=1/3$. They are correlated, so updating one coefficient changes the partial score for the other.

Start with $\beta_1=\beta_2=0$. For the first update, $c_1=n^{-1}x_1'y=3/3=1$, so $\beta_1=S(1,0.2)/(2/3)=0.8/(2/3)=1.2$. For the second coordinate, use the partial residual $r_2=y-1.2x_1$. Its score is $c_2=n^{-1}x_2'r_2=1-1.2(1/3)=0.6$, giving $\beta_2=S(0.6,0.2)/(2/3)=0.4/(2/3)=0.6$.

Now make another pass. For coordinate 1, the partial residual is $y-0.6x_2$, so $c_1=1-0.6(1/3)=0.8$, and the update is $\beta_1=(0.8-0.2)/(2/3)=0.9$. For coordinate 2, the partial residual is $y-0.9x_1$, giving $c_2=1-0.9(1/3)=0.7$, and $\beta_2=(0.7-0.2)/(2/3)=0.75$. The coefficients change from $(1.2,0.6)$ to $(0.9,0.75)$ because each coordinate update accounts for the current value of the other predictor. A numerical implementation continues cycling until a chosen convergence criterion is met.

Penalty Scaling and Units

The coefficient scale and the penalty scale are linked. If a predictor is replaced by $x_j^*=a x_j$, for $a>0$, the same fitted contribution requires $\beta_j^*=\beta_j/a$. But its penalty becomes $\lambda|\beta_j^*|=(\lambda/a)|\beta_j|$. Holding the numerical $\lambda$ fixed therefore changes how strongly that contribution is penalized. Standardizing predictors gives a consistent scale for this reason; it does not make different predictors statistically equivalent or remove their correlation.

The response scale matters too. If $y$ is replaced by $y^*=b y$, with $b>0$, an equivalent fit has coefficients multiplied by $b$. The squared-error term scales by $b^2$, while the coefficient penalty scales by $b$. To express the same trade-off under the stated objective, the penalty must also be multiplied by $b$. Thus, a penalty value has meaning only alongside the outcome units, predictor scaling, and objective convention.

Worked Example: Rescaling a Predictor Changes the Penalty

Return to the one-predictor data $x=(-1,1,-1,1)$, $y=2x$, and $\lambda=0.5$. The update gave coefficient $1.5$ and fitted values $1.5x$. Now record the predictor in units ten times as large, $z=10x$, without changing the numerical penalty.

The mean square of $z$ is $100$, and its score is $n^{-1}z'y=20$. The updated coefficient on $z$ is $\gamma=S(20,0.5)/100=19.5/100=0.195$. Its fitted contribution is $0.195z=1.95x$, not $1.5x$. The difference arises because the same numerical penalty is smaller relative to the coefficient required in the enlarged predictor units. If instead the penalty for $z$ were $5$, equal to $10\lambda$, then $\gamma=S(20,5)/100=15/100=0.15$, and $0.15z=1.5x$. This restores the original fitted contribution.

Using Coordinate Descent Responsibly

For a fixed $\lambda$, cyclic coordinate descent repeatedly applies the one-coordinate minimizer. Each update does not increase the objective, because it minimizes that objective over the chosen coordinate with the others held fixed. The least-squares objective plus the absolute-value penalty is convex, so coordinate descent is a standard practical method for finding a minimizer. If predictors are perfectly redundant, however, there may be more than one coefficient vector that minimizes the objective; a fitted value can be well-defined even when individual coefficients are not unique.

This optimization procedure alone does not say that a selected predictor has a causal effect, establish that the selected variables are the true model, or provide valid standard errors. It computes a penalized fit for the chosen data, scaling, and penalty. Statistical guarantees and inference require additional assumptions and methods.

Common Mistakes

Ignoring the objective convention. A penalty value from an objective with residual sum of squares may not match one using residual sum of squares divided by $2n$. Compare the full objectives, not just the displayed $\lambda$.
Penalizing the intercept by accident. The intercept is unpenalized here; center the variables or explicitly exclude it from the absolute-value penalty.
Using a full residual for every update. The coordinate score uses the partial residual that excludes the predictor currently being updated.
Assuming standardization is cosmetic. Rescaling a predictor changes the coefficient magnitude and therefore its penalty unless the penalty is adjusted as well.
Calling a zero coefficient proof of irrelevance. It is a feature of the penalized fit at a particular $\lambda$, not proof that the predictor has no relationship with the outcome.

Key takeaway: Under the objective $\mathrm{RSS}/(2n)+\lambda\sum_j|\beta_j|$, coordinate descent updates a coefficient by soft-thresholding its partial-residual score and dividing by the predictor's mean square. Centering, standardization, and the precise objective convention determine how that update and its penalty should be interpreted.

Check Your Understanding

Use the stated objective and scaling conventions in your answers.

Why is the intercept excluded from the lasso penalty, and how does centering allow it to be handled separately?
A centered predictor has mean square $4$, partial-residual score $1.1$, and $\lambda=0.3$. What is its coordinate update?
If a predictor is multiplied by five, what coefficient change preserves its fitted contribution? Why can the same numerical penalty change the fitted result?
In the correlated-predictor example, why does the score for the second coordinate change after updating the first?
Does a lasso coefficient of zero establish that a predictor has no causal effect? Explain what the optimization result does establish.