Predictor Scaling and Lasso Selection

Why Predictor Units Matter

A lasso fit can select different predictors when the same measurements are expressed in different units. A dollar amount recorded in dollars rather than thousands of dollars describes the same underlying quantity, but it changes the numerical scale of the predictor. Because lasso penalizes coefficient magnitudes, changing that scale can change the penalty a predictor effectively faces. This matters when comparing predictors measured in different units and when interpreting a model fitted after standardization.

The central distinction is between a predictor's units and the underlying information it contains. Rescaling a column does not change its information, but it changes the coefficient needed to represent a given effect. Lasso penalizes that coefficient, so the units can affect both shrinkage and selection.

Definition: For centered response $y$ and centered predictors $x_1,\ldots,x_p$, consider the lasso objective $$ \frac{1}{2n}\sum_{i=1}^{n}\left(y_i-\sum_{j=1}^{p}x_{ij}\beta_j\right)^2+\lambda\sum_{j=1}^{p}|\beta_j|, $$ where $\lambda\geq 0$. The intercept, if included in an uncentered model, is not penalized. A predictor is selected when its fitted coefficient is nonzero. Rescaling predictor $x_j$ generally changes the coefficient value and the effective penalty on that predictor.

Centering allows us to focus on the slopes: with an unpenalized intercept, centering the response and predictors separates the intercept calculation from the slope calculations. The objective above uses the $1/(2n)$ loss convention. Other conventions can use a different numerical value of $\lambda$, so penalty values should only be compared when the objective conventions agree.

The Rescaling Relationship

Suppose one centered predictor is replaced by $x'=c x$, where $c\neq 0$. To preserve the same fitted contribution, its coefficient must change in the opposite direction: $\beta' x'=\beta x$. This parameter change explains how units alter the penalty.

\beta'= \frac{\beta}{c}, \qquad \lambda|\beta'|=\frac{\lambda}{|c|}|\beta|.

Thus, when the model is expressed using the original predictor $x$, fitting with penalty $\lambda|\beta'|$ after rescaling is equivalent to penalizing its original-unit coefficient $\beta$ by $\lambda/|c|$. Multiplying a predictor by a large positive number reduces the effective penalty on its original-unit coefficient; dividing it by a large number increases that effective penalty. This does not mean that the fitted coefficient in the rescaled model is numerically larger. Its value is in the new units.

Write the predictor transformation.
Record the rescaling as $x'=cx$, including the value and sign of $c$.

Translate the coefficient.
Set $\beta=c\beta'$ so that the fitted contribution is unchanged.

Translate the penalty.
Substitute $\beta'=\beta/c$ into the L1 penalty and check whether the effective penalty changed.

A One-Predictor Formula and Its Justification

A simple formula makes the selection threshold visible. Assume the response and predictor are centered, there are no other predictors in the fit, and define the empirical second moment $q=(1/n)\sum_i x_i^2>0$ and empirical cross-product $r=(1/n)\sum_i x_i y_i$. Under the objective above, the coefficient minimizes a quadratic loss plus $\lambda|\beta|$.

\widehat{\beta} = \frac{\operatorname{sign}(r)(|r|-\lambda)_+}{q}, \qquad (a)_+=\max(a,0).

To justify the formula, expand the part of the objective that depends on $\beta$. The terms that depend on it are $\tfrac12q\beta^2-r\beta+\lambda|\beta|$. If $r>\lambda$, the minimizing coefficient is positive, so differentiating gives $q\beta-r+\lambda=0$, or $\beta=(r-\lambda)/q$. If $r<-\lambda$, the coefficient is negative, and the derivative gives $q\beta-r-\lambda=0$, or $\beta=(r+\lambda)/q$. When $-\lambda\leq r\leq\lambda$, the minimum occurs at zero: neither one-sided derivative points away from zero toward a lower objective. These three cases give the displayed expression.

In particular, selection occurs exactly when $|r|>\lambda$ in this one-predictor setting. The denominator $q$ affects the nonzero coefficient's magnitude, but not this threshold. With multiple correlated predictors, this single-predictor rule does not by itself determine the full lasso solution; the predictors' joint relationships matter.

Worked Example: Rescaling a Predictor by Two

Suppose a centered predictor has empirical second moment $q=1$ and cross-product $r=0.45$ with a centered response. Fit the one-predictor lasso with $\lambda=0.30$. Then replace the predictor by $x'=2x$, keep $\lambda=0.30$, and calculate the new coefficient and its equivalent coefficient on the original predictor. State whether the predictor is selected in each fit.

Method. Apply the one-predictor soft-threshold formula in each set of units. The common mistake to avoid is to convert the rescaled coefficient in the wrong direction.

Original units. Since $r=0.45>0.30$, the coefficient is $$ \widehat{\beta}=\frac{0.45-0.30}{1}=0.15. $$ It is nonzero, so the predictor is selected.

Rescaled predictor. The cross-product becomes $r'=2(0.45)=0.90$, and the empirical second moment becomes $q'=2^2(1)=4$. Therefore, $$ \widehat{\beta}'=\frac{0.90-0.30}{4}=\frac{0.60}{4}=0.15. $$ This is the coefficient on $x'=2x$. Its fitted contribution is $0.15(2x)=0.30x$, so the equivalent coefficient on the original $x$ is $0.30\widehat{\beta}'=0.30$. It is nonzero, so the predictor is selected after rescaling as well.

Check. The rescaled fit has an original-unit coefficient of $0.30$, not $0.15$. The two fitted contributions differ because the same numerical $\lambda$ penalizes the rescaled coefficient by less when expressed in original units.

Standardization Changes the Penalty

A common approach is to center each predictor and divide it by a scale $s_j>0$. For transparent calculations under the $1/(2n)$ objective, define $s_j$ as the centered root-mean-square scale, so that $s_j^2=(1/n)\sum_i(x_{ij}-\bar{x}_j)^2$. The standardized predictor is $z_{ij}=(x_{ij}-\bar{x}_j)/s_j$, which has empirical second moment one. If $\theta_j$ is its coefficient, then the corresponding original-unit slope is $\beta_j=\theta_j/s_j$.

The standardized model penalizes $|\theta_j|=s_j|\beta_j|$. Consequently, fitting with the same $\lambda$ after standardization corresponds, in original units, to applying the penalty $\lambda s_j|\beta_j|$ to predictor $j$. Predictors with larger scales therefore have a larger penalty per unit of their original coefficient in this comparison. The practical aim of standardization is not to preserve the original-unit penalty; it is to put predictors on comparable scales so that the common penalty acts on comparable standardized coefficients.

In routine data analysis, scales are often computed using a sample standard deviation with denominator $n-1$, rather than the root-mean-square scale just defined. The transformation and penalty relationship still holds when $s_j$ is the scale actually used, but the standardized empirical second moment under a $1/n$ loss convention will not be exactly one. Always use the stated scaling definition when doing a numerical calculation.

Worked Example: Comparing Raw and Standardized Fits

Consider two centered predictors that are orthogonal to each other, with empirical second moments $q_1=9$ and $q_2=1$. Their cross-products with the centered response are $r_1=2.4$ and $r_2=0.4$. Use $\lambda=0.5$, first with the raw predictors and then with each predictor standardized to empirical second moment one. Find the coefficients in both representations.

Method. Orthogonality makes the objective separate into one-predictor problems. The common mistake to avoid is to treat the standardized-model coefficient as though it were already in original units.

Raw predictors. For predictor 1, $$ \widehat{\beta}_1=\frac{2.4-0.5}{9}=\frac{1.9}{9}=0.211111\ldots. $$ For predictor 2, $r_2=0.4<0.5$, so $\widehat{\beta}_2=0$. Thus only predictor 1 is selected.

Standardized predictors. The scales are $s_1=\sqrt{9}=3$ and $s_2=\sqrt{1}=1$. The standardized cross-products are $r_{z_1}=2.4/3=0.8$ and $r_{z_2}=0.4/1=0.4$, while both standardized second moments equal one. Hence, $$ \widehat{\theta}_1=\frac{0.8-0.5}{1}=0.3, \qquad \widehat{\theta}_2=0. $$ Convert the first coefficient back: $\widehat{\beta}_1=\widehat{\theta}_1/s_1=0.3/3=0.1$. The second remains zero. The selected predictor is unchanged here, but its original-unit coefficient changes from $0.211111\ldots$ to $0.1$.

Penalty Factors for Deliberate Weighting

Standardizing gives each predictor a penalty proportional to its scale in original units. A more general option is to assign predictor-specific penalty factors $\gamma_j\geq 0$:

\frac{1}{2n}\sum_{i=1}^{n}\left(y_i-\sum_{j=1}^{p}x_{ij}\beta_j\right)^2 +\lambda\sum_{j=1}^{p}\gamma_j|\beta_j|.

A larger $\gamma_j$ makes predictor $j$ more costly to include, all else equal; $\gamma_j=0$ leaves its coefficient unpenalized. Such choices can encode a deliberate modeling preference, but they do not establish that a predictor is less useful or more credible. Their consequences should be explained, and comparisons of fitted models should account for the chosen penalty factors.

Worked Example: Preserving a Selection Decision Across Units

A centered predictor has empirical second moment $1$, cross-product $0.45$, and penalty factor $\gamma=1$. Let $\lambda=0.5$. First find its coefficient. Then express the predictor as $x'=10x$. Compare the result using penalty factor $1$ with the result using the unit-preserving penalty factor.

Method. Apply the soft-threshold formula with threshold $\lambda\gamma$. The common mistake to avoid is rescaling the predictor but leaving its penalty factor unchanged when the goal is to preserve the original objective.

Original units. The threshold is $\lambda\gamma=0.5$, which exceeds $0.45$, so $\widehat{\beta}=0$. The predictor is not selected.

Rescaled units, unchanged factor. The cross-product is $10(0.45)=4.5$, and the second moment is $10^2(1)=100$. With factor $1$, the threshold is still $0.5$, giving $$ \widehat{\beta}'=\frac{4.5-0.5}{100}=\frac{4}{100}=0.04. $$ This is nonzero. Its equivalent original-unit coefficient is $10(0.04)=0.4$, so leaving the factor unchanged has changed the selection result.

Rescaled units, adjusted factor. To preserve the penalty, set $\gamma'=|10|(1)=10$. The threshold is now $\lambda\gamma'=0.5(10)=5$. Since $4.5<5$, $\widehat{\beta}'=0$, matching the original fit.

Check. The adjusted factor preserves the penalty and the zero-coefficient decision. The unchanged factor does not: it makes the rescaled predictor easier to select.

Practical Interpretation and Exam Traps

Standardization is a modeling choice with consequences, not a cosmetic rewrite. It often makes a common $\lambda$ more comparable across predictors, but it changes the penalty relative to the original units. A coefficient reported after fitting standardized predictors must be translated before it is described as an effect per dollar, per year, or per other original measurement unit. If predictors are centered as well as scaled, the intercept must also be interpreted on the original response scale; centering does not mean the uncentered response has mean zero.

In practice, software can standardize predictors internally and may return coefficients on the original scale. The behavior and available penalty-factor options vary, so check the software's objective convention, standardization setting, and coefficient reporting. When evaluating predictive performance, estimate any data-dependent centering and scaling values from the training data only, then apply those same values to validation or test data.

Wrong conversion direction. If $x'=cx$, then $\beta'= \beta/c$, while the equivalent original-unit coefficient is $\beta=c\beta'$. Dividing the new coefficient by $c$ again is incorrect.
Ignoring changed moments. Under $x'=cx$, the cross-product changes by $c$ and the second moment by $c^2$. Updating only one gives an incorrect fitted coefficient.
Assuming selection is unit-invariant. With the same numerical $\lambda$ and no penalty-factor adjustment, it need not be. Scaling can move a predictor across the selection threshold.
Comparing coefficients without units. A coefficient on a standardized predictor is not directly the effect per original measurement unit; divide by the scale to recover the original-unit slope.

Key takeaway: Lasso penalizes coefficient magnitude, so predictor units affect effective shrinkage and can affect selection. Standardization changes the penalty in original units; predictor-specific penalty factors can deliberately adjust it or preserve it under a unit conversion.

These scaling relationships are aligned with the SOA Exam PA (Predictive Analytics) syllabus. In a model comparison, state how predictors were scaled, interpret coefficients in the correct units, and distinguish a change in units from a change in the underlying predictive information.

Check Your Understanding

For each problem, use the $1/(2n)$ lasso convention and show the threshold and any coefficient conversion.

A centered predictor has empirical second moment $4$, cross-product $1.2$, and $\lambda=0.4$. Find its coefficient. Then use $x'=2x$ with the same $\lambda$; give the coefficient on $x'$, its equivalent coefficient on $x$, and the selection decision. (Answer: Original coefficient $0.2$; rescaled coefficient $0.125$; equivalent original-unit coefficient $0.25$; selected in both fits.)
A centered predictor has empirical second moment $16$, cross-product $2.4$, and $\lambda=0.6$. Find the raw coefficient and then the coefficient after scaling to empirical second moment one. (Answer: Raw coefficient $0.1125$; standardized coefficient $0$, so it is not selected after standardization.)
A centered predictor has second moment $1$, cross-product $0.7$, and $\lambda=0.4$. What happens to selection after replacing it with $x'=0.5x$ while keeping $\lambda$ unchanged? Give the rescaled coefficient. (Answer: The original coefficient is $0.3$ and is selected; after rescaling the cross-product is $0.35$, the second moment is $0.25$, and the coefficient is $0$, so it is not selected.)
A centered predictor has second moment $1$, cross-product $0.5$, penalty factor $1$, and $\lambda=0.4$, so its original coefficient is $0.1$. Rescale to $x'=3x$. What penalty factor preserves the original objective, and what coefficient does the rescaled fit give? (Answer: Use penalty factor $3$; the rescaled coefficient is $0.1/3=0.0333\ldots$, equivalent to original coefficient $0.1$.)
Two orthogonal centered predictors have scales $s_1=5,s_2=1$, original-unit cross-products $1.5,0.45$, and $\lambda=0.4$. Compare which predictors are selected after standardization. (Answer: Standardized cross-products are $0.30$ and $0.45$; predictor 1 is not selected and predictor 2 is selected with standardized coefficient $0.05$, corresponding to original-unit coefficient $0.05$.)

What You'll Learn