Tutorials › Exam PA - Predictive Analytics › Why Predictor Standardization Changes the Penalty

Home  /  Tutorials  /  Exam PA / Tuning and Regularized Model Use

Exam PA · Tuning and Regularized Model Use · Tutorial 459 of 966

Why Predictor Standardization Changes the Penalty

Learn why standardization changes the effective penalty, how glmnet handles scaling, and how to interpret coefficients on their original scales.

Advanced 10 min read

What You'll Learn

  • Derive the original-scale penalty induced by standardizing predictors for ridge and lasso
  • Calculate shrinkage for a centered predictor under both penalties
  • Explain why standardization makes penalization less sensitive to measurement units
  • Describe glmnet’s predictor and Gaussian-response standardization
  • Convert standardized coefficients into slopes expressed in original units
  • Identify common errors when comparing penalty values across differently scaled models

Why Predictor Scale Matters

Ridge and lasso penalize coefficient size. But a coefficient’s size depends on how its predictor is measured: a one-unit change in dollars is not the same as a one-unit change in thousands of dollars. Consequently, applying the same penalty to coefficients can shrink otherwise comparable predictors by different amounts. Standardizing predictors changes which coefficient scale the penalty acts on.

The calculations below use a squared-error objective with an intercept and centered predictors and response. Centering lets us focus on slope coefficients: the intercept is not penalized, and its fitted value is the response mean. Let \(x_j\) be a centered predictor, \(y\) a centered response, and \(s_j\) the predictor’s root-mean-square scale, \(s_j=\sqrt{\sum_i x_{ij}^2/n}\). Define \(z_j=x_j/s_j\), so that \(\sum_i z_{ij}^2/n=1\).

Definition: Predictor standardization replaces \(x_j\) with \(z_j=x_j/s_j\), where \(s_j\) is its scale. If the model is written as \(\theta_j z_j\), then the same fitted contribution in the original predictor units is \(\beta_j x_j\), with \(\theta_j=s_j\beta_j\). Standardization therefore changes the coefficient used by the penalty, even though it need not change the set of predictions the model can represent.

For clarity, use the following common objective convention, where \(\lambda\geq 0\), and \(\alpha=0\) gives ridge while \(\alpha=1\) gives lasso:

$$ \frac{1}{2n}\sum_{i=1}^n (y_i-\beta_0-\sum_j x_{ij}\beta_j)^2 +\lambda\left(\alpha\sum_j|\beta_j|+\frac{1-\alpha}{2}\sum_j\beta_j^2\right). $$

The factor \(1/(2n)\) and the factor \(1/2\) in the ridge penalty are conventions that fix the numerical meaning of \(\lambda\). Other presentations may place constants differently. Comparisons of penalty values are meaningful only when the objective convention and preprocessing are also specified.

What Standardization Does to Each Penalty

Substitute \(\theta_j=s_j\beta_j\), or equivalently \(\beta_j=\theta_j/s_j\). The fitted values are unchanged because \(x_{ij}\beta_j=z_{ij}\theta_j\). But the penalty, when expressed in standardized-model coefficients, is different from an equal penalty on original-scale coefficients.

$$ \text{Ridge: }\sum_j\theta_j^2=\sum_j s_j^2\beta_j^2, \qquad \text{Lasso: }\sum_j|\theta_j|=\sum_j s_j|\beta_j|. $$

Thus, fitting the standardized predictors with a common ridge penalty corresponds, on the original scale, to a ridge penalty weighted by \(s_j^2\). For lasso, the corresponding original-scale weights are \(s_j\). This is not a flaw in either method: it is a consequence of choosing the scale on which coefficient sizes are compared.

Worked Example: Ridge Shrinkage Before and After Scaling

A centered predictor has \(s=2\), \(x^\mathsf{T}x/n=4\), and \(x^\mathsf{T}y/n=12\). Fit a one-predictor ridge model with \(\lambda=2\), first using \(x\) as recorded and then using \(z=x/2\). Use the objective with squared-error term \(1/(2n)\) and ridge penalty \(\lambda\beta^2/2\). Find each coefficient and compare the slopes in the original predictor units.

Method. Differentiate the objective with respect to the slope and set the derivative to zero. For a centered single predictor, the ridge solution is

$$ \hat{\beta}_{\text{ridge}} =\frac{x^\mathsf{T}y/n}{x^\mathsf{T}x/n+\lambda}. $$

With the recorded predictor, the numerator is \(12\) and the denominator is \(4+2=6\). Therefore, \(\hat{\beta}_{\text{ridge}}=12/6=2\).

For the standardized predictor, \(z^\mathsf{T}z/n=1\), and \(z^\mathsf{T}y/n=(x^\mathsf{T}y/n)/2=6\). The coefficient on \(z\) is \(\hat{\theta}_{\text{ridge}}=6/(1+2)=2\). Since \(z=x/2\), its equivalent slope on \(x\) is \(\hat{\beta}=\hat{\theta}/2=1\).

The standardized fit therefore has slope \(1\) per original unit of \(x\), compared with \(2\) in the unstandardized fit. The check is that the unpenalized slope is \(12/4=3\); both penalized slopes are smaller, but the standardized fit shrinks more in original units at this same numerical \(\lambda\). The common mistake is to report the standardized coefficient \(2\) as the slope per original unit. It is the slope per one unit of \(z\), not per one unit of \(x\).

Worked Example: Lasso Thresholds Depend on Predictor Scale

A centered predictor has \(x^\mathsf{T}x/n=9\), \(x^\mathsf{T}y/n=18\), and hence \(s=3\). Fit a one-predictor lasso model with \(\lambda=4\), using the squared-error term \(1/(2n)\). Calculate the slope when using \(x\), then when using \(z=x/3\), and express the second result on the original scale.

Method. In a centered one-predictor lasso model, the slope is obtained by soft-thresholding the predictor-response cross-product:

$$ \hat{\beta}_{\text{lasso}} =\frac{\operatorname{sign}(x^\mathsf{T}y/n) \left(|x^\mathsf{T}y/n|-\lambda\right)_+}{x^\mathsf{T}x/n}, \qquad (a)_+=\max(a,0). $$

For \(x\), the cross-product is positive. The numerator after thresholding is \(18-4=14\), and the denominator is \(9\). Thus, \(\hat{\beta}=14/9=1.5556\), rounded to four decimals.

For \(z=x/3\), the cross-product becomes \(18/3=6\), and \(z^\mathsf{T}z/n=1\). Its lasso coefficient is \(\hat{\theta}=6-4=2\). Converting to the original units gives \(\hat{\beta}=\hat{\theta}/3=2/3=0.6667\), rounded to four decimals.

The original-scale lasso penalty induced by fitting \(z\) is \(4|3\beta|=12|\beta|\). Accordingly, the original-scale threshold is \(12\), giving \((18-12)/9=6/9=2/3\), as before. The check confirms the conversion. A common error is to subtract \(\lambda=4\) from the original-scale cross-product in the standardized fit; the threshold on the original scale is \(\lambda s=12\), not \(4\).

Why These Formulas Hold

The ridge formula follows directly from minimizing the one-predictor objective. With centered \(x\) and \(y\), the intercept is zero in this centered calculation, so the objective as a function of \(\beta\) is

$$ Q(\beta)=\frac{1}{2n}(y-x\beta)^\mathsf{T}(y-x\beta) +\frac{\lambda}{2}\beta^2. $$

Differentiate, set the derivative to zero, and solve:

$$ \frac{dQ}{d\beta} =-\frac{x^\mathsf{T}y}{n} +\frac{x^\mathsf{T}x}{n}\beta+\lambda\beta=0 \quad\Longrightarrow\quad \hat{\beta} =\frac{x^\mathsf{T}y/n}{x^\mathsf{T}x/n+\lambda}. $$

For lasso, the absolute-value penalty has a corner at zero. When the coefficient is positive, the derivative of \(\lambda|\beta|\) is \(+\lambda\); when negative, it is \(-\lambda\). A nonzero positive solution must therefore satisfy \(x^\mathsf{T}y/n-x^\mathsf{T}x\beta/n-\lambda=0\). It exists only if \(x^\mathsf{T}y/n>\lambda\), yielding \((x^\mathsf{T}y/n-\lambda)/(x^\mathsf{T}x/n)\). The negative case is symmetric, and if \(|x^\mathsf{T}y/n|\leq\lambda\), zero minimizes the objective. Combining these cases gives the soft-threshold formula. This derivation assumes one centered predictor and the stated objective scaling; with multiple correlated predictors, the coordinate solutions are not generally obtained by applying this formula independently.

Standardization in glmnet and Coefficient Interpretation

In glmnet, standardize=TRUE is the default for predictors. In the usual model with an intercept, the predictors are centered and scaled for fitting, while the coefficients returned to the user are on the original predictor scales. Thus, if a coefficient is returned for an input measured in dollars, it describes the fitted change per dollar, not per standardized unit. Setting standardize=FALSE changes the effective penalty unless the predictors were already on comparable scales.

For the Gaussian family, glmnet also standardizes the response to unit variance when constructing its sequence of \(\lambda\) values, then rescales the fitted coefficients back to the response’s original units. This does not mean that every response type is standardized in the same way: binomial and Poisson responses, for example, do not have the same continuous-response scaling interpretation. When comparing fits, record the family, predictor preprocessing, response units, and penalty convention. In particular, the same numerical \(\lambda\) with different predictor scaling need not represent the same amount of shrinkage.

Worked Example: Converting a Gaussian Fit Back to Original Units

A hypothetical Gaussian model uses predictor \(x\), with mean \(50\) and scale \(10\), and response \(y\), with mean \(5{,}000\) dollars and standard deviation \(1{,}000\) dollars. In standardized coordinates, suppose the fitted relationship is \(y^*=0.30+0.08z\), where \(z=(x-50)/10\) and \(y^*=(y-5{,}000)/1{,}000\). Convert the fit to dollars and original predictor units, then predict at \(x=60\).

Method. Undo response scaling first: \(y=5{,}000+1{,}000y^*\). Substitute the standardized fitted relationship:

$$ \hat{y}=5{,}000+1{,}000(0.30+0.08z) =5{,}300+80z. $$

Now substitute \(z=(x-50)/10\). The coefficient per original unit of \(x\) is \(80/10=8\), and the original-scale intercept is \(5{,}300-8(50)=4{,}900\). Therefore, \(\hat{y}=4{,}900+8x\).

At \(x=60\), \(z=(60-50)/10=1\). The standardized prediction is \(y^*=0.30+0.08(1)=0.38\), so \(\hat{y}=5{,}000+1{,}000(0.38)=5{,}380\) dollars. The original-scale equation gives \(4{,}900+8(60)=5{,}380\) dollars as a check. The common mistake is to treat \(0.08\) as dollars per one original unit of \(x\). It is the change in standardized response per one standardized predictor unit; after conversion, the slope is \(8\) dollars per unit of \(x\).

Practical Consequences and Exam Trap

Standardizing predictors is especially useful when their units or natural scales differ. Without it, a common coefficient penalty can favor predictors partly because of their units rather than because they offer more predictive information. Standardization makes the penalty act on comparable predictor-scale changes; it does not guarantee equal shrinkage in a correlated multivariable model, nor does it make coefficients causal effects.

Factor indicators require care. Standardizing a zero-one indicator changes its scale according to its prevalence, so the penalty’s effect on that indicator can depend on the proportion in each level. Whether this is a desirable comparison depends on the modeling goal and encoding. The coefficient returned by software is still to be interpreted according to the original model-matrix column and its units.

A frequent exam trap is to compare two models’ \(\lambda\) values as if they implied identical shrinkage when one model standardized predictors and the other did not. They do not, in general. In a one-predictor ridge fit, scaling \(x\) by \(c\) changes \(x^\mathsf{T}x/n\) by \(c^2\); in lasso it changes the cross-product and threshold relationship by \(c\). Always identify the parameterization and convert coefficients before interpreting them.

Key takeaway: Standardization changes the coefficient scale being penalized. For \(z_j=x_j/s_j\), ridge on standardized coefficients corresponds to an original-scale penalty proportional to \(s_j^2\beta_j^2\), while lasso corresponds to \(s_j|\beta_j|\). Convert coefficients back to the units of the predictors and response before explaining their meaning.

Check Your Understanding

For each problem, use the stated objective convention and distinguish standardized coefficients from original-scale slopes.

  1. A centered predictor has \(s=4\), \(x^\mathsf{T}x/n=16\), and \(x^\mathsf{T}y/n=40\). Under ridge with \(\lambda=4\), find the coefficient using the raw predictor and the equivalent original-scale coefficient when fitting its standardized version. (Answer: raw \(=40/(16+4)=2\); standardized coefficient \(=10/(1+4)=2\), equivalent raw-scale coefficient \(=2/4=0.5\).)
  2. A centered predictor has \(s=2\), \(x^\mathsf{T}x/n=4\), and \(x^\mathsf{T}y/n=12\). Under lasso with \(\lambda=3\), find the coefficient using the raw predictor and the original-scale coefficient from a standardized fit. (Answer: raw \(=(12-3)/4=2.25\); standardized coefficient \(=6-3=3\), equivalent raw-scale coefficient \(=3/2=1.5\).)
  3. A Gaussian response is standardized by subtracting its mean and dividing by \(2{,}500\). A fitted coefficient on a standardized predictor is \(0.12\). What is the corresponding response-scale change for a one-standard-deviation increase in that predictor? (Answer: \(0.12(2{,}500)=300\) response units.)
  4. A predictor has scale \(s=6\), and a standardized lasso fit returns coefficient \(\theta=1.8\). What is the corresponding coefficient per original predictor unit? (Answer: \(\beta=\theta/s=1.8/6=0.3\).)
  5. In a centered one-predictor ridge model, \(x^\mathsf{T}x/n=25\), \(x^\mathsf{T}y/n=50\), \(s=5\), and \(\lambda=5\). Calculate the raw-predictor ridge slope and the original-scale slope after standardizing \(x\). Explain which is smaller. (Answer: raw \(=50/(25+5)=5/3\); standardized coefficient \(=10/(1+5)=5/3\), so its original-scale slope is \((5/3)/5=1/3\), which is smaller.)

Previous

Next

Back to Tutorials
Clinical Biostats