Tutorials › Generalized Linear Models › Standardizing Predictors Before Penalization

Home  /  Tutorials  /  Penalized GLM Estimation

Penalized GLM Estimation · Tutorial 904 of 981

Standardizing Predictors Before Penalization

See how standardizing predictors changes the meaning of ridge and lasso penalties, and how to recover coefficients on the original scale without changing fitted values.

Advanced 10 min read

What You'll Learn

  • Express a GLM using centered and scaled predictors.
  • Transform slopes and intercepts between standardized and original scales.
  • Derive the equivalent ridge and lasso penalties on the original coefficients.
  • Check how changing measurement units affects an unstandardized penalty.
  • Identify when standardization leaves fitted values unchanged and when penalty choices still matter.

Why Predictor Units Matter

A GLM penalty acts on coefficients, but coefficient magnitudes depend on how predictors are measured. A slope is measured in linear-predictor units per unit of its predictor: for example, log-odds per year in logistic regression. If years are replaced by decades, the numerical slope changes even when the model describes exactly the same fitted values. A penalty that treats all numerical slopes alike can therefore penalize a model differently just because a predictor was recorded in different units.

Standardizing predictors gives a common scale for the penalty: each slope then describes a change in the linear predictor per one standard deviation of that predictor. This does not make the predictors statistically identical, and it does not dictate that standardization is always appropriate. It does make the consequence of applying a common ridge or lasso penalty explicit. The central task is to keep straight the relationship between the standardized coefficients being penalized and the coefficients used to report the model in its original units.

Definition and Coefficient Transformation

Suppose a GLM has observations \(i=1,\ldots,n\), predictors \(x_{ij}\) for \(j=1,\ldots,p\), an intercept, and a fixed offset \(o_i\). Write its linear predictor as

$$ \eta_i=o_i+\beta_0+\sum_{j=1}^p \beta_j x_{ij}. $$

For each predictor, choose a center \(a_j\) and a positive scale \(s_j\), and define \(z_{ij}=(x_{ij}-a_j)/s_j\). Common choices are the training-sample mean for \(a_j\) and the training-sample standard deviation for \(s_j\). A predictor with zero scale is constant on the data and cannot be standardized this way; it should be handled separately, since its contribution is indistinguishable from an intercept contribution.

Definition: The standardized-predictor parameterization writes the same linear predictor as \(\eta_i=o_i+\gamma_0+\sum_{j=1}^p\gamma_j z_{ij}\). The parameters and original-scale coefficients are related by \(\gamma_j=s_j\beta_j\) and \(\gamma_0=\beta_0+\sum_{j=1}^p a_j\beta_j\). Equivalently, \(\beta_j=\gamma_j/s_j\) and \(\beta_0=\gamma_0-\sum_{j=1}^p a_j\gamma_j/s_j\).

The offset is unchanged. The slope transformation follows from replacing \(x_{ij}\) by \(a_j+s_jz_{ij}\). The intercept must also change because the predictors have been centered; transforming the slopes alone would generally change the linear predictor.

Worked Example: Transforming a Logistic-Regression Coefficient

Suppose a logistic regression uses age \(x\), measured in years, with \(\beta_0=-1\) and \(\beta_1=0.08\). Center age at \(a=50\) years and scale it by \(s=10\) years, so \(z=(x-50)/10\). Find the standardized coefficients and verify the linear predictor at age 60.

The slope per standardized unit is \(\gamma_1=s\beta_1=10(0.08)=0.8\). The intercept is \(\gamma_0=\beta_0+a\beta_1=-1+50(0.08)=3\). Thus the standardized form is \(\eta=3+0.8z\), apart from any offset.

At age 60, \(z=(60-50)/10=1\), so this form gives \(\eta=3+0.8(1)=3.8\). The original form gives \(\eta=-1+0.08(60)=-1+4.8=3.8\). The equality confirms that the change of parameterization preserves the fitted linear predictor. Here \(0.08\) is in log-odds per year, whereas \(0.8\) is in log-odds per 10 years.

How Standardization Changes the Penalty

Let \(L\) denote the likelihood expressed in either parameterization, and consider an objective consisting of negative log-likelihood plus a penalty. Suppose the intercept is not penalized, as is usual for these ridge and lasso objectives. Penalizing the standardized slopes means penalizing \(\gamma_j=s_j\beta_j\), not \(\beta_j\) directly. The resulting objective can be written in either parameterization.

Result: A ridge penalty \(\frac{1}{2}\sum_{j=1}^p\gamma_j^2\) on standardized slopes is exactly \(\frac{1}{2}\sum_{j=1}^p s_j^2\beta_j^2\) on original-scale slopes. A lasso penalty \(\sum_{j=1}^p|\gamma_j|\) is exactly \(\sum_{j=1}^p s_j|\beta_j|\). If the objective uses a common multiplier \(\lambda\), that multiplier applies to either expression. These identities assume positive scales and an unpenalized intercept.

To verify the result, substitute \(\gamma_j=s_j\beta_j\) into the ridge penalty:

$$ \frac{1}{2}\sum_{j=1}^p\gamma_j^2 =\frac{1}{2}\sum_{j=1}^p(s_j\beta_j)^2 =\frac{1}{2}\sum_{j=1}^p s_j^2\beta_j^2. $$

For lasso, \(s_j>0\) gives \(|\gamma_j|=|s_j\beta_j|=s_j|\beta_j|\), and hence

$$ \sum_{j=1}^p|\gamma_j| =\sum_{j=1}^p s_j|\beta_j|. $$

These substitutions prove the penalty identities. Moreover, the coefficient transformation is one-to-one when every \(s_j>0\), and it preserves every \(\eta_i\). Therefore it preserves the likelihood as well. Minimizing the objective over standardized coefficients is thus equivalent to minimizing the corresponding objective over original coefficients with the displayed transformed penalty. This is an equivalence of parameterizations and objectives, not a claim that the numerical coefficients or the penalty itself are unchanged by standardization.

In particular, applying an unweighted penalty directly to original-scale slopes generally gives a different objective. The standardized ridge penalty weights \(\beta_j^2\) by \(s_j^2\), while the standardized lasso penalty weights \(|\beta_j|\) by \(s_j\). A larger predictor scale therefore changes how a given original-scale coefficient contributes to the standardized penalty.

Worked Example: Changing Metres to Centimetres

Suppose a predictor is measured in metres, has scale \(s=2\) metres, and has original-scale slope \(\beta=0.3\) in linear-predictor units per metre. The same measurements recorded in centimetres have scale \(200\) centimetres. Find the corresponding slope in centimetres and compare the ridge penalties.

Since one metre is 100 centimetres, the centimetre value \(x_{\mathrm{cm}}\) is \(100x_{\mathrm{m}}\). To preserve the predictor contribution, \(\beta_{\mathrm{cm}}x_{\mathrm{cm}}=\beta_{\mathrm{m}}x_{\mathrm{m}}\), so \(\beta_{\mathrm{cm}}=0.3/100=0.003\) per centimetre. The standardized slope in either unit is \(s\beta\): in metres it is \(2(0.3)=0.6\), and in centimetres it is \(200(0.003)=0.6\).

An unweighted original-scale ridge penalty \(\beta^2/2\) is \(0.3^2/2=0.045\) in metres but \(0.003^2/2=0.0000045\) in centimetres. Those are different penalties for the same predictor contribution. The standardized penalty is \(0.6^2/2=0.18\) in either unit. The equivalent original-scale weighted penalty confirms this: \(s^2\beta^2/2=2^2(0.3)^2/2=0.18\) in metres, and \(200^2(0.003)^2/2=0.18\) in centimetres.

Returning to the Original Scale

A model fitted with standardized predictors may be easier to fit or compare, while its coefficients may need to be reported in familiar units. Use the inverse transformation, including the intercept adjustment. The following calculation also checks that predictions remain the same at a particular covariate value.

Worked Example: Back-Transforming Lasso Coefficients

Suppose two predictors have centers \(a_1=10\), \(a_2=100\) and scales \(s_1=2\), \(s_2=20\). A fitted model using standardized predictors has \(\gamma_0=1.4\), \(\gamma_1=0.6\), and \(\gamma_2=-0.4\). Convert the model to the original scale, and verify its linear predictor at \(x_1=12\), \(x_2=80\). Ignore any offset.

The original slopes are \(\beta_1=\gamma_1/s_1=0.6/2=0.3\) and \(\beta_2=\gamma_2/s_2=-0.4/20=-0.02\). The intercept is \(\beta_0=\gamma_0-a_1\beta_1-a_2\beta_2=1.4-10(0.3)-100(-0.02)=1.4-3+2=0.4\). Thus the original-scale model is \(\eta=0.4+0.3x_1-0.02x_2\).

At the stated values, \(z_1=(12-10)/2=1\) and \(z_2=(80-100)/20=-1\). The standardized form gives \(\eta=1.4+0.6(1)-0.4(-1)=2.4\). The original-scale form gives \(\eta=0.4+0.3(12)-0.02(80)=0.4+3.6-1.6=2.4\). As a penalty check, the standardized lasso penalty is \(|0.6|+|-0.4|=1.0\); its equivalent original-scale penalty is \(2|0.3|+20|-0.02|=0.6+0.4=1.0\).

Practical Choices and a Common Pitfall

When centers and scales are estimated from data, they are part of the fitted procedure. For prediction on new observations, use the centers and scales obtained from the training data; recomputing them on the new observations changes the coordinate system and, in general, the fitted predictions. If a model is refitted on a different training set, its centers and scales may differ, so comparisons of coefficients should account for that difference.

Whether to standardize is a modeling choice. A common penalty after standardization measures shrinkage relative to the chosen predictor scales; it does not erase differences in scientific meaning, correlation, or data quality. For a binary predictor, for example, scaling changes the coefficient to a contrast per standard deviation rather than per unit change from 0 to 1. The intended penalty and the interpretation of reported coefficients should guide the choice.

  • Common mistake: “The standardized fit has the same slopes and intercept as the original fit because the predictions are unchanged.”
  • Correction: The fitted linear predictors are preserved by transforming all slopes and the intercept. The slopes generally change by \(\gamma_j=s_j\beta_j\), and the intercept changes by \(\gamma_0=\beta_0+\sum_j a_j\beta_j\).
  • Another pitfall: Applying the same unweighted penalty to raw coefficients is not equivalent to penalizing standardized coefficients. The equivalent raw-scale penalty includes scale factors, as derived above.
Key takeaway: Standardization is a change of coordinates for the linear predictor, with \(\gamma_j=s_j\beta_j\) and an accompanying intercept adjustment. It leaves fitted values unchanged when coefficients are transformed, but it changes the penalty unless that penalty is transformed too.

Check Your Understanding

Work through these questions using positive predictor scales.

  1. A predictor is centered at \(a=4\) and scaled by \(s=3\). If its original slope is \(\beta=0.2\), what is the standardized slope, and how much does it contribute to the standardized intercept?
  2. Write the original-scale ridge penalty equivalent to a ridge penalty on standardized slopes when there are two predictors with scales \(s_1\) and \(s_2\).
  3. A lasso fit has standardized slopes \(\gamma_1=-0.5\) and \(\gamma_2=0.3\), with scales \(s_1=5\) and \(s_2=2\). Find both original slopes and verify that the standardized and equivalent original-scale lasso penalties agree.
  4. Why can replacing a predictor measured in metres with the same predictor measured in centimetres change an unweighted raw-coefficient ridge fit?
  5. When using a standardized fit to predict on new data, which centers and scales should be used, and why?

Previous

Next

Back to Tutorials
Clinical Biostats