Scaling Predictors Fairly for Penalized Models

Why Predictor Scale Matters

A coefficient is measured in units determined by its predictor. A coefficient of 0.004 per dollar and a coefficient of 4 per year may represent similar changes in predicted outcome if one standard deviation is $1,000 and the other is one year. Yet a penalized regression does not automatically recognize that equivalence: it penalizes coefficient values, not the practical size of the associated effects. Predictors on larger numerical scales can therefore receive less effective shrinkage.

Scaling is not a universal requirement for every model. Its special importance here is that ridge, lasso, and elastic net constrain or penalize coefficients. For an unpenalized GLM, a change of units can be offset by a corresponding coefficient change. For a tree, a positive affine rescaling preserves the ordering of observations and therefore can preserve its candidate splits. The comparison is fair when the same information and evaluation design are used, while each algorithm receives the representation it needs.

Definition: For a quantitative predictor $x_j$, training-set standardization uses its training mean $\bar{x}_j$ and training standard deviation $s_j$ to form $z_j=(x_j-\bar{x}_j)/s_j$. The training values of $z_j$ have mean zero and sample standard deviation one. The same $\bar{x}_j$ and $s_j$ must be used to transform validation or scoring observations.

The sample standard deviation is $s_j=\sqrt{\sum_{i=1}^{n}(x_{ij}-\bar{x}_j)^2/(n-1)}$. In practice, some software uses a different convention, such as a divisor of $n$. Either convention can be used if it is documented and applied consistently; the resulting coefficient and penalty values depend on it. A constant predictor has $s_j=0$, so it cannot be standardized by this formula and ordinarily provides no information for estimating variation in the response.

How the Penalty Depends on Units

For illustration, consider a Gaussian regression with an intercept and $p$ predictors. Let $\beta_j$ denote the slope on predictor $j$. Under a common convention, ridge minimizes residual sum of squares plus $\lambda\sum_j\beta_j^2$, while lasso minimizes residual sum of squares plus $\lambda\sum_j|\beta_j|$. The intercept is usually not penalized. Elastic net combines the two penalties. In a GLM, the same scaling issue arises when a penalty is added to the negative log-likelihood rather than to residual sum of squares.

-\ell(\boldsymbol{\beta})+ \lambda\left[ \frac{1-\alpha}{2}\sum_{j=1}^{p}\beta_j^2+ \alpha\sum_{j=1}^{p}|\beta_j| \right], \qquad 0\leq\alpha\leq1

Here $\ell$ is the model log-likelihood, $\lambda$ controls the overall penalty, and $\alpha=0$ gives ridge while $\alpha=1$ gives lasso. The precise constants used in software may differ, but the dependence on coefficient scale remains. Standardizing quantitative predictors makes a unit change in each standardized predictor correspond to one training standard deviation, giving the penalty a more comparable meaning across those predictors.

Key result: If a predictor’s scale is multiplied by a positive constant, its coefficient must be divided by that constant to preserve predictions. Its squared or absolute coefficient penalty then changes. Thus, without scale adjustment, a common penalty parameter does not impose comparable shrinkage across differently scaled predictors.

A Short Derivation: Changing Units Changes the Penalty

Write the raw predictor as $x_j=\bar{x}_j+s_jz_j$, where $z_j=(x_j-\bar{x}_j)/s_j$. A raw-scale linear predictor includes $\beta_jx_j$. Substitution gives $\beta_j\bar{x}_j+\beta_js_jz_j$. The first term can be absorbed into the intercept, so the coefficient on the standardized predictor is $\gamma_j=s_j\beta_j$. This re-expression leaves the fitted linear predictor unchanged.

\beta_jx_j+\beta_0 = (\beta_0+\beta_j\bar{x}_j)+(s_j\beta_j)z_j

The equivalent ridge penalty on this coefficient is $\gamma_j^2=s_j^2\beta_j^2$; the equivalent lasso penalty is $|\gamma_j|=s_j|\beta_j|$. If predictors have different standard deviations, these factors differ across predictors. Standardizing before penalization is therefore not merely a cosmetic change: it defines the scale on which the model treats coefficient sizes as comparable.

Worked Example: Comparing Penalties on Two Units

Method. Compare the penalty contributions for two predictors after expressing both effects as changes per one training standard deviation. Suppose a predictor measured in dollars has $s_1=\$1{,}000$, and a second predictor measured in years has $s_2=1$ year. In a fitted standardized-coordinate representation, their coefficients are both 4. Find the corresponding raw-scale slopes and compare the ridge and lasso contributions. Use the conventions $\sum_j\beta_j^2$ and $\sum_j|\beta_j|$, without the common multiplier $\lambda$.

The standardized coefficient is the change in the linear predictor for a one-standard-deviation increase. From $\gamma_j=s_j\beta_j$, the raw-scale slope is $\beta_j=\gamma_j/s_j$. Thus the dollar slope is $4/1{,}000=0.004$ per dollar, and the year slope is $4/1=4$ per year.

On the standardized scale, the two ridge contributions sum to $4^2+4^2=16+16=32$. Their lasso contributions sum to $|4|+|4|=4+4=8$. On the raw scale, the corresponding ridge contributions would be $0.004^2+4^2=0.000016+16=16.000016$, and the lasso contributions would be $|0.004|+|4|=4.004$.

Check. Both standardized coefficients represent a four-unit linear-predictor change for a one-standard-deviation increase, while the raw slopes use different units. Their raw penalty totals differ from the standardized totals, as expected. The common mistake avoided is interpreting raw coefficient magnitudes as comparable measures of effect without accounting for predictor units.

Fit the Scaler on Training Data Only

A validation or test set represents observations that were not available when the model was trained. Its values must not determine the means and standard deviations used in training. Otherwise, even without using response values, information about the held-out predictor distribution has entered the training procedure. For cross-validation, estimate the scaler separately within each training fold, then apply that fold’s stored values to its held-out fold.

Separate the data.
Make the training and validation or test partitions before estimating any scaling quantities.

Estimate training quantities.
For each quantitative predictor, calculate its mean and standard deviation from the training observations only.

Transform consistently.
Apply the stored training mean and standard deviation to training, validation, and later scoring values. Do not recalculate them on each new batch.

Compare predictions.
Evaluate each candidate on the same held-out observations and target definition. Retain the scaler as part of the penalized model’s fitted procedure.

Worked Example: A Held-Out Value Must Not Set the Scale

Method. A training set has predictor values 2, 4, and 6. A validation observation has value 100. Calculate the validation standardized value using the training statistics, then calculate what it would be if the validation value incorrectly influenced the scaler.

The training mean is $(2+4+6)/3=4$. The training sample standard deviation is $\sqrt{[(2-4)^2+(4-4)^2+(6-4)^2]/(3-1)}=\sqrt{8/2}=2$. Therefore the correctly standardized validation value is $(100-4)/2=48$.

If all four values were used to estimate the scaler, their mean would be $(2+4+6+100)/4=28$. The sum of squared deviations would be $(-26)^2+(-24)^2+(-22)^2+72^2=676+576+484+5{,}184=6{,}920$. The sample standard deviation would be $\sqrt{6{,}920/3}=\sqrt{2{,}306.6667}$, approximately 48.0278. The validation value would then be $(100-28)/48.0278$, approximately 1.4991.

Check. The held-out value is 48 under the training-only scaler, not approximately 1.4991. The latter result is not a valid out-of-sample transformation because the validation observation changed both the mean and standard deviation used to fit the model. The common mistake avoided is treating predictor-only information from a held-out set as harmless.

What Scaling Changes—and What It Does Not

An unpenalized GLM with a linear predictor is unchanged in fitted values by this affine transformation, provided the transformed model includes the adjusted intercept and coefficient. This is a reparameterization, not a change in the information supplied to the model. The same statement need not hold if the model is constrained, regularized, or given a different set of derived terms.

For a tree, standardization with a positive standard deviation preserves the order of the observations. A split at a raw value $c$ corresponds to a split at standardized value $(c-\bar{x})/s$, and it sends the same observations to each side. This does not mean scaling is necessary for ordinary tree splits. It does mean that scaling one candidate model while leaving another unscaled need not give them different underlying information.

Worked Example: Keep GLM Predictions and Tree Partitions Aligned

Method. A training predictor has values 8, 10, 12, and 14. Use sample standardization to convert an unpenalized logistic regression and a hypothetical tree split. The raw-scale logistic model is $\operatorname{logit}(p)=-2+0.25x$. Find the equivalent standardized model, the predicted probability at $x=14$, and the standardized form of the tree rule $x\leq12$.

The training mean is $(8+10+12+14)/4=11$. The sum of squared deviations is $9+1+1+9=20$, so the sample standard deviation is $\sqrt{20/3}$, approximately 2.5820. At $x=14$, $z=(14-11)/2.5820$, approximately 1.1619.

Since $x=11+2.5820z$, substitution into the raw model gives $\operatorname{logit}(p)=-2+0.25(11+2.5820z)=0.75+0.6455z$. At $x=14$, this is $0.75+0.6455(1.1619)$, approximately $0.75+0.75=1.5000$, the same value as $-2+0.25(14)=1.5000$. Thus $p=1/(1+e^{-1.5})$, approximately 0.8176.

The raw tree condition $x\leq12$ becomes $z\leq(12-11)/2.5820$, or $z\leq0.3873$. The four training values map in order to approximately $-1.1619,-0.3873,0.3873,1.1619$, so the first three observations satisfy either split rule and the fourth satisfies neither.

Check. Both logistic parameterizations give a linear predictor of 1.5000 at $x=14$, and both tree rules assign the same training observations to each side. The common mistake avoided is assuming that standardized inputs inherently change an unpenalized GLM or a tree’s split membership.

Making the Comparison Fair

For a comparison with unpenalized GLMs and trees, the goal is not to force every algorithm to use identical numerical columns. The goal is to ensure that scaling itself neither leaks information nor silently changes the comparison’s target, observations, or available predictors. A penalized model can use training-standardized continuous predictors, while an unpenalized GLM or tree can use their original units. Record the transformations so that scoring applies the same fitted procedure.

Pay particular attention to features whose interpretation depends on construction. Standardizing an original predictor and then forming its square or interaction is not generally the same operation as forming the derived feature first and standardizing that feature. Either approach can be defensible, but it defines a different penalty geometry. For categorical indicator columns, scaling also changes the effective penalty assigned to their coefficients; do not assume that the continuous-predictor rule automatically settles how indicators should be treated. State the preprocessing policy and apply it consistently within each training partition.

Model type	Role of scaling	Fair-comparison practice
Ridge, lasso, elastic net	Changes coefficient penalties and can affect shrinkage or selection	Standardize from training data and store the fitted quantities
Unpenalized GLM	Affine rescaling can be offset by coefficient and intercept changes	Keep the same model terms and interpret coefficients in the units used
Tree	Positive affine rescaling preserves ordering and corresponding splits	Do not imply that scaling is needed to make ordinary split values comparable

Scaling is aligned with the SOA Exam PA (Predictive Analytics) syllabus because it connects preprocessing decisions to model fitting and out-of-sample comparisons. The central diagnostic question is not simply, “Were the predictors standardized?” It is: “Were the penalty, the training-only transformation, and the comparison all defined consistently?”

Common exam trap: A candidate compares penalized-model coefficients on raw scales, concludes that a small numerical coefficient is unimportant, or computes scaling statistics using the full data set before validation. Correct the first error by interpreting effects in meaningful units or on the standardized scale; correct the second by fitting each scaler only on the corresponding training data. Keep the unpenalized GLM or tree representation appropriate to that model, without changing the comparison data or predictor information.

Check Your Understanding

For each problem, use the stated sample-standard-deviation convention and distinguish training quantities from held-out values.

Training values for a predictor are 5, 7, 9, and 11. A validation value is 15. Find the training mean, sample standard deviation, and correctly standardized validation value. (Answer: mean 8, standard deviation $\sqrt{20/3}\approx2.5820$, standardized value approximately 2.7111.)
Two predictors have training standard deviations 10 and 5. Their standardized-model coefficients are 3 and $-2$. Find their raw slopes and the sum of their raw squared-coefficient contributions. (Answer: raw slopes 0.3 and $-0.4$; squared contributions sum to $0.09+0.16=0.25$.)
An unpenalized logistic model is $\operatorname{logit}(p)=-1+0.2x$. The training mean and standard deviation are 5 and 2. Write its standardized form and find the probability at $x=9$, rounding the probability to four decimals. (Answer: $\operatorname{logit}(p)=0+0.4z$; at $z=2$, $p=1/(1+e^{-0.8})\approx0.6900$.)
Training values are 4, 6, and 8; a held-out value is 20. Calculate its standardized value using training data, then explain why recalculating the mean and standard deviation after adding 20 would be inappropriate. (Answer: training mean 6, sample standard deviation 2, standardized held-out value 7; including 20 would let held-out predictor information affect the fitted transformation.)
A tree uses the rule $x\leq30$. A training-only scaler has mean 20 and standard deviation 5. Give the equivalent rule on the standardized scale, and state which side contains an observation with $x=25$. (Answer: $z\leq2$; $x=25$ gives $z=1$ and is on the $z\leq2$ side.)

What You'll Learn