Why Predictor Units Matter
A ridge or lasso penalty acts on coefficient values, not directly on the biological contribution of a predictor. If one gene is recorded in counts and another in thousands of counts, equivalent changes in fitted values can require coefficients with very different numerical magnitudes. Applying the same penalty to those coefficients can therefore shrink the predictors unequally for reasons that come only from their units.
Let \(X\) be the \(n \times p\) predictor matrix, with \(x_{ij}\) the value of predictor \(j\) for sample \(i\), and let \(y_i\) be the response. With an intercept that is not penalized, the penalized least-squares objectives are
Here \(\lambda \geq 0\) controls the penalty. The loss measures mismatch between observed and fitted responses; the penalty discourages large coefficients. The key issue is that changing predictor units can change the penalty even when fitted values are unchanged.
How Rescaling Changes a Penalized Fit
First consider one predictor and write \(x=a+cz\). A model with intercept and coefficient \(b\) on \(x\) has fitted value \(\beta_0+bx\). Replacing \(x\) with \(z\) gives the same fitted value when the transformed coefficient is \(\theta=cb\) and the transformed intercept is \(\gamma_0=\beta_0+ab\):
Thus centering and scaling can preserve every fitted value, but the coefficient appearing in the penalty changes. A ridge penalty on the transformed coefficient is \(\theta^2=c^2b^2\), while a lasso penalty is \(|\theta|=c|b|\). If each transformed predictor has its own scale \(c_j\), the corresponding penalties expressed in the original coefficients are \(\lambda\sum_j c_j^2\beta_j^2\) for ridge and \(\lambda\sum_j c_j|\beta_j|\) for lasso.
This gives the central interpretation: applying the same penalty to standardized coefficients is equivalent to applying predictor-specific penalty weights to coefficients in the original units. Standardization is not a change to the observed response or a new source of information. It changes how the penalty treats a given pattern of fitted values.
Worked Example: The Same Predictor in Meters and Centimeters
Suppose a hypothetical response is modeled using a length predictor. One version records length in meters, \(x\), and another records the same values in centimeters, \(x^{*}=100x\). Consider two equivalent fits: a coefficient of \(3\) response units per meter in the first model, and a coefficient of \(0.03\) response units per centimeter in the second. For any sample, the predictor contributions agree because \(3x=0.03(100x)\).
Set \(\lambda=0.2\). The ridge penalty contribution in the meters model is \(0.2(3^2)=0.2(9)=1.8\). In the centimeters model it is \(0.2(0.03^2)=0.2(0.0009)=0.00018\). The lasso penalty contributions are \(0.2|3|=0.6\) and \(0.2|0.03|=0.006\), respectively.
The two coefficient choices give identical predictions and residual sums of squares, but the raw objectives assign very different penalties to them. Consequently, minimizing the objective can favor the centimeter-coded predictor for a reason unrelated to its predictive information. Scaling predictors consistently before applying a common penalty addresses this units problem.
Standardizing Continuous Predictors
For a continuous predictor, a common choice is to subtract its sample mean and divide by its sample standard deviation. This gives a predictor with sample mean zero and sample standard deviation one, provided the predictor is not constant. When an intercept is included, centering makes the intercept correspond to the fitted response at the predictor means. Scaling puts coefficients on a common per-standard-deviation scale for penalization.
Worked Example: Standardizing a Continuous Expression Feature
Suppose a hypothetical expression feature has values \(4, 6, 8, 10, 12\) across five samples, in a fixed measurement unit. Its mean is \(\bar{x}=(4+6+8+10+12)/5=8\). The sum of squared deviations is \(16+4+0+4+16=40\), so its sample standard deviation is \(s=\sqrt{40/(5-1)}=\sqrt{10}\approx 3.162\).
The standardized values are \(z_i=(x_i-8)/3.162\), approximately \(-1.265,-0.6325,0,0.6325,1.265\). Suppose a coefficient of \(b=0.5\) response units per original expression unit is under consideration. The equivalent coefficient on \(z\) is \(\theta=sb=3.162(0.5)=1.581\), with the intercept adjusted by \(8b=4\) response units. The two coefficient-and-intercept pairs yield the same fitted values.
The ridge penalty on the original coefficient with \(\lambda=0.1\) is \(0.1(0.5^2)=0.025\). On the standardized coefficient it is \(0.1(1.581^2)\approx 0.250\). The lasso contributions are \(0.1|0.5|=0.050\) and \(0.1|1.581|\approx 0.1581\). These are different penalties for the same fit, illustrating why the choice of units must be made deliberately before fitting.
After standardization, the coefficient \(\theta\) describes the fitted response change associated with a one-sample-standard-deviation increase in the feature, holding the other predictors fixed. The original coefficient \(b\) instead describes the change per original measurement unit.
Binary and Genotype Predictors
Binary predictors require an explicit scaling decision. For an indicator coded \(0\) or \(1\), leaving it unscaled makes its coefficient the fitted difference between the two groups. Centering it changes the reference point for the intercept but not that group difference. Dividing by its standard deviation changes the coefficient to a per-standard-deviation contrast and changes how the penalty treats that feature.
Genotype dosage commonly takes values \(0,1,2\), representing the number of copies of a specified allele. Its raw coefficient is the fitted response change per additional allele, under the model’s additive linear specification. If dosage is centered and divided by its sample standard deviation, its penalized coefficient is instead on a standard-deviation scale. This can put genotype features on a comparable scale to continuous expression features, but it does not make the two coefficient interpretations identical.
Worked Example: Scaling a Genotype-Dosage Feature
Consider a hypothetical genotype feature with dosages \(0,0,0,1,1,2\) in six samples. Its mean is \(\bar{x}=4/6=0.6667\). The sum of squared deviations is \(10/3\), so its sample variance is \((10/3)/(6-1)=2/3\), and its sample standard deviation is \(s=\sqrt{2/3}\approx0.8165\).
Suppose the coefficient per additional allele in the original model is \(b=0.4\) response units. The equivalent coefficient for the standardized dosage is \(\theta=sb=0.8165(0.4)\approx0.3266\). Centering adjusts the intercept by \(\bar{x}b=0.6667(0.4)\approx0.2667\) response units. The fitted values remain unchanged.
With \(\lambda=0.1\), the raw-coefficient ridge contribution is \(0.1(0.4^2)=0.016\); the standardized-coefficient contribution is \(0.1(0.3266^2)\approx0.01067\). The lasso contributions are \(0.1(0.4)=0.040\) and \(0.1(0.3266)\approx0.03266\). Standardizing has changed the penalty even though it has not changed the fit represented by these coefficients.
For these six samples, the standardized values are approximately \(-0.8165,-0.8165,-0.8165,0.4082,0.4082,1.633\). The observation with dosage 2 is more than one and a half sample standard deviations above the mean. With rare variants, a small number of carriers can therefore have substantial influence after standardization. Variance scaling can equalize the penalty scale, but it cannot create information about a feature observed in very few samples.
Choosing and Checking a Scaling Rule
There is no universally correct scaling rule for every genomic analysis. If the goal is to apply a common ridge or lasso penalty to effects per standard deviation, centering and scaling continuous predictors is a natural choice. For a binary indicator or genotype dosage, scaling can serve the same purpose, but it changes interpretation from a group or per-allele contrast to a standard-deviation contrast. Leaving binary features unscaled can be preferable when the per-group or per-allele effect is the intended penalized quantity. The decision should be made before interpreting coefficients and should be applied consistently with the scientific aim.
A constant predictor has \(s_j=0\), so division by its standard deviation is undefined; it cannot provide within-sample information and must not be standardized by this formula. Features with very small standard deviations deserve inspection because dividing by a small number can produce extreme standardized values. For binary and dosage features, check their observed frequencies and the number of distinct values rather than assuming that variance scaling is harmless.
Estimate centers and scales using only the training data when evaluating prediction by a held-out set or cross-validation. For each validation split, compute \(a_j\) and \(c_j\) from that split’s training portion and apply those same values to its validation portion. Computing them from all samples lets information from validation predictors enter the training procedure. Although this does not use validation responses, it violates a clean separation of training and evaluation data and can distort the estimated predictive performance.
Identify whether it is continuous, binary, dosage-valued, constant, or has very low variation.
Decide whether the common penalty should apply to effects per standard deviation, per original unit, per group contrast, or per allele.
Record each center and scale, apply the same transformation to corresponding validation data, and do not divide a constant feature by zero.
Report coefficients on the standardized scale if that is the fitted representation, or transform them back before giving effects per original unit.
Comparing Scaling Choices and Common Pitfalls
| Choice | What the coefficient represents | Penalty implication |
|---|---|---|
| Raw continuous predictor | Change per original measurement unit | Features with different units can be penalized unequally for comparable fitted changes |
| Centered and standardized continuous predictor | Change per sample standard deviation | A common penalty acts on coefficients expressed on a comparable scale |
| Unscaled binary indicator or dosage | Group difference or change per allele | Penalty is attached directly to that contrast, which may be the intended choice |
| Centered and standardized binary feature | Change per sample standard deviation | Penalty scale is altered; rare features may be unstable because few samples carry the feature |
A frequent mistake is to say that standardization “does not affect the model” because the fitted values can be preserved by transforming the coefficients. That is true for the unpenalized prediction equation, but not for a penalized fit with the same numerical \(\lambda\): the penalty changes with the coefficients. Another mistake is to assume that every binary feature must be standardized. The correct choice depends on which effect scale should receive equal penalization and on how much information the feature contains.
A useful diagnostic is to compare the scales of predictors before fitting, then confirm that the stored centers and scales are finite and nonzero where used. After fitting, verify the coefficient scale before interpreting it: a coefficient for a standardized genotype is not automatically an effect per allele. Finally, check that preprocessing was repeated within each training split during validation. If changing units changes which features are strongly shrunk despite equivalent predictor information, the scaling and penalty convention are not aligned.
Check Your Understanding
Try each question, then compare your reasoning with the solutions.
- A predictor is multiplied by 10 while the fitted values are held fixed. By what factor does its equivalent coefficient change, and how do its ridge and lasso penalty contributions change under a common \(\lambda\)?
- A centered continuous feature has sample standard deviation \(2\), and its original-scale coefficient is \(0.3\). What is the equivalent standardized coefficient?
- Why might an analyst leave a binary genotype feature unscaled rather than divide by its sample standard deviation?
- In cross-validation, where should the center and scale for a predictor be computed?
Solutions. (1) If \(x^{*}=10x\), the equivalent coefficient is \(\beta^{*}=\beta/10\). Its ridge penalty contribution falls by a factor of \(10^2=100\), and its lasso contribution falls by a factor of \(10\), because the penalties use the squared and absolute coefficient, respectively. (2) The standardized coefficient is \(s\beta=2(0.3)=0.6\), with the intercept adjusted as needed to preserve fitted values. (3) Leaving it unscaled makes the coefficient and penalty correspond to a group contrast or per-allele effect. Standardization instead makes the coefficient a per-standard-deviation effect and can magnify the role of a feature carried by few samples. (4) Compute the center and scale from the training portion of each split, then apply those same values to that split’s validation portion.