Why Equal Lambda Values May Not Mean Equal Shrinkage
A penalty acts on coefficients, but a coefficient’s numerical size depends on how its predictor is measured and represented. A coefficient for dollars is numerically different from a coefficient for thousands of dollars, even when both describe the same fitted relationship. Consequently, comparing lambda values across differently prepared designs can give a misleading impression of how strongly a model is regularized.
The key question is not just “What is lambda?” It is what coefficient scale and predictor representation does the penalty act on? This tutorial focuses on that question for ridge, lasso, and elastic-net fits, including the effect of glmnet’s predictor standardization.
The Objective and the Scale Conversion
For a Gaussian response, a common elastic-net objective, using an unweighted sample of \(n\) observations, is
Here \(\beta_0\) is an unpenalized intercept, \(\beta_j\) is the coefficient of predictor column \(j\), \(\lambda\geq 0\) controls the overall penalty, and \(0\leq\alpha\leq1\) sets the lasso-ridge mixture. At \(\alpha=1\), the penalty is lasso; at \(\alpha=0\), it is ridge. This expression states the Gaussian objective for this discussion; other model families use a different loss term, so the loss scale and lambda values should not be assumed interchangeable across families.
Suppose a predictor column \(x_j\) is replaced by \(x_j^*=c x_j\), where \(c\neq0\). To preserve the same linear predictor, its coefficient must become \(\beta_j^*=\beta_j/c\), since \(x_j\beta_j=x_j^*\beta_j^*\). But the penalty contribution changes: the lasso term becomes \(\lambda\alpha|\beta_j|/|c|\), and the ridge term becomes \(\lambda(1-\alpha)\beta_j^2/(2c^2)\). Thus, without standardization or a corresponding adjustment, a unit change in the predictor changes the penalty on an equivalent fitted effect.
Worked Example: Converting Dollars to Thousands of Dollars
A hypothetical pricing model uses annual repair cost as a predictor. In one design, the column \(x\) is measured in thousands of dollars; in another, \(x^*=1000x\) is measured in dollars. A fitted contribution is \(0.4x\). Compare the equivalent coefficients and lasso penalty contributions when \(\lambda=0.2\), \(\alpha=1\), and predictors are not standardized.
Method. Preserve the fitted contribution, then evaluate the lasso term \(\lambda|\beta|\) for each coefficient. This isolates the effect of units; it does not assert that either design has been optimally fitted.
In thousands of dollars, \(\beta=0.4\), so the contribution is \(0.4x\). Since \(x^*=1000x\), the equivalent coefficient in dollars is \(\beta^*=0.4/1000=0.0004\). The fitted contribution is unchanged: \(0.0004x^*=0.0004(1000x)=0.4x\).
The original-scale lasso contribution is \(0.2|0.4|=0.08\). In dollars it is \(0.2|0.0004|=0.00008\). The ratio is \(0.08/0.00008=1000\). Although the predictions match, the coefficient penalty contributions differ by a factor of 1000.
Check. Substitution gives the same fitted contribution in both designs. The mismatch is in the penalty, not the predictions. The common mistake is to treat equal predictions as proof that an unstandardized penalty is unchanged.
What Standardization Changes
With predictor standardization, each column is centered and divided by a scale \(s_j\), commonly its sample standard deviation. Write the standardized column as \(z_j=(x_j-\bar{x}_j)/s_j\). With an intercept, centering can be absorbed into the intercept. If \(\theta_j\) is the coefficient of \(z_j\), then the corresponding original-scale slope is \(\beta_j=\theta_j/s_j\), or equivalently \(\theta_j=s_j\beta_j\).
Therefore, a penalty applied to the standardized coefficient translates into a scale-dependent penalty on the original-scale coefficient. For the lasso, it is proportional to \(s_j|\beta_j|\); for ridge, it is proportional to \(s_j^2\beta_j^2\). A one-standard-deviation change in a predictor has the same coefficient scale across columns, but a one-unit change generally does not.
The reason is direct substitution. Since \(\theta_j=s_j\beta_j\), the lasso term \(\lambda\alpha|\theta_j|\) becomes \(\lambda\alpha s_j|\beta_j|\). The ridge term \(\lambda(1-\alpha)\theta_j^2/2\) becomes \(\lambda(1-\alpha)s_j^2\beta_j^2/2\). If a predictor is multiplied by a positive constant and then standardized using the correspondingly multiplied standard deviation, its standardized column is unchanged. The effective penalty for the same standardized effect is therefore unchanged as well.
In glmnet, predictor standardization is on by default. The fit is computed using standardized predictor columns, while the reported coefficients are returned on the original predictor scale. Setting standardize to FALSE means the model instead penalizes the supplied predictor-scale coefficients. Check the actual design matrix, standardization setting, intercept, and any penalty-factor choices before comparing fits. Standardization does not make lambda values universally comparable: the response scale, loss, family, and other objective settings also matter.
Worked Example: Comparing Raw and Standardized Lasso Penalties
Two predictors have standard deviations \(s_1=2\) and \(s_2=10\). Consider original-scale coefficients \(\beta_1=0.5\) and \(\beta_2=0.1\), with \(\lambda=0.4\) and pure lasso. Compare the penalties when the coefficients are penalized directly with the penalties after standardizing both predictors.
Method. First calculate the unstandardized penalty \(\lambda(|\beta_1|+|\beta_2|)\). Then convert each slope to its standardized-predictor coefficient \(\theta_j=s_j\beta_j\) and apply the same lasso formula.
Without standardization, the penalty is \(0.4(0.5+0.1)=0.4(0.6)=0.24\).
After standardization, \(\theta_1=2(0.5)=1\) and \(\theta_2=10(0.1)=1\). The penalty is \(0.4(|1|+|1|)=0.8\). The two standardized coefficients describe the same effect size per predictor standard deviation; the original-scale coefficients do not.
Check. Converting back gives \(\beta_1=\theta_1/2=0.5\) and \(\beta_2=\theta_2/10=0.1\), as specified. The common mistake is to compare the original coefficients alone and overlook that standardization changes the coefficient scale on which the penalty operates.
Standardization Does Not Make Every Representation Equivalent
Rescaling a column and standardizing it are not the same as changing the model’s functional form or its coefficient parameterization. Replacing \(x\) by \(x^2\), for example, changes the shape of the fitted relationship; equalizing the columns’ standard deviations does not turn those two predictors into interchangeable versions of one another. It only puts their columns on a comparable spread scale.
The same caution applies when one underlying feature is represented by multiple columns. Ridge penalizes the sum of squared coefficients, so splitting one fitted contribution across duplicate columns can reduce that sum. Lasso penalizes the sum of absolute coefficients: for exact duplicates, splitting a same-sign contribution leaves that sum unchanged, but the allocation may not be unique. Neither fact means that the two representations have identical model interpretation or selection behavior.
Worked Example: Duplicated Columns Change the Ridge Cost
Suppose a model has two identical, already standardized columns \(z_1=z_2=z\). The desired fitted contribution is \(z\), so the coefficients must satisfy \(\beta_1+\beta_2=1\). Compare the smallest ridge sum of squares with the penalty for representing the same contribution using just one column. Ignore the common ridge multiplier \(\lambda/2\).
Method. Minimize \(\beta_1^2+\beta_2^2\) subject to \(\beta_1+\beta_2=1\). Write \(\beta_2=1-\beta_1\), giving \(g(\beta_1)=\beta_1^2+(1-\beta_1)^2=2\beta_1^2-2\beta_1+1\). Its derivative is \(g'(\beta_1)=4\beta_1-2\), which is zero at \(\beta_1=0.5\). Since \(g''(\beta_1)=4>0\), this is the minimum, with \(\beta_2=0.5\).
The minimum two-column sum of squares is \(0.5^2+0.5^2=0.5\). With one column, the coefficient must be \(1\), and the sum of squares is \(1^2=1\). Thus the duplicated-column representation has half the ridge penalty contribution for the same fitted signal.
For lasso, the two-column contribution is \(|\beta_1|+|\beta_2|\). Under the same constraint, its minimum is \(1\), attained by any nonnegative pair summing to \(1\); the one-column contribution is also \(1\). This checks that duplication changes the ridge cost here, while it does not lower the minimum lasso cost. The common mistake is to infer that standardizing duplicate columns removes their multiplicity effect.
Factor indicators and contrast coding are another reason to inspect the columns rather than just the named variable. Standardizing an indicator column changes its scale according to its prevalence; for a binary indicator with proportion \(p\), its standard deviation is \(\sqrt{p(1-p)}\). A coefficient on a rare indicator and the same numerical coefficient on a common indicator need not incur the same original-scale penalty after standardization. Changing the reference level or contrast coding can also change which coefficients describe the same category comparisons. Standardization does not turn a coefficient-wise penalty into a penalty that is invariant to every recoding.
A Practical Comparison Checklist
Record the units, transformations, indicator coding, and any duplicated or overlapping columns. A predictor name alone does not specify the design matrix.
Check whether columns were centered and scaled, and whether the same preprocessing was used for both fits. If comparing a glmnet fit with another implementation, verify its conventions rather than assuming they match.
For a standardized column, use \(\theta_j=s_j\beta_j\) to compare effects on the standardized scale. For a simple unit change \(x_j^*=cx_j\), use \(\beta_j^*=\beta_j/c\) to preserve predictions.
Check the penalty type, lambda, alpha, loss scaling, response scale, intercept treatment, and penalty factors. Only then interpret a difference in shrinkage.
Common Exam Trap and Practical Implications
A frequent error is to say that two fits have equal penalty strength because they use the same lambda. That statement is incomplete. If one fit uses dollars and the other thousands of dollars without standardization, equivalent predictions can have very different coefficient penalties. Even if both use standardized predictors, a different set of columns or a different coding can change the penalty geometry.
This matters when comparing coefficient magnitudes, tuning results, or a model prepared with a different set of transformations. A small coefficient does not necessarily indicate a small predictor effect: the coefficient’s units matter. Nor does a common lambda guarantee that every predictor receives identical shrinkage on its original scale. For a fair comparison, state the predictor representation and preprocessing alongside the tuning settings.
These distinctions are aligned with the SOA Exam PA (Predictive Analytics) syllabus. In an applied analysis, they also help prevent an accidental change in model behavior when a predictor is converted to different units or encoded differently. Standardization is a useful convention, not a substitute for understanding what the columns represent.
Check Your Understanding
For each problem, compare the equivalent effects and penalty contributions using the stated assumptions.
- A predictor is changed from meters \(x\) to centimeters \(x^*=100x\). An unstandardized lasso coefficient is \(\beta=0.3\) in meters. What equivalent coefficient preserves predictions, and by what factor does its absolute-coefficient penalty change? (Answer: \(\beta^*=0.003\); the unstandardized lasso contribution is \(1/100\) as large.)
- A binary indicator has prevalence \(p=0.25\). Its standard deviation is \(\sqrt{p(1-p)}\). What is the standardized coefficient when its original-scale coefficient is \(2\), and what is its lasso penalty contribution for \(\lambda=0.5\), \(\alpha=1\)? (Answer: \(2\sqrt{0.1875}\approx0.8660\); penalty contribution \(\approx0.4330\).)
- Four identical standardized columns represent a fitted contribution whose total coefficient must be \(1\). What is the smallest ridge sum of squared coefficients, and how does it compare with a one-column representation? (Answer: coefficients \(0.25\) each give \(4(0.25^2)=0.25\), one quarter of the one-column value \(1\).)
- Two unstandardized predictors have standard deviations \(3\) and \(0.5\). For each, consider an original-scale coefficient that gives a one-standard-deviation effect of \(1\). Calculate the coefficients and their ridge squared-coefficient contributions. (Answer: coefficients \(1/3\) and \(2\); contributions \(1/9\) and \(4\), respectively. After standardization, both coefficients are \(1\) and each squared contribution is \(1\).)