Lambda Conventions Across Software Implementations

Why Lambda Values Can Be Hard to Compare

A penalized regression fit balances two quantities: how closely predictions match the observed responses, and how strongly the coefficient vector is penalized. The tuning parameter \(\lambda\) controls that balance. But software packages do not all write the data-fit part of the objective in the same scale. One may use a mean loss, while another uses a sum. As a result, the same numerical value of \(\lambda\) can represent different amounts of penalization.

The key is to compare the whole objective, not just the number printed as lambda. If the loss is multiplied by a constant, the penalty must be multiplied by the same constant to preserve the minimizer. This tutorial develops that relationship for ridge and lasso, works through conversions, and explains why sample size can change the numerical value needed for an equivalent fit.

Two Common Scales for the Objective

Let \(y_i\) be the response and let \(\mathbf{x}_i\) be the predictor vector for observation \(i\). Write the fitted value as \(\beta_0+\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}\), where the intercept \(\beta_0\) is not penalized. Define the residual sum of squares by

For clarity, suppose two implementations use the same observations, predictors, response scale, intercept treatment, and coefficient scaling. They differ only in whether the loss is normalized by sample size. A common normalized form is

and a common unnormalized form is

Here \(P(\boldsymbol{\beta})\) is the penalty, and the subscripts distinguish the two lambda conventions. For lasso, a common choice is \(P(\boldsymbol{\beta})=\sum_j|\beta_j|\). For ridge, a common choice is \(P(\boldsymbol{\beta})=\frac{1}{2}\sum_j\beta_j^2\). The factor of one-half in the ridge penalty is a convention; some objectives omit it. What matters for a conversion is using the same penalty definition on both sides.

Deriving the Conversion

The normalized objective can be multiplied by \(n\) without changing which coefficient values minimize it, because \(n\) is positive. Multiplying every term by \(n\) gives

This is the unnormalized objective when \(\lambda_{\mathrm{sum}}=n\lambda_{\mathrm{norm}}\). The two objectives therefore have exactly the same minimizers under the stated conditions. This is an algebraic equivalence, not an approximation.

The conversion also illustrates why the word “lambda” alone does not specify a universal numerical scale. More generally, suppose two objectives are \(c_A\mathrm{RSS}+\lambda_A P\) and \(c_B\mathrm{RSS}+\lambda_B P\), with the same penalty. Dividing each objective by its loss coefficient shows that they have the same relative penalty strength when

Thus the sample-size conversion applies specifically to the displayed pair of loss scales. If an implementation uses \(\mathrm{RSS}\) instead of \(\mathrm{RSS}/2\), or uses a penalty with a different constant factor, the conversion changes accordingly.

Check the Loss and Penalty Factors

A reliable conversion starts by writing down both objectives, including every constant. For example, \(\mathrm{RSS}/n+\lambda P\) and \(\mathrm{RSS}/(2n)+\lambda P\) do not use the same relative scale for the penalty at the same numerical lambda. The loss coefficient in the first objective is twice as large, so its lambda must also be twice as large to match the second objective.

This rule is about relative scaling, not a special property of lasso or ridge. It works whenever the same penalty function is used in both objectives. However, if one implementation defines the ridge penalty as \(\sum_j\beta_j^2\) and another as \(\frac{1}{2}\sum_j\beta_j^2\), their penalty functions themselves differ by a factor of two. That difference must be included along with any change in loss scale.

Why Sample Size Changes the Numerical Lambda

The factor of \(n\) has a direct interpretation. A sum loss grows with the number of observations, while a mean loss divides that sum by \(n\). If the penalty is left unchanged as the loss is rescaled, their relative influence changes. Consequently, holding a numerical lambda fixed under an unnormalized sum-loss convention does not generally have the same meaning as holding it fixed under a normalized mean-loss convention when sample size changes.

A useful exact illustration is to replicate every observation the same number of times. The replicated data have a larger sample size and a proportionally larger RSS at every fixed coefficient vector. Their mean squared-error contribution is unchanged, while their sum-loss contribution is multiplied by the replication count. Thus, under the normalized convention, the same lambda yields the same objective and fit; under the unnormalized convention, lambda must increase by the replication count to preserve the same relative penalty.

What Must Match Before Comparing Lambda

The algebraic conversion isolates loss normalization, but software implementations can differ in other ways as well. A conversion based only on \(n\) is valid only when the rest of the objective is aligned. Before treating two lambda values as equivalent, check:

• Loss definition: Is the loss RSS, RSS divided by \(n\), or RSS divided by \(2n\)?
• Penalty definition: Is there a factor such as one-half in the ridge penalty, and are the same coefficients included?
• Intercept treatment: Is the intercept excluded from the penalty in both fits?
• Predictor and response scales: Were the predictors and response transformed in the same way before fitting?
• Data and weights: Are the same observations used, with the same observation weights and missing-data handling?

If one of these differs, the relationship \(\lambda_{\mathrm{sum}}=n\lambda_{\mathrm{norm}}\) may not apply by itself. For example, standardizing predictors changes the units in which coefficients—and therefore their penalties—are expressed. Matching the loss scale cannot undo a difference in predictor scaling.

Lambda values are therefore best compared only after the objective has been written in full. When objectives are genuinely equivalent, the conversion can be calculated exactly. When important details are unknown, matching the printed numbers is not evidence of matching fits; compare the definitions and, where appropriate, the resulting coefficients or predictions.

Common Mistake

Wrong reasoning: “Both packages report lambda \(=0.2\), so they apply the same penalty.” This ignores the loss scale. If one package uses \(\mathrm{RSS}/(2n)+0.2P\) and the other uses \(\mathrm{RSS}/2+0.2P\), the objectives generally do not have the same minimizer. With the stated conventions, the equivalent unnormalized lambda is \(0.2n\), not \(0.2\).

Correction: Write both objective functions, identify the loss coefficient and penalty definition in each, and solve for the lambda values that make the objectives proportional. State the sample size and other conditions used in the conversion. A lambda value has meaning through its place in a complete objective, not on its own.