How Sample Size Scaling Changes Penalty Parameters

Why the Same Penalty Number Can Mean Different Things

A penalty parameter is meaningful only in relation to the scale of the loss it penalizes. One implementation may minimize the sum of the observations’ losses, while another minimizes their average. These objectives can produce exactly the same fit, but only when their penalty parameters are rescaled consistently. If the same numerical penalty is used in both, the relative influence of the penalty is generally different.

This matters when comparing results from software packages, implementations, or data sets of different sizes. A reported value such as \(\lambda=0.1\) does not, on its own, identify how strongly coefficients were penalized. The objective function—including its loss normalization and penalty definition—must also be known.

Two Objective Conventions

Let \(y_i\) be the response for observation \(i\), \(x_i\) its predictor vector, and \(\beta\) the coefficient vector. Write \(\ell_i(\beta)\) for the loss contributed by observation \(i\). For example, squared-error regression may use \(\ell_i(\beta)=\tfrac12(y_i-x_i^\mathsf{T}\beta)^2\), while logistic regression can use the negative log-likelihood contribution for observation \(i\). Let \(P(\beta)\) be a penalty, such as the lasso penalty \(\|\beta\|_1=\sum_j|\beta_j|\). In the formulations below, the intercept, if included, is not penalized.

The factor \(1/n\) changes the scale of the loss but not the definition of \(P\). Thus, to compare these objectives, we need to account for that factor in the penalty too. The relationship is an algebraic one; it does not require a probability model or an assumption that the observations are independent. Statistical assumptions may be needed to justify a particular loss for a scientific problem, but they are not needed for the conversion itself.

Deriving the Conversion

Divide the entire summed-loss objective by \(n\). Multiplication of an objective by a positive constant does not change which \(\beta\) minimizes it. After division, the loss is averaged, and the penalty coefficient is also divided by \(n\):

For this expression to equal the averaged-loss objective, the penalty coefficients must satisfy the following relation.

This result applies to any penalty function \(P\), not just the lasso. For example, an elastic-net penalty can be written as \(P(\beta)=\alpha\|\beta\|_1+(1-\alpha)\|\beta\|_2^2/2\), where \(\alpha\) sets the mixture of the two penalty components. If \(\alpha\) is unchanged, the whole penalty coefficient scales by \(n\); the mixture itself does not change.

What Happens If the Number Is Not Rescaled?

Using the same numerical \(\lambda\) with both conventions does not generally preserve the fit. The summed loss grows with the number of observations, while the averaged loss is on a per-observation scale. Consequently, a fixed penalty in a summed objective becomes relatively less influential as \(n\) increases. In an averaged objective, a fixed penalty remains on the same scale relative to the average loss, provided the distribution of per-observation losses and the predictor scaling stay comparable.

The example is deliberately simple, but the distinction is general. For fixed data, two correctly converted objectives have the same minimizer. Two objectives using the same numerical penalty but different loss normalization generally do not. A numerical penalty value therefore cannot be interpreted independently of the convention used to define the loss.

Applying the Rule to Different Penalties and Losses

The conversion is not specific to squared-error regression. Suppose a penalized logistic-regression objective uses the sum of the observations’ negative log-likelihood contributions and an \(\ell_1\) penalty. Dividing the full objective by \(n\) again divides the penalty coefficient by \(n\). The same reasoning applies to ridge and elastic net, because the derivation requires only that the same penalty function be used in both formulations.

Some objective functions include an additional constant in their loss definition. For instance, squared-error loss is often written with a factor \(1/2\), and likelihood software may report negative log likelihood or twice the negative log likelihood. A constant multiplying the loss must also be accounted for when translating the penalty parameter. If one objective is \(c\sum_i\ell_i(\beta)+\lambda P(\beta)\), dividing by \(c\) gives \(\sum_i\ell_i(\beta)+(\lambda/c)P(\beta)\). It is therefore unsafe to compare penalties by sample size alone if the loss definitions also differ.

Likewise, “average loss” might mean an average over rows, an average over total observation weights, or an average over binomial trials. These denominators need not be the same. If the loss is weighted, write down the actual objective and its denominator before converting the penalty. The relevant scale is the multiplier applied to the loss actually being optimized, not a presumed sample size.

Checking and Reporting the Convention

When comparing penalty values, first identify the exact objective. Check whether the loss is summed or averaged, whether it includes a constant such as \(1/2\) or \(2\), and how the penalty is defined. Also check whether the intercept is penalized and whether predictors or responses have been rescaled. The sum-to-average conversion assumes these choices are otherwise the same.

A useful check is to express both objectives on a common scale. For example, divide the summed objective by \(n\), then compare its penalty coefficient and penalty function with those in the averaged objective. If they match, the objectives have the same minimizers. When reproducing a reported result, record the objective formula alongside \(\lambda\), the number of loss contributions, and any preprocessing or weighting that changes the scale.

A common mistake is to say, “Both programs report \(\lambda=0.1\), so they applied the same regularization.” That conclusion is not justified if one program averages the loss and the other sums it. For \(n=100\), an averaged-loss parameter of \(0.1\) corresponds to a summed-loss parameter of \(10\), not \(0.1\). The correction is to compare the complete objectives after rescaling, rather than comparing the printed numbers alone.

A second, related mistake is to assume that a penalty value can be compared across data sets without accounting for sample size. Under a summed-loss convention, keeping \(\lambda_{\mathrm{sum}}\) fixed while increasing \(n\) changes the penalty relative to the loss. Under an averaged-loss convention, the parameter is expressed relative to average loss, but comparisons can still be affected by changes in predictor scaling, response scaling, weights, and loss definition.

Limitations of the Conversion

The identity \(\lambda_{\mathrm{sum}}=n\lambda_{\mathrm{avg}}\) is exact only when the objectives differ by the loss normalization described above. It does not imply that equal converted penalty values across separate studies yield the same fitted coefficients: their observations, predictors, response scales, and fitted models may differ. Nor does it make penalty values universal across software packages that standardize variables differently or use distinct loss and penalty conventions.

For a fixed data set and a fixed penalty function, the conversion preserves the entire optimization problem’s minimizers, including any nonuniqueness. It does not resolve whether the optimizer has converged, whether the modeling assumptions are appropriate, or whether a particular penalty is scientifically useful. Those questions require examining the fit and the analysis context separately.