How Response Units Affect Ridge Penalties

Why Response Units Matter

A fitted response might be recorded in dollars, thousands of dollars, or another unit. A change of unit changes the numerical values in the response column, so the fitted coefficients must be interpreted in that new unit too. The question is whether the ridge penalty parameter must also change.

The answer depends on the exact objective being minimized. For the conventional ridge objective with squared-error loss and a squared-coefficient penalty, multiplying every response by the same nonzero constant multiplies the fitted coefficients by that constant, but leaves the numerical value of lambda unchanged. This result assumes that the predictors, their preprocessing, and the objective convention are held fixed.

Definition: For $n$ observations, let $y_i$ be the response and let $x_i$ be a vector of $p$ predictors. Ridge regression with an unpenalized intercept minimizes $$ Q_{\lambda,y}(\beta_0,\boldsymbol{\beta}) =\sum_{i=1}^{n}(y_i-\beta_0-x_i^\mathsf{T}\boldsymbol{\beta})^2 +\lambda\sum_{j=1}^{p}\beta_j^2, \qquad \lambda\geq 0. $$ Here $\beta_0$ is the intercept, $\boldsymbol{\beta}=(\beta_1,\ldots,\beta_p)^\mathsf{T}$ is the slope vector, and $\lambda$ is the ridge penalty parameter. The intercept is not included in the penalty.

This objective specifies the convention used below: the loss is a sum of squared errors, not an average, and the penalty is the sum of squared slopes. Some software or texts divide the squared-error term by $n$, so the numerical lambda values from different conventions need not match. That distinction is about objective convention, not response units.

Assume the response is transformed to $y_i^*=c y_i$, where $c\ne 0$, while the predictor values and penalty convention stay fixed. The transformed coefficients are denoted $\beta_0^*$ and $\boldsymbol{\beta}^*$. No distributional assumption about the errors is needed for the scaling result: it follows from the optimization objective itself.

How the Minimizers Transform

For any candidate coefficient pair $(\beta_0,\boldsymbol{\beta})$, consider the corresponding scaled pair $(c\beta_0,c\boldsymbol{\beta})$ for the transformed response. Each residual is multiplied by $c$, so its square is multiplied by $c^2$. Each slope coefficient is also multiplied by $c$, so the penalty is multiplied by $c^2$. Therefore,

Q_{\lambda,cy}(c\beta_0,c\boldsymbol{\beta}) =c^2 Q_{\lambda,y}(\beta_0,\boldsymbol{\beta}).

Because $c^2>0$, multiplying the objective by $c^2$ preserves which candidates minimize it. More precisely, the transformation of coefficients is a one-to-one mapping when $c\ne 0$: every coefficient pair for the original response corresponds to exactly one scaled pair, and every pair for the transformed response has an unscaled counterpart. Thus the transformation maps the set of minimizers for $y$ onto the set of minimizers for $cy$.

Theorem: Under the stated ridge objective, with predictors and objective convention unchanged, scaling the response by $c\ne 0$ scales the full set of minimizing coefficient pairs by $c$, without changing $\lambda$. If the minimizer is unique, then $\beta_0^*=c\beta_0$ and $\boldsymbol{\beta}^*=c\boldsymbol{\beta}$. If it is not unique, the set-mapping statement remains valid, but a particular selected minimizer follows this rule only if the fitting rule selects corresponding minimizers.

The uniqueness qualification matters, especially when $\lambda=0$ or the design matrix is rank deficient. A scaling argument shows how minimizers correspond; by itself, it does not establish that there is only one minimizer. For $\lambda>0$, the slope penalty makes the slope solution unique for a fixed intercept, and with an intercept and observations the resulting fitted ridge solution is unique. At $\lambda=0$, uniqueness requires sufficient rank in the design.

A One-Predictor Formula and Its Justification

A simple formula makes the scaling visible. Suppose there is one predictor $x_i$, an intercept is included but unpenalized, and the predictor is centered so that $\sum_i x_i=0$. For any fixed slope $\beta$, minimizing over the intercept gives $\hat{\beta}_0=\bar{y}$, where $\bar{y}$ is the sample mean of the response. After centering the response, the slope-dependent objective is

\sum_{i=1}^{n}\bigl((y_i-\bar{y})-x_i\beta\bigr)^2+\lambda\beta^2.

Differentiate with respect to $\beta$, set the derivative equal to zero, and solve. Since $\sum_i x_i=0$, the result is

-2\sum_{i=1}^{n}x_i(y_i-\bar{y}) +2\beta\sum_{i=1}^{n}x_i^2+2\lambda\beta=0, \qquad \hat{\beta} =\frac{\sum_{i=1}^{n}x_i(y_i-\bar{y})} {\sum_{i=1}^{n}x_i^2+\lambda}.

Under response scaling, the numerator and the centered intercept both scale by $c$, while the denominator is unchanged. Hence the slope scales by $c$, with the same $\lambda$. The denominator is positive when $\lambda>0$, so this one-predictor slope is uniquely determined in that case.

Worked Example: Scaling a Small Response

Suppose three observations have centered predictor values $x=(-1,0,1)$ and responses $y=(0,2,4)$. Fit ridge regression with an intercept and $\lambda=2$, then express the response in units that are 1,000 times as large. Use the sum-of-squares objective defined above.

Method. The predictor is centered, so the intercept is the response mean. The response mean is $\bar{y}=(0+2+4)/3=2$. The centered cross-product is $\sum_i x_i(y_i-\bar{y})=(-1)(-2)+0(0)+(1)(2)=4$, and $\sum_i x_i^2=1+0+1=2$.

The ridge slope is therefore $\hat{\beta}=4/(2+2)=1$, and the intercept is $\hat{\beta}_0=2$. The fitted values are $1,2,3$. The residuals are $-1,0,1$, so the squared-error sum is $1+0+1=2$. The penalty is $2(1^2)=2$, giving objective value $2+2=4$.

Now multiply each response by $c=1000$, giving $y^*=(0,2000,4000)$. The mean is $2000$; the centered cross-product is $4000$; and the predictor sum of squares remains $2$. With the same $\lambda=2$, the slope is $4000/(2+2)=1000$, and the intercept is $2000$. The fitted values are $1000,2000,3000$.

The new residuals are $-1000,0,1000$, so their squared sum is $2{,}000{,}000$. The penalty is $2(1000^2)=2{,}000{,}000$, giving objective value $4{,}000{,}000=1000^2(4)$, as required. Check: the fitted values and residuals are each exactly 1,000 times their original values. The common mistake avoided here is changing lambda just because the response numbers changed.

Worked Unit Conversion

A response recorded in dollars and the same response recorded in thousands of dollars differ by $c=0.001$. If predictors are unchanged, predictions and all fitted coefficients—including the intercept—must convert by that same factor. The penalty value does not need a unit-conversion adjustment under this ridge objective.

Worked Example: Converting Dollar Predictions to Thousands

A hypothetical model uses a centered predictor with values $x=(-2,0,2)$. The responses are $y=(1000,4000,7000)$ dollars, and the ridge penalty is $\lambda=4$. Find the fitted coefficients in dollars and in thousands of dollars, and check the objective scaling.

Method. The predictor values sum to zero, so the intercept is the response mean: $\bar{y}=(1000+4000+7000)/3=4000$. The predictor sum of squares is $(-2)^2+0^2+2^2=8$. The centered cross-product is $(-2)(1000-4000)+0(0)+(2)(7000-4000)=6000+6000=12000$.

Thus the slope in dollars per predictor unit is $\hat{\beta}=12000/(8+4)=1000$, and the intercept is $4000$ dollars. The fitted values at $x=-2,0,2$ are $2000,4000,6000$ dollars. The residuals are $-1000,0,1000$, whose squared sum is $2{,}000{,}000$. The penalty is $4(1000^2)=4{,}000{,}000$, so the objective is $6{,}000{,}000$.

In thousands of dollars, the response values are $y^*=(1,4,7)$, so $c=0.001$. The mean is $4$, the centered cross-product is $(-2)(1-4)+(2)(7-4)=6+6=12$, and the predictor sum of squares remains $8$. With the same $\lambda=4$, the slope is $12/(8+4)=1$, and the intercept is $4$. The fitted values are $2,4,6$ thousand dollars.

The residuals in thousands are $-1,0,1$, giving squared-error sum $2$; the penalty is $4(1^2)=4$; and the objective is $6$. This equals $0.001^2(6{,}000{,}000)=6$. Check: the converted coefficients and fitted values are exactly $0.001$ times their dollar versions. The lambda value remains $4$.

Lambda and Objective Conventions

Some ridge objectives use mean squared error rather than the sum of squared errors. With the penalty written in the same way, these objectives are

Q_{\text{sum}}=\mathrm{RSS}+\lambda_{\text{sum}}\|\boldsymbol{\beta}\|_2^2, \qquad Q_{\text{mean}}=\frac{\mathrm{RSS}}{n} +\lambda_{\text{mean}}\|\boldsymbol{\beta}\|_2^2.

Multiplying the second objective by $n$ gives $nQ_{\text{mean}}=\mathrm{RSS}+n\lambda_{\text{mean}}\|\boldsymbol{\beta}\|_2^2$. Since multiplying an objective by a positive constant does not change its minimizers, the equivalent penalty values satisfy $\lambda_{\text{sum}}=n\lambda_{\text{mean}}$. This conversion is about the scaling of the loss term, not the units of the response. Under either convention, multiplying the response by a nonzero constant still multiplies the loss and squared-coefficient penalty by the same factor.

Worked Example: Translating Between Sum and Mean Loss

Use $x=(-1,0,1)$, $y=(2,0,4)$, and $n=3$. Compare the sum-of-squares objective with $\lambda_{\text{sum}}=6$ to the mean-loss objective with $\lambda_{\text{mean}}=2$. Then multiply the response by 1,000.

Method. The response mean is $(2+0+4)/3=2$. The centered cross-product is $(-1)(2-2)+(1)(4-2)=2$, and the predictor sum of squares is $2$. For the sum objective, the slope is $2/(2+6)=0.25$. For the mean objective, the slope formula has numerator and predictor sum of squares divided by $n$, so it is $(2/3)/(2/3+2)= (2/3)/(8/3)=0.25$. The intercept is $2$ under both conventions.

The equivalence also follows from $n\lambda_{\text{mean}}=3(2)=6=\lambda_{\text{sum}}$. The fitted values are $1.75,2,2.25$; the residuals are $0.25,-2,1.75$; and the squared-error sum is $0.25^2+(-2)^2+1.75^2=0.0625+4+3.0625=7.125$. The sum-objective penalty is $6(0.25^2)=0.375$, for total $7.5$. The mean objective is $7.125/3+2(0.25^2)=2.375+0.125=2.5$, exactly one-third of $7.5$.

After multiplying the response by 1,000, the intercept is $2000$ and the slope is $250$ under either convention; the respective lambdas remain $6$ and $2$. The sum of squared errors is multiplied by $1{,}000{,}000$, and the squared-slope penalty is also multiplied by $1{,}000{,}000$. Check: the sum and mean objectives remain in the same ratio of $3$, and their fitted values differ from the original fitted values by a factor of $1{,}000$.

Common Mistakes and Practical Checks

When a fitted response is converted between units, the coefficient values are not directly comparable unless they are expressed in the same response unit. For example, a slope of $1000$ dollars per predictor unit and a slope of $1$ thousand dollars per predictor unit describe the same fitted relationship.

A frequent incorrect argument is that lambda must be multiplied by $c^2$ because the squared-error term is multiplied by $c^2$. The ridge penalty is also multiplied by $c^2$ when the slopes scale by $c$, so the ratio between the two terms is already preserved. Changing lambda by $c^2$ would generally change the amount of shrinkage rather than preserve the fit in converted units.

For the dollar example above, converting to thousands gives cross-product $12$ and predictor sum of squares $8$. The correct slope with $\lambda=4$ is $12/(8+4)=1$. The incorrect adjustment $\lambda^*=0.001^2(4)=0.000004$ would instead give $12/(8+0.000004)$, approximately $1.5000$, not $1$. This does not reproduce the original ridge fit in thousands.

Scale coefficients, not predictors. A response conversion multiplies the intercept and all slopes by the response conversion factor.
Keep lambda fixed for response-unit changes. This conclusion assumes the same predictors and the same objective convention.
Check whether the loss is a sum or a mean. Those conventions use penalty values that differ by a factor of $n$ when they are to produce the same fit.
Do not claim uniqueness without conditions. The scaling relation applies to the full set of minimizers, even when an individual minimizer is not unique.

Before comparing ridge fits, state the response unit, predictor preprocessing, penalty convention, and lambda. This prevents a change in displayed coefficient size from being mistaken for a change in the underlying fitted relationship. Clear reporting of these choices is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

Work each problem using the ridge objective and assumptions stated in this tutorial.

A ridge fit has a unique intercept of $5$, slopes $(2,-1)$, and $\lambda=3$. Every response is converted by $c=0.01$, with predictors unchanged. What are the transformed coefficients and lambda? (Answer: intercept $0.05$, slopes $(0.02,-0.01)$, and $\lambda=3$.)
A one-predictor centered design has $\sum_i x_i(y_i-\bar y)=15$, $\sum_i x_i^2=5$, and $\lambda=5$. Find the ridge slope, then find it when the response is multiplied by $-2$. (Answer: original slope $15/(5+5)=1.5$; transformed slope $-3$.)
An objective uses $\mathrm{RSS}/n+\lambda_{\text{mean}}\|\boldsymbol{\beta}\|_2^2$ with $n=8$ and $\lambda_{\text{mean}}=0.75$. What value in the sum-of-squares convention gives the same minimizers? Does converting the response from dollars to cents alter either equivalent value? (Answer: $\lambda_{\text{sum}}=8(0.75)=6$; the response conversion does not alter either value.)
For a centered one-predictor model, $\sum_i x_i(y_i-\bar y)=24$, $\sum_i x_i^2=4$, and $\lambda=8$. A colleague converts the response to one-tenth of its original values and also changes lambda to $0.1^2(8)$. Find the correct slope in the converted units and the colleague's slope. (Answer: correct slope $2.4$; colleague's slope $2.4/(4+0.08)\approx0.5882$.)
With $\lambda=0$ and a rank-deficient design, a dataset has more than one least-squares minimizer. What can be concluded after multiplying the response by a nonzero constant $c$, and what cannot be concluded without an additional fitting rule? (Answer: scaling by $c$ maps the full original minimizer set onto the transformed minimizer set; one cannot assert that every fitting rule selects the particular scaled version of its original selected minimizer.)

What You'll Learn