Penalized Least Squares and the Lambda Tradeoff

Why the Objective Has Two Parts

Ordinary least squares chooses coefficients to make the observed responses fit the model as closely as possible, as measured by the residual sum of squares. Ridge and lasso regression modify this goal: they also charge a penalty for the sizes of the predictor coefficients. The resulting objective makes explicit the tradeoff between fitting the observed data and keeping slopes small.

This tradeoff is controlled by a nonnegative tuning parameter, \(\lambda\). With a small \(\lambda\), residual fit has more influence; with a large \(\lambda\), the coefficient penalty has more influence. The penalty applies to the slopes, not usually to the intercept. The two methods differ in how they measure the size of those slopes.

The Ridge and Lasso Objectives

Suppose there are \(n\) observations and \(p\) predictors. For observation \(i\), let \(y_i\) be the response and \(x_{ij}\) the value of predictor \(j\). Write the fitted value as \(\hat{y}_i=\beta_0+\sum_{j=1}^{p}\beta_jx_{ij}\). Here \(\beta_0\) is the intercept and \(\beta_1,\ldots,\beta_p\) are the slopes. The residual sum of squares, abbreviated RSS, is the sum of the squared differences between the observed and fitted responses.

Definition: For \(\lambda\geq 0\), the ridge objective is RSS plus \(\lambda\) times the sum of squared slopes. The lasso objective is RSS plus \(\lambda\) times the sum of the absolute values of the slopes. In both objectives, the intercept is not included in the penalty.

Q_{\text{ridge}}(\beta_0,\beta_1,\ldots,\beta_p) = \sum_{i=1}^{n} \left(y_i-\beta_0-\sum_{j=1}^{p}\beta_jx_{ij}\right)^2 + \lambda\sum_{j=1}^{p}\beta_j^2

Q_{\text{lasso}}(\beta_0,\beta_1,\ldots,\beta_p) = \sum_{i=1}^{n} \left(y_i-\beta_0-\sum_{j=1}^{p}\beta_jx_{ij}\right)^2 + \lambda\sum_{j=1}^{p}|\beta_j|

A penalized fit chooses coefficients that minimize the relevant objective. When \(\lambda=0\), the penalty contributes nothing, so the objective reduces to RSS and the procedure is ordinary least squares. When \(\lambda>0\), a coefficient choice with smaller slopes may be preferred even if it has a larger RSS. In these formulas, RSS is not divided by \(n\), and there is no factor of one-half in front of either term. Other conventions are possible, but they change the numerical value corresponding to a given \(\lambda\).

The intercept is excluded because it sets the baseline level of the response, rather than representing an effect of a predictor. Penalizing it would make the fitted baseline depend on the arbitrary choice of response origin. For example, changing a response measurement from a score out of 100 to the same score with 50 points subtracted would change the intercept, even though the relationships represented by the slopes need not change.

Worked Example: Comparing Fit and Penalty

Suppose a hypothetical data set has predictor values \(x=0,1,2\) and response values \(y=2,3,4\). Compare two candidate lines: one with intercept 2 and slope 1, and an intercept-only line with intercept 3 and slope 0. Calculate their ridge and lasso objective values when \(\lambda=0.5\) and when \(\lambda=3\).

For the line with intercept 2 and slope 1, the fitted responses are 2, 3, and 4, so all three residuals are zero and \(\text{RSS}=0\). Its ridge penalty is \(1^2=1\), and its lasso penalty is \(|1|=1\). Thus both objectives equal \(0+\lambda\).

For the intercept-only line with intercept 3, the fitted responses are 3, 3, and 3. The residuals are \(-1,0,1\), so \(\text{RSS}=(-1)^2+0^2+1^2=2\). Its slope is zero, so both penalties are zero. Both objectives equal 2.

At \(\lambda=0.5\), the first line has objective \(0+0.5(1)=0.5\), compared with 2 for the intercept-only line. At \(\lambda=3\), the first line has objective \(0+3(1)=3\), compared with 2 for the intercept-only line. Among these two candidates, the first has the smaller objective at \(\lambda=0.5\), while the intercept-only line has the smaller objective at \(\lambda=3\). Comparing two candidates alone does not establish which coefficients minimize the objective over all possible choices.

What Lambda Changes

Lambda sets the price attached to the penalty relative to the price attached to RSS. It does not directly specify how much the fitted error will change, and the same numerical value need not have the same practical effect in different data sets. The objective makes the direction of the tradeoff precise: as \(\lambda\) increases, the minimum achievable RSS cannot decrease, while the penalty value at a minimizing fit cannot increase.

To see why, let \(R\) denote RSS and \(P\) the relevant penalty, either \(\sum_j\beta_j^2\) or \(\sum_j|\beta_j|\). For two values \(0\leq\lambda_1<\lambda_2\), write \((R_1,P_1)\) and \((R_2,P_2)\) for the RSS and penalty at minimizing coefficient choices. The choice at \(\lambda_1\) is available at \(\lambda_2\), and the choice at \(\lambda_2\) is available at \(\lambda_1\). Therefore, optimality gives both inequalities below.

R_1+\lambda_1P_1\leq R_2+\lambda_1P_2 \qquad\text{and}\qquad R_2+\lambda_2P_2\leq R_1+\lambda_2P_1

Adding the inequalities and cancelling \(R_1+R_2\) gives \((\lambda_2-\lambda_1)(P_2-P_1)\leq 0\). Since \(\lambda_2-\lambda_1>0\), it follows that \(P_2\leq P_1\). Substituting that result into the first inequality gives \(R_1\leq R_2+\lambda_1(P_2-P_1)\leq R_2\), so \(R_1\leq R_2\) as well. Thus a larger penalty weight leads to a fit with no larger penalty and no smaller RSS. This result concerns the total penalty and RSS; it does not say that every individual slope must move monotonically toward zero.

At very large \(\lambda\), the slopes are strongly discouraged. In the limit, the fitted model approaches an intercept-only fit, whose intercept is the sample mean of the responses. This limiting description is not a prescription to choose a large \(\lambda\): it explains the direction of the tradeoff, not which value is suitable for a particular purpose.

Why the Intercept Is Not Penalized

The unpenalized intercept has a useful consequence that can be derived directly. Fix any set of slopes, and define the residual for observation \(i\) as \(r_i=y_i-\beta_0-\sum_j\beta_jx_{ij}\). The penalty does not depend on \(\beta_0\). Therefore, for these fixed slopes, minimizing either objective over \(\beta_0\) is the same as minimizing the sum of squared residuals.

Differentiate that sum with respect to \(\beta_0\), set the derivative equal to zero, and solve. The second derivative is \(2n>0\), so this stationary point is the unique minimum over the intercept.

\frac{\partial}{\partial\beta_0}\sum_{i=1}^{n}r_i^2 =-2\sum_{i=1}^{n}r_i=0 \quad\Longrightarrow\quad \beta_0=\bar{y}-\sum_{j=1}^{p}\beta_j\bar{x}_j

Here \(\bar{y}\) is the sample mean response and \(\bar{x}_j\) is the sample mean of predictor \(j\). This formula says that, for any fixed slopes, the fitted residuals sum to zero when the intercept is chosen to minimize the objective. The penalty affects which slopes are favored, but it does not alter this intercept adjustment.

Worked Example: Finding the Unpenalized Intercept

Suppose a hypothetical data set records \(x=1,2,3\) and \(y=5,7,9\). Hold the slope fixed at 2. Find the intercept that minimizes either penalized objective, then calculate the ridge and lasso objectives at \(\lambda=4\).

The sample means are \(\bar{x}=2\) and \(\bar{y}=7\). The intercept formula gives \(\beta_0=7-2(2)=3\). The fitted responses are \(3+2(1)=5\), \(3+2(2)=7\), and \(3+2(3)=9\), so \(\text{RSS}=0\).

The ridge penalty is \(2^2=4\), giving \(Q_{\text{ridge}}=0+4(4)=16\). The lasso penalty is \(|2|=2\), giving \(Q_{\text{lasso}}=0+4(2)=8\). If the intercept were incorrectly included in the ridge penalty, the added amount would be \(4(3^2)=36\); if it were included in the lasso penalty, the added amount would be \(4|3|=12\). Those extra charges are absent from the stated ridge and lasso objectives.

The Two Penalties Are Not Interchangeable

Ridge and lasso share the same residual-fit term, but their penalties measure coefficient size differently. Ridge squares each slope before summing; lasso sums absolute slope values. Consequently, even when two coefficient choices have the same RSS, their ridge and lasso penalty values can differ. Their objective values can also respond differently as \(\lambda\) changes.

Worked Example: Ridge and Lasso Can Rank Candidates Differently

Consider hypothetical data with two predictors and three observations:

Observation	\(x_1\)	\(x_2\)	\(y\)
1	-1	1	-1
2	0	-2	0
3	1	1	2

Compare candidate A, with \((\beta_0,\beta_1,\beta_2)=(0,2,0)\), and candidate B, with \((\beta_0,\beta_1,\beta_2)=(0,1,1)\). Candidate A predicts \(-2,0,2\); its residuals are \(1,0,0\), so its RSS is \(1\). Candidate B predicts \(0,-2,2\); its residuals are \(-1,2,0\), so its RSS is \(1+4=5\).

For ridge, the slope penalties are \(2^2+0^2=4\) for A and \(1^2+1^2=2\) for B. Hence their objectives are \(1+4\lambda\) and \(5+2\lambda\). At \(\lambda=1\), these are 5 and 7, so A has the smaller objective. At \(\lambda=3\), they are 13 and 11, so B has the smaller objective. They tie when \(1+4\lambda=5+2\lambda\), which gives \(\lambda=2\).

For lasso, both candidates have penalty 2: \(|2|+|0|=2\), and \(|1|+|1|=2\). Their objectives are \(1+2\lambda\) and \(5+2\lambda\), respectively, so A has the smaller objective for every \(\lambda\geq0\) in this two-candidate comparison. These calculations show why the penalties must be specified before comparing fits; they do not, by themselves, identify the minimizing fit among every possible coefficient choice.

Common Mistakes and Practical Meaning

Penalizing the intercept. The stated ridge and lasso penalties apply to \(\beta_1,\ldots,\beta_p\), not \(\beta_0\). Including the intercept changes the model objective.
Assuming more penalty means better fit to the observed data. Increasing \(\lambda\) can reduce the penalty, but the minimum RSS cannot decrease. The point is to balance fit with coefficient size, not to minimize RSS alone.
Comparing lambda values without checking the objective convention. Dividing RSS by \(n\), adding a factor of one-half, or changing the penalty definition changes the scale of \(\lambda\). State the objective before reporting a lambda value.
Assuming every slope must shrink monotonically as lambda rises. The total penalty is nonincreasing at minimizing fits, but individual coefficients can behave differently, particularly when predictors are related.

The units also matter. With unscaled RSS, the squared residual term has units of response squared, while slope units depend on the predictor. Thus a numerical \(\lambda\) is tied to the measurement scales and objective convention; its value is not meaningful in isolation. Predictor scaling is an important preparation for penalization, but the central point here is simply to interpret \(\lambda\) relative to the particular objective being minimized.

Penalization matters because a slightly worse fit to the current observations may be preferable to coefficients that are large and sensitive to the data. The objective gives a transparent mathematical statement of that preference. It does not by itself select \(\lambda\), establish causal effects, or guarantee a particular performance on new observations.

Check Your Understanding

Answer each question by referring to the objective and, where useful, showing the arithmetic.

Write the ridge and lasso penalties for slopes \(\beta_1=3\) and \(\beta_2=-2\). Which penalty is larger before multiplying by \(\lambda\)?
In the stated convention, what does the objective reduce to when \(\lambda=0\)?
For fixed slopes, why does minimizing either objective over the intercept make the residuals sum to zero?
As \(\lambda\) increases, what happens to the minimum RSS and the total penalty at a minimizing fit?
Why can two different objective conventions assign different numerical values to lambda while expressing a similar fit-versus-penalty tradeoff?