How Lambda Controls Shrinkage

Introduction

In elastic net regression, lambda controls the strength of the penalty. Holding alpha fixed isolates the effect of changing that strength: the relative mix of the L1 and L2 penalties stays the same, while their combined influence on the fitted coefficients changes. Larger lambda generally means a more constrained fit, but it does not justify every intuitive claim about individual coefficients or the number of selected predictors.

The key distinction is between results that hold for any design matrix and results that require special structure. The training residual sum of squares cannot decrease as lambda increases along globally optimal fits. In an orthogonal design, each coefficient’s absolute value also decreases, until it reaches zero when alpha is positive. With correlated predictors, however, an individual coefficient can increase in magnitude over part of the path.

Objective and the Role of Lambda

Use a fixed design matrix \(X\), response vector \(y\), and coefficient vector \(\beta\). Take the intercept to be unpenalized, and absorb it by centering \(y\) and the predictor columns. For a fixed \(\alpha\) between 0 and 1 and a tuning parameter \(\lambda\geq 0\), the elastic net estimate minimizes the following objective:

Here, \(\|y-X\beta\|_2^2\) is the residual sum of squares (RSS), \(\|\beta\|_1=\sum_j|\beta_j|\), and \(\|\beta\|_2^2=\sum_j\beta_j^2\). At fixed alpha, increasing lambda multiplies the same penalty by a larger amount; it does not change the relative weights of the L1 and L2 parts. The case \(\alpha=0\) is ridge regression, while \(\alpha>0\) includes an L1 penalty and can produce exact zero coefficients.

Unless a special design condition is stated, claims below refer to global minimizers of this objective for the same data and the same alpha. When the minimizer is not unique, the objective-value comparisons still apply to any chosen minimizers.

Orthogonal Design: An Explicit Shrinkage Path

A useful case for seeing how lambda acts is an orthogonal design with centered, scaled predictors satisfying \(X^\mathsf{T}X/n=I\). Define \(z_j=X_j^\mathsf{T}y/n\), where \(X_j\) is column \(j\). The objective separates into one-variable problems, and the solution for coefficient \(j\) is:

When \(|z_j|\leq\lambda\alpha\), the numerator is zero, so the coefficient is zero. Otherwise it keeps the sign of \(z_j\). If \(\alpha>0\), the zero threshold is \(\lambda\geq |z_j|/\alpha\). If \(\alpha=0\), there is no L1 threshold: the ridge coefficient approaches zero as lambda grows but is not set exactly to zero by this formula unless \(z_j=0\).

For a nonzero orthogonal-design coefficient, its magnitude is \[ |\hat\beta_j(\lambda)| = \frac{|z_j|-\lambda\alpha}{1+\lambda(1-\alpha)} \] on the range \(|z_j|>\lambda\alpha\). Differentiating gives:

The derivative applies only while the coefficient is nonzero; after it reaches zero, its magnitude remains zero. It establishes monotone shrinkage in this orthogonal setting, not for every design matrix.

What Is Guaranteed for Any Design?

The orthogonal formula is not needed to establish two important general results. Let \(\lambda_1<\lambda_2\), and let \(\hat\beta_1\) and \(\hat\beta_2\) minimize the objective at those respective values of lambda, with alpha and the data fixed. Write \(R_k=R(\hat\beta_k)\) and \(P_k=P_\alpha(\hat\beta_k)\). Optimality at each value gives:

Adding these inequalities yields \((\lambda_2-\lambda_1)(P_2-P_1)\leq0\), so \(P_2\leq P_1\): the value of the combined penalty at the fitted solution cannot increase. The first inequality also gives \(R_1-R_2\leq\lambda_1(P_2-P_1)\leq0\), so \(R_1\leq R_2\). Thus, as lambda increases, the optimized training RSS cannot decrease. This proof compares optimal objective values; it does not require orthogonal predictors.

The penalty is a weighted combination, so its decrease alone does not establish that both \(\|\hat\beta\|_1\) and \(\|\hat\beta\|_2^2\) must separately decrease for every design. Nor does a larger training RSS by itself show that predictive performance on new data worsens; validation data are needed to assess that question.

Lambda, Model Complexity, and Exam Traps

A larger lambda usually produces a more regularized fit: coefficients are more constrained, and fitted values tend to be less responsive to the observed data. For \(\alpha>0\), exact zeros are possible, so the number of nonzero coefficients is one practical measure of model complexity. But correlated predictors can enter or leave along a path, so this count need not fall at every increase in lambda. When \(\alpha=0\), ridge generally does not set coefficients exactly to zero; counting nonzero coefficients then does not capture its continuous shrinkage.

At sufficiently large lambda, the zero vector is optimal when \(\alpha>0\): at \(\beta=0\), the coordinatewise optimality conditions require each absolute component of \(X^\mathsf{T}y/n\) to be at most \(\lambda\alpha\). If \(\alpha=0\), the L2 penalty instead drives coefficients toward zero as lambda grows, without generally making them exactly zero at a finite value. These statements assume the centered, unpenalized-intercept setup used in the objective above.

A frequent exam error is to take the right conclusion from a derivative but use an incorrect derivative. In the orthogonal case, for \(z_j>0\) while its coefficient is nonzero, \[ \hat\beta_j(\lambda)=\frac{z_j-\lambda\alpha}{1+\lambda(1-\alpha)}. \] The quotient rule gives:

For instance, if \(z_j=4\), \(\alpha=0.5\), and \(\lambda=0.5\), then \(\hat\beta_j=(4-0.25)/1.25=3\). Its derivative is \(-[0.5+0.5(4)]/1.25^2=-2.5/1.5625=-1.6\). The tempting expression \(-z_j/[1+\lambda(1-\alpha)]^2\) would give \(-4/1.5625=-2.56\), which is wrong. The numerator derivative must account for both the changing threshold and the changing denominator.