Why Extreme Penalties Matter
A penalty parameter controls how strongly a regression fit is discouraged from using large slopes. Its behavior near zero and at very large values gives useful boundary checks: a vanishing penalty should recover a least-squares fit, while a sufficiently strong lasso penalty makes every slope exactly zero. The limits are not identical for ridge and lasso, and rank-deficient predictors require particular care.
Throughout, use the objective normalization below. Changing the scale of the loss changes the numerical value of the penalty threshold, so the objective must be identified before calculating an extreme-penalty result. The intercept is not penalized.
These are deterministic least-squares objectives; no probability distribution for the errors is needed to establish the limits. The results also do not require predictors to be standardized, but their scales affect the coefficient penalties and therefore the numerical paths and thresholds.
The Limits as the Penalty Vanishes
At \(\lambda=0\), both objectives reduce to least squares. If the least-squares slope vector is unique—for example, if \(X\) has full column rank—both ridge and lasso approach that vector as \(\lambda\) decreases to zero. When the least-squares solution is not unique, the two penalties select according to different measures of coefficient size.
A short comparison argument explains the selection rule. Write \(L(\beta)=\|y_c-X\beta\|^2/(2n)\), let \(L_*\) be its minimum, and let \(P\) be the penalty norm. For a penalized minimizer \(\hat\beta_\lambda\) and any least-squares minimizer \(\beta_*\), optimality gives
Because \(L(\hat\beta_\lambda)\geq L_*\), this implies \(P(\hat\beta_\lambda)\leq P(\beta_*)\). It also gives \(L(\hat\beta_\lambda)\leq L_*+\lambda P(\beta_*)\), so the loss approaches its least-squares minimum as \(\lambda\to0^+\). The coefficients are bounded by the penalty comparison; any convergent subsequence therefore has a least-squares limit whose penalty is no larger than that of any other least-squares solution. Taking \(P=\|\cdot\|_2^2\) for ridge or \(P=\|\cdot\|_1\) for lasso yields the respective minimum-norm rule.
Worked Example: Ridge at Small and Large Penalties
A hypothetical centered regression design has \(G=\begin{pmatrix}4&0\\0&1\end{pmatrix}\), \(c=(8,2)^\mathsf{T}\), and response mean \(\bar y=5\). Find the ridge coefficients at \(\lambda=0.5\), their zero-penalty limit, and their behavior as \(\lambda\) becomes large.
Method. Differentiating the ridge objective with respect to \(\beta\) gives the normal equation \((G+\lambda I)\hat\beta=c\). Since \(G\) is diagonal here, solve each coordinate separately:
At \(\lambda=0.5\), \(\hat\beta_1=8/4.5=16/9\) and \(\hat\beta_2=2/1.5=4/3\). Thus \(\hat\beta=(16/9,4/3)^\mathsf{T}\). As \(\lambda\to0^+\), the coefficients approach \((8/4,2/1)^\mathsf{T}=(2,2)^\mathsf{T}\), the least-squares solution. As \(\lambda\to\infty\), both coefficients approach zero; more precisely, \(\lambda\hat\beta\to(8,2)^\mathsf{T}\). The intercept remains \(5\) because the predictors are centered and it is unpenalized.
Check and trap avoided. Substituting the small-penalty coefficients into \((G+0.5I)\hat\beta=c\) gives \((4.5)(16/9)=8\) and \((1.5)(4/3)=2\). A common error is to shrink the intercept along with the slopes; that would incorrectly move the limiting fitted constant away from \(\bar y=5\).
Ridge at Very Large Penalties
For every \(\lambda>0\), the ridge normal equation is \( (G+\lambda I)\hat\beta=c\). The penalty makes this equation invertible even if the predictors are linearly dependent. Large diagonal terms \(\lambda I\) dominate \(G\), so the slopes tend to zero. The fitted values therefore tend to the intercept-only fit, which is \(\bar y\) when the predictors have been centered.
Ridge generally does not set a coefficient exactly to zero at any finite \(\lambda\). An exception occurs if the corresponding predictor has zero sample covariance with the centered response and the other coefficients do not induce a nonzero value through the design; in particular, when \(c=0\), the entire ridge slope vector is zero for every positive \(\lambda\). In general, “shrinks toward zero” is not the same claim as “becomes exactly zero.”
Lasso at Very Large Penalties
Unlike ridge, lasso has a finite threshold beyond which all slopes are exactly zero. This follows from the lasso optimality conditions. At \(\beta=0\), the gradient of the loss is \(-c\), while the subgradient of the \(L_1\) penalty can have each coordinate anywhere in \([-1,1]\). Thus zero is optimal precisely when each coordinate of \(c\) can be balanced by a penalty subgradient:
Consequently, for \(\lambda\geq\lambda_{\max}\), the all-zero slope vector is a lasso solution. If \(G\) is positive definite, the objective is strictly convex and this solution is unique. If \(G\) is singular, there can be other minimizers with the same fitted values and objective, although the zero vector is still a minimizer. The intercept remains \(\bar y\).
Worked Example: Finding the Lasso Zeroing Threshold
For a hypothetical centered design, suppose \(G=\begin{pmatrix}2&1\\1&2\end{pmatrix}\), \(c=(5,1)^\mathsf{T}\), and \(\bar y=12\). Find the zeroing threshold, determine the slopes at \(\lambda=4\), and find their limit as \(\lambda\to0^+\).
Threshold. The largest absolute coordinate of \(c\) is \(\max(5,1)=5\), so \(\lambda_{\max}=5\). For every \(\lambda\geq5\), \(\hat\beta=(0,0)^\mathsf{T}\) is optimal. At \(\lambda=5\), the subgradient condition is satisfied because \(c/5=(1,0.2)^\mathsf{T}\) has both coordinates in \([-1,1]\).
At \(\lambda=4\). Consider \(\hat\beta=(0.5,0)^\mathsf{T}\). The gradient of the loss is \(G\hat\beta-c=(1,0.5)^\mathsf{T}-(5,1)^\mathsf{T}=(-4,-0.5)^\mathsf{T}\). For the first, nonzero coefficient, the \(L_1\) subgradient is \(1\), so the first optimality equation is \(-4+4(1)=0\). For the second, zero coefficient, choose subgradient \(z_2=0.125\); then \(-0.5+4(0.125)=0\), and \(z_2\in[-1,1]\). The lasso conditions hold, so these are the fitted slopes. Since \(G\) is positive definite, the solution is unique.
As \(\lambda\to0^+\). Solve the least-squares equations \(G\beta=c\): \(2\beta_1+\beta_2=5\) and \(\beta_1+2\beta_2=1\). Doubling the second equation gives \(2\beta_1+4\beta_2=2\); subtracting the first gives \(3\beta_2=-3\), so \(\beta_2=-1\), and then \(\beta_1=3\). The zero-penalty limit is \((3,-1)^\mathsf{T}\). The intercept stays \(12\).
Check and trap avoided. At \(\lambda=4\), the inactive-coordinate subgradient must lie in \([-1,1]\); the value \(0.125\) does, verifying the candidate. A frequent error is to use a threshold of \(5/2\) by dividing by the number of predictors. Under the stated objective, the threshold is \(\|X^\mathsf{T}y_c/n\|_\infty=5\); a different loss normalization would require a correspondingly different threshold.
What Rank Deficiency Changes
If predictors are duplicated or linearly dependent, different coefficient vectors can give exactly the same fitted values. The zero-penalty limit then describes a choice among equivalent least-squares coefficients, not necessarily a unique vector. The following example shows why it is important to state which penalty norm selects that limit.
Worked Example: Lasso with Duplicate Predictors
Suppose two centered predictors are identical, with \(x^\mathsf{T}x/n=1\), and the centered response is \(y_c=2x\). Let \(\bar y=7\). Determine the lasso fitted slopes as the penalty approaches zero and for large penalties.
Write \(s=\beta_1+\beta_2\). Because both columns equal \(x\), the fitted centered response is \(sx\), and the loss is
For any fixed \(s\), the smallest possible value of \(|\beta_1|+|\beta_2|\) is \(|s|\), by the triangle inequality. Therefore the best lasso objective at that sum is \(\frac12(2-s)^2+\lambda|s|\). For \(0\leq\lambda<2\), the minimizing sum is positive; differentiating on \(s>0\) gives \(s-2+\lambda=0\), so \(s=2-\lambda\). At \(\lambda\geq2\), the minimum is attained at \(s=0\). Thus the slopes are zero for \(\lambda\geq2\), while as \(\lambda\to0^+\) they approach any nonnegative pair summing to \(2\). For example, \((2,0)\) and \((1,1)\) are both possible minimum-\(L_1\) least-squares solutions. The intercept remains \(7\).
Check and trap avoided. Every nonnegative pair with sum \(2\) has \(L_1\) norm \(2\), so the lasso limiting coefficient vector is not unique here. Claiming that the slopes must converge to \((1,1)\) would impose an unsupported equal-splitting rule. The fitted value, however, is unique.
Boundary Behavior at a Glance
| Penalty and limit | Slopes | Intercept with centered predictors |
|---|---|---|
| Ridge, \(\lambda\to0^+\) | Least-squares solution of minimum Euclidean norm | \(\bar y\) |
| Lasso, \(\lambda\to0^+\) | Least-squares solution of minimum \(L_1\) norm | \(\bar y\) |
| Ridge, \(\lambda\to\infty\) | Approach zero, generally not exactly at finite \(\lambda\) | \(\bar y\) |
| Lasso, \(\lambda\geq\|c\|_\infty\) | Zero is a minimizer; unique when the objective is strictly convex | \(\bar y\) |
These limits are useful checks on calculations, but they do not by themselves identify a suitable penalty for an applied model. In particular, the lasso threshold depends on the chosen loss scaling and the predictor scales. The intercept result also relies on leaving it unpenalized: penalizing it would change the limiting constant.
Check Your Understanding
Use the centered-data objectives and normalization stated above. Each answer identifies the slopes unless the question also asks about the intercept.
- A one-predictor centered design has \(G=3\), \(c=6\), and \(\bar y=4\). Find the ridge slope at \(\lambda=3\), its limit as \(\lambda\to0^+\), and the intercept. (Answer: \(\hat\beta=1\) at \(\lambda=3\); zero-penalty limit \(2\); intercept \(4\).)
- A centered lasso problem has \(c=(-2,4,-1)^\mathsf{T}\) and \(\bar y=9\). What is the smallest penalty at which zero slopes are guaranteed to be optimal, and what are the slopes and intercept at that threshold? (Answer: \(\lambda_{\max}=4\); zero slopes; intercept \(9\).)
- Two identical centered predictors satisfy \(x^\mathsf{T}x/n=2\), and \(y_c=3x\). For lasso at \(\lambda=2\), find the sum of the slopes. State the possible minimum-\(L_1\) coefficient pairs at the zero-penalty limit. (Answer: \(\beta_1+\beta_2=2\); at the limit any nonnegative pair summing to \(3\).)
- A centered ridge problem has \(G=\begin{pmatrix}1&0\\0&4\end{pmatrix}\), \(c=(2,-8)^\mathsf{T}\), and \(\bar y=6\). Find the slopes at \(\lambda=2\), their zero-penalty limit, and the large-penalty limit. (Answer: \((2/3,-4/3)^\mathsf{T}\); zero-penalty limit \((2,-2)^\mathsf{T}\); large-penalty limit \((0,0)^\mathsf{T}\); intercept \(6\).)
- For a lasso objective with centered data, \(c=(3,-5)^\mathsf{T}\). A candidate says the slopes are necessarily uniquely zero at \(\lambda=5\), regardless of the design matrix. Is that conclusion justified? (Answer: Zero is a minimizer at \(\lambda=5\), since \(\|c\|_\infty=5\); uniqueness is not guaranteed without strict convexity or another uniqueness condition.)