Ridge Regression As A Route To Ridgeless Interpolation

Why a Vanishing Penalty Can Select an Interpolator

In an underdetermined linear model, there are more coefficients than observations. When the design matrix has full row rank, any response vector can be fitted exactly, but there are generally many coefficient vectors that do so. Ridge regression makes the optimization problem unique by penalizing coefficient size. The question here is what happens when that penalty is reduced toward zero while the data and design are held fixed.

The answer is not merely that “the penalty disappears.” The ridge estimates converge to a particular exact fit: the interpolating coefficient vector with the smallest Euclidean norm. This conclusion requires a full-row-rank design. If the design is rank deficient, the zero-penalty limit still has a precise least-squares interpretation, but it need not interpolate the observed responses.

Setup and Definitions

Let \(X\in\mathbb{R}^{n\times p}\) be a fixed design matrix, with \(p>n\), and let \(y\in\mathbb{R}^n\) be the response vector. For \(\lambda>0\), define the ridge estimator as the minimizer of the squared-error objective plus a squared Euclidean-norm penalty. We use this objective without dividing the squared-error term by \(n\); other scaling conventions amount to a corresponding rescaling of \(\lambda\).

Because \(\lambda>0\), the objective is strictly convex in \(\beta\), so its minimizer is unique. Differentiating the objective gives the normal equation \((X^\mathsf{T}X+\lambda I_p)\hat\beta_\lambda=X^\mathsf{T}y\). The matrix on the left is positive definite, since for every nonzero \(v\), \(v^\mathsf{T}(X^\mathsf{T}X+\lambda I_p)v=\|Xv\|_2^2+\lambda\|v\|_2^2>0\). Thus the ridge estimate exists uniquely for every \(\lambda>0\).

For the limit result, assume additionally that \(X\) has full row rank \(n\). Then \(XX^\mathsf{T}\) is invertible, and the equation \(X\beta=y\) has a solution for every \(y\in\mathbb{R}^n\). The dual expression for ridge is \(\hat\beta_\lambda=X^\mathsf{T}(XX^\mathsf{T}+\lambda I_n)^{-1}y\). It follows either from the normal equation or by substituting the right-hand side into it.

The Zero-Penalty Limit

The singular value decomposition makes the limit and its rate explicit. Under full row rank, write \(X=U D V^\mathsf{T}\), where \(U\in\mathbb{R}^{n\times n}\) and the columns of \(V\in\mathbb{R}^{p\times n}\) are orthonormal, and \(D=\operatorname{diag}(s_1,\ldots,s_n)\) with every \(s_i>0\). Substitution into the dual expression yields \[ \hat\beta_\lambda =V\operatorname{diag}\!\left(\frac{s_i}{s_i^2+\lambda}\right)U^\mathsf{T}y. \]

Proof. In the displayed singular-value representation, each scalar factor \(s_i/(s_i^2+\lambda)\) converges to \(1/s_i\). Since there are finitely many terms, the coefficient vector converges to \(\beta^\dagger=VD^{-1}U^\mathsf{T}y=X^\mathsf{T}(XX^\mathsf{T})^{-1}y\). Applying \(X=UDV^\mathsf{T}\) gives \(X\beta^\dagger=UDV^\mathsf{T}VD^{-1}U^\mathsf{T}y=y\), so the limit interpolates.

To establish minimum norm, let \(\beta\) be any other solution to \(X\beta=y\). Then \(X(\beta-\beta^\dagger)=0\), so \(\beta-\beta^\dagger\) lies in the null space of \(X\). The vector \(\beta^\dagger\) lies in the row space of \(X\), which is orthogonal to that null space. Consequently, \(\|\beta\|_2^2=\|\beta^\dagger\|_2^2+\|\beta-\beta^\dagger\|_2^2\geq\|\beta^\dagger\|_2^2\), with equality only when \(\beta=\beta^\dagger\). This proves both minimality and uniqueness.

The same decomposition also quantifies convergence. If \(s_{\min}=\min_i s_i\), then \(\|\hat\beta_\lambda-\beta^\dagger\|_2\leq \lambda s_{\min}^{-3}\|y\|_2\). Indeed, the absolute difference between the \(i\)th scalar factors is \(\lambda/[s_i(s_i^2+\lambda)]\leq\lambda/s_{\min}^3\), and multiplication by \(V\) preserves the Euclidean norm. This is a fixed-design bound; its constant can be large when the smallest singular value is small. \(\square\)

Predictions and Fitted Values

For a fixed test feature vector \(x\in\mathbb{R}^p\), the ridge prediction is \(x^\mathsf{T}\hat\beta_\lambda\). Coefficient convergence immediately gives prediction convergence: \(|x^\mathsf{T}\hat\beta_\lambda-x^\mathsf{T}\beta^\dagger|\leq\|x\|_2\|\hat\beta_\lambda-\beta^\dagger\|_2\to0\). Thus, at each fixed test point, vanishing-penalty ridge approaches the prediction made by minimum-norm interpolation.

This statement also extends to random test features under a moment condition. If \(Z\in\mathbb{R}^p\) is a random feature vector with \(E[\|Z\|_2^2]<\infty\), then \(E[(Z^\mathsf{T}\hat\beta_\lambda-Z^\mathsf{T}\beta^\dagger)^2]\leq E[\|Z\|_2^2]\|\hat\beta_\lambda-\beta^\dagger\|_2^2\to0\). The inequality follows from Cauchy–Schwarz applied for each realization of \(Z\). It establishes convergence of the difference in predictions in mean square, without asserting that either prediction is accurate for the response.

For the training fitted values, the singular value representation gives \(X\hat\beta_\lambda=U\operatorname{diag}(s_i^2/(s_i^2+\lambda))U^\mathsf{T}y\). The residual in the \(i\)th left-singular direction is therefore multiplied by \(\lambda/(s_i^2+\lambda)\). Every such factor tends to zero under full row rank, which verifies exact interpolation in the limit. At positive \(\lambda\), the fit is generally not exact.

When the Design Is Rank Deficient

Full row rank is essential to the interpolation conclusion. For an arbitrary design matrix, including a rank-deficient one, the general ridge expression is \(\hat\beta_\lambda=(X^\mathsf{T}X+\lambda I_p)^{-1}X^\mathsf{T}y\). Using an SVD that includes only the positive singular values, each coefficient in a positive-singular-value direction converges to the corresponding least-squares coefficient, while the null-space component remains zero. Hence the limit is the minimum-norm least-squares solution. It interpolates if and only if \(y\) lies in the column space of \(X\).

Common Pitfall and Practical Meaning

A common mistake is to reason that because \(\lambda\) tends to zero, ridge must converge to an interpolator for every design. That argument overlooks whether the observed response is even attainable as \(X\beta\). The correction is to check the rank condition: with \(p>n\) and full row rank, every \(y\) is attainable and the limit is the minimum-norm interpolator; without full row rank, the limit is instead the minimum-norm least-squares solution and may leave nonzero residuals.

The limit here is taken for a fixed finite matrix \(X\) and fixed response \(y\). It does not by itself describe what happens when \(n\) and \(p\) change, nor does it establish that the limiting predictor generalizes well. Those are separate questions about the data-generating distribution and the sequence of learning problems. Also, an unpenalized intercept changes the optimization problem: the formulas above apply when every coefficient in \(\beta\) is penalized as specified.

The central connection is exact and useful: under full row rank, ridgeless minimum-norm interpolation can be obtained as the zero-penalty limit of ordinary ridge regression. This provides a continuous path of unique penalized solutions toward a particular interpolating solution, while making clear that small penalty and good prediction are not synonymous.