Early Stopping Near The Interpolation Limit

Why Compare a Finite-Time Fit With Interpolation?

In an overparameterized linear model, gradient descent can reach zero training error while fitting noise as well as signal. But the prediction risk need not improve all the way to that limit. At a finite time, some directions have not yet been fully fitted; this can leave signal unexplained, while also limiting how much label noise is fitted. Early stopping is useful when the reduction in noise-related error outweighs the remaining signal-related error.

The comparison depends on how quickly each direction is learned and how strongly that direction contributes to prediction on new data. We make those roles explicit in a finite-dimensional spectral model. The model isolates the stopping-time effect; it is not a claim that every design or data-generating process has the same behavior.

A Linear Spectral Model

Let there be \(p\) orthogonal modes. In mode \(j\), let \(\lambda_j>0\) be the curvature of the training loss, \(\beta_j\) the true signal coefficient, and \(Z_j=\beta_j+\varepsilon_j\) the coefficient that exactly fits the observed training target. Assume the errors are independent, with \(E[\varepsilon_j]=0\) and \(\operatorname{Var}(\varepsilon_j)=\tau_j^2\). The squared training loss, up to an irrelevant positive scale, is

We study gradient flow initialized at \(\theta_j(0)=0\). For prediction, let \(\mu_j\geq 0\) be the weight of mode \(j\) under the test distribution. For example, if test covariates have diagonal covariance with entries \(\mu_j\), the excess prediction risk of \(\theta\), relative to the true regression function, is \(\sum_j\mu_j(\theta_j-\beta_j)^2\). The \(\lambda_j\) and \(\mu_j\) need not be equal: training curvature controls the speed of fitting, while test weights control the prediction cost of an error.

The coordinate-wise gradient-flow equation is \(\dot\theta_j(t)=-\lambda_j(\theta_j(t)-Z_j)\). Multiplying by the integrating factor \(e^{\lambda_jt}\), or directly solving this scalar differential equation with the stated initial value, gives

Thus the weight applied to the observed coefficient is \(1-e^{-\lambda_jt}\). A large-curvature mode approaches its fitted value more quickly; as \(t\to\infty\), every mode with \(\lambda_j>0\) converges to \(Z_j\).

Risk at Finite Time and at the Limit

Write \(q_j(t)=e^{-\lambda_jt}\), so that \(\theta_j(t)=(1-q_j(t))(\beta_j+\varepsilon_j)\). The estimation error in mode \(j\) is

The first term is the remaining signal error; the second is the fitted noise. The following result gives the exact risk, not just an asymptotic approximation.

Proof. Square the mode-wise error. Its expected squared value is

Because \(E[\varepsilon_j]=0\) and \(E[\varepsilon_j^2]=\tau_j^2\), this equals \(q_j(t)^2\beta_j^2+(1-q_j(t))^2\tau_j^2\). Multiply by \(\mu_j\) and sum over the finite set of modes to obtain \(R(t)\). Since every \(\lambda_j>0\), \(q_j(t)\to0\) as \(t\to\infty\), so the formula gives \(R(\infty)=\sum_j\mu_j\tau_j^2\). \(\square\)

When Does Stopping Beat the Interpolating Limit?

Subtracting the limiting risk from the finite-time risk gives an exact comparison. In each mode, stopping leaves a squared signal error but avoids some of the noise error that full interpolation incurs.

Proof. For each mode, subtract its contribution to \(R(\infty)\) from its contribution to \(R(t)\). With \(q_j=e^{-\lambda_jt}\), the difference is

Summing over \(j\) yields \(R(t)-R(\infty)\). The inequality \(R(t)<R(\infty)\) holds exactly when the sum of the signal-error terms is less than the sum of the avoided-noise terms, which is the displayed condition. \(\square\)

This condition is not a guarantee that early stopping helps. Signal strength, noise, test weights, and learning rates all matter. In particular, a mode with a large \(\mu_j\) can dominate prediction risk even if it has little influence on the training loss. Conversely, a mode with small \(\mu_j\) may be fitted quickly without materially changing test risk.

Choosing a Stopping Time in One Mode

In one mode with \(\lambda>0\), \(\mu>0\), \(\beta^2>0\), and \(\tau^2>0\), the risk as a function of \(q=e^{-\lambda t}\in(0,1]\) is

Differentiating with respect to \(q\) gives \(R'(q)=2\mu\{(\beta^2+\tau^2)q-\tau^2\}\). Since \(R''(q)=2\mu(\beta^2+\tau^2)>0\), the unique minimum occurs at \(q=\tau^2/(\beta^2+\tau^2)\). This lies strictly between zero and one, and hence corresponds to the finite time

Substitution gives \(R(t^*)=\mu\beta^2\tau^2/(\beta^2+\tau^2)\), while \(R(\infty)=\mu\tau^2\). Under the stated positive-signal, positive-noise assumptions, \(R(t^*)<R(\infty)\). The optimal time increases as the signal-to-noise ratio increases: when the signal is stronger relative to the noise, more fitting is worthwhile. This one-mode calculation does not supply a common optimal time for multiple modes, since their curvatures and signal-to-noise ratios may differ.

Discrete Gradient Descent and a Common Pitfall

The same calculation applies to ordinary gradient descent on the stated quadratic loss. With step size \(\eta\), the update in mode \(j\) is \(\theta_j^{(k+1)}=\theta_j^{(k)}-\eta\lambda_j(\theta_j^{(k)}-Z_j)\). If \(0<\eta<1/\max_j\lambda_j\), then iteration from zero gives \(\theta_j^{(k)}=[1-(1-\eta\lambda_j)^k]Z_j\). The continuous-time factor \(e^{-\lambda_jt}\) is replaced by \((1-\eta\lambda_j)^k\), and the risk formulas follow with that substitution. This step-size condition ensures the factors lie in \((0,1)\); larger stable step sizes can produce oscillating factors, so the simple monotone-shrinkage interpretation no longer applies directly.

Common mistake: “The interpolating solution has zero training error, so it must have the smallest prediction risk.” This confuses fitting the observed targets with predicting the true signal. In the model, the interpolator has risk \(\sum_j\mu_j\tau_j^2\), not necessarily zero: its prediction error includes the fitted label noise. At finite time, the estimator can have nonzero training error and lower prediction risk, as the exact comparison condition and worked examples show. The reverse mistake is also possible: early stopping is not automatically beneficial. If the remaining signal error is large relative to the noise avoided, stopping increases risk.