Kernel Ridge Saturation Beyond Its Qualification

Why More Smoothness Can Stop Helping

Kernel ridge regression shrinks every covariance direction, but it does not remove that shrinkage equally in all directions. The prediction bias is small in directions with small covariance eigenvalues and is of order $\lambda$ in directions with eigenvalues well above the regularization level $\lambda$. A source condition describes how much of the target lies in those directions. For sufficiently smooth targets, the low-eigenvalue part becomes less important, but the shrinkage in the well-resolved directions remains.

This produces saturation: beyond a threshold source exponent, the worst-case prediction-bias rate over a source ball no longer improves as the target is assumed smoother. The threshold depends on the norm in which bias is measured. Here the source condition is in the covariance scale, while prediction error includes an additional square root of the covariance operator.

Population Bias in Spectral Coordinates

Let $\mathcal H$ be a Hilbert space and let $T:\mathcal H\to\mathcal H$ be the bounded, positive, self-adjoint covariance operator of the inputs. Assume $T$ is compact, as in the usual kernel setting. Write its positive eigenvalues and orthonormal eigenvectors as $(\mu_j,e_j)$. Components in the null space of $T$ do not affect prediction error. For a target $f_*\in\mathcal H$, the population kernel ridge solution at $\lambda>0$ is $f_\lambda=(T+\lambda I)^{-1}Tf_*$.

The prediction norm of an error $h\in\mathcal H$ is $\|T^{1/2}h\|_{\mathcal H}$; its square is the corresponding squared prediction error. The source condition used here is $f_*=T^r g$, where $r\geq 0$ and $g\in\mathcal H$. A source ball of radius $R$ consists of such targets with $\|g\|_{\mathcal H}\leq R$.

Definition: The prediction bias at regularization level $\lambda$ is $T^{1/2}(f_\lambda-f_*)$. Under the source condition $f_*=T^r g$, its spectral expression is $$ T^{1/2}(f_\lambda-f_*)=-\lambda T^{r+1/2}(T+\lambda I)^{-1}g. $$ The squared prediction bias is the squared norm of this vector. Its scalar multiplier at eigenvalue $\mu$ is $\lambda\mu^{r+1/2}/(\mu+\lambda)$.

The identity follows from $f_\lambda-f_*=-\lambda(T+\lambda I)^{-1}f_*$. The exponent $r+1/2$, rather than $r$, appears because prediction error applies $T^{1/2}$. This distinction is essential: Tikhonov regularization has qualification one in the unweighted solution norm, corresponding here to a prediction-norm source threshold of $r=1/2$.

Worked Example: One Eigen-direction Shows the Residual Shrinkage

Suppose $Te=e$ for a unit eigenvector $e$, and take $g=e$, so that $f_*=T^r g=e$ for every $r\geq0$. At $\lambda=0.01$, $$ f_\lambda=(T+\lambda I)^{-1}Te=\frac{1}{1.01}e, \qquad f_\lambda-f_*=-\frac{0.01}{1.01}e. $$

Since $T^{1/2}e=e$, the prediction-bias norm is $0.01/1.01\approx0.009901$, and the squared prediction bias is $(0.01/1.01)^2\approx0.00009803$. Increasing $r$ does not change this calculation: on this eigen-direction, $T^r$ acts as the identity. Thus smoothness does not eliminate the order-$\lambda$ residual shrinkage in directions with a fixed positive eigenvalue.

The Saturation Bound

Theorem: Suppose $f_*=T^r g$, $T$ is bounded and positive, and $\|g\|_{\mathcal H}\leq R$. For $0\leq r\leq 1/2$, there is a constant $C_r$, independent of $\lambda$ and $g$, such that $$ \|T^{1/2}(f_\lambda-f_*)\|_{\mathcal H} \leq C_r R\lambda^{r+1/2}. $$ For $r\geq1/2$, $$ \|T^{1/2}(f_\lambda-f_*)\|_{\mathcal H} \leq R\|T\|^{r-1/2}\lambda. $$ Consequently, the uniform squared prediction-bias bound has order $\lambda^{\min(2r+1,\,2)}$, up to constants. The exponent saturates at $2$ once $r\geq1/2$.

Proof. By the spectral theorem, the operator norm of the bias map from $g$ to prediction bias is the supremum, over positive eigenvalues $\mu\leq\|T\|$, of $$ m_{\lambda,r}(\mu)=\frac{\lambda\mu^{r+1/2}}{\mu+\lambda}. $$ For $0\leq r<1/2$, set $\alpha=r+1/2$, so $0<\alpha<1$, and write $\mu=\lambda x$. Then $$ m_{\lambda,r}(\mu)=\lambda^\alpha\frac{x^\alpha}{1+x}. $$ The function $x^\alpha/(1+x)$ on $x\geq0$ is bounded: its derivative has the sign of $\alpha-(1-\alpha)x$, so its maximum occurs at $x=\alpha/(1-\alpha)$. Hence the supremum is at most $C_r\lambda^{r+1/2}$. At $r=1/2$, the multiplier is $\lambda\mu/(\mu+\lambda)\leq\lambda$, giving the stated bound there too.

For $r>1/2$, factor the multiplier as $$ m_{\lambda,r}(\mu)=\lambda\mu^{r-1/2}\frac{\mu}{\mu+\lambda} \leq\lambda\|T\|^{r-1/2}. $$ Multiplying these operator-norm bounds by $\|g\|_{\mathcal H}\leq R$ proves both claims for the bias norm. Squaring gives the stated orders for squared bias. $\square$

The bound for $r\geq1/2$ is not just a loose consequence of the proof. If $T$ has a positive eigenvalue $\mu_0$ with unit eigenvector $e_0$, choose $g=e_0$. Then $$ \|T^{1/2}(f_\lambda-f_*)\|_{\mathcal H} =\frac{\lambda\mu_0^{r+1/2}}{\mu_0+\lambda} \sim \lambda\mu_0^{r-1/2} \quad\text{as }\lambda\downarrow0. $$ For this fixed source element, the bias is of order $\lambda$, not $o(\lambda)$. Therefore no uniform bound of smaller order than $\lambda$ in bias norm can hold over the entire source ball when $T$ has a positive eigenvalue. The constant can depend on $r$ and on the operator; the saturation statement concerns the power of $\lambda$.

This is a worst-case statement, not a claim that every smoother fixed target has exactly the same bias for every spectrum. Some targets have no component in large-eigenvalue directions and can have faster bias decay. Saturation says that increasing the source exponent alone cannot guarantee that improvement uniformly over the source ball.

Worked Example: Comparing Source Exponents at a Fixed Regularization Level

Assume the spectrum lies in $[0,1]$, take $\lambda=0.01$, and consider source elements with $\|g\|_{\mathcal H}\leq1$. For $r=1/4$, the multiplier is $0.01\mu^{3/4}/(\mu+0.01)$. Its maximum over positive $\mu$ occurs at $\mu=3\lambda=0.03$, by the maximizer calculation in the proof. The maximum value is $$ 0.01\frac{0.03^{3/4}}{0.04}\approx0.01802. $$ So the bound on the bias norm is about $0.01802$, and on squared bias it is about $0.0003247$.

For $r=1$, the multiplier is $0.01\mu^{3/2}/(\mu+0.01)$, which is increasing on $[0,1]$ and reaches its maximum at $\mu=1$. Its maximum is $0.01/1.01\approx0.009901$, giving squared bias about $0.00009803$. A naive continuation of the below-threshold formula would predict a bias norm of order $\lambda^{3/2}=0.001$, which is not a valid uniform bound: the eigen-direction at $\mu=1$ already has bias $0.009901$.

Why the Prediction-Norm Threshold Is One-Half

In the unweighted RKHS norm, the source-$r$ residual multiplier is $\lambda\mu^r/(\mu+\lambda)$. Its uniform source bound improves as $\lambda^r$ only up to $r=1$; above that, it is of order $\lambda$. This is the usual qualification-one statement for Tikhonov regularization. Prediction error adds a factor $\mu^{1/2}$, shifting the source exponent at which the corresponding saturation occurs from $1$ to $1/2$. The filter itself has not changed; the norm and source convention determine how its qualification is expressed.

Worked Example: Saturation in a Polynomial-Spectrum Rate

Suppose the covariance eigenvalues have polynomial decay $\mu_j\asymp j^{-2}$, and use an effective-dimension variance bound of order $\lambda^{-1/2}/n$. Consider the usual upper-bound balance between this variance term and squared prediction bias. If $r=1/4$, the bias order from the theorem is $\lambda^{2r+1}=\lambda^{3/2}$. Balancing gives $$ \lambda^{3/2}\asymp \frac{\lambda^{-1/2}}{n}, \qquad \lambda^2\asymp n^{-1}, \qquad \lambda\asymp n^{-1/2}. $$ At this choice, each term is of order $n^{-3/4}$, since $\lambda^{3/2}=n^{-3/4}$ and $\lambda^{-1/2}/n=n^{-3/4}$.

Now take $r=1$, which is above the prediction-norm saturation threshold. The squared bias is of order $\lambda^2$, not $\lambda^3$. The balance is $$ \lambda^2\asymp \frac{\lambda^{-1/2}}{n}, \qquad \lambda^{5/2}\asymp n^{-1}, \qquad \lambda\asymp n^{-2/5}. $$ Both terms are then of order $n^{-4/5}$. Any still larger source exponent leaves this bias–variance balance unchanged under the same assumptions. These are rates obtained by balancing the stated bounds; they do not assert a matching lower bound for every model.

Common Mistakes and Practical Meaning

Using the wrong threshold. Saying “qualification one, therefore prediction bias improves until $r=1$” confuses the unweighted RKHS norm with prediction norm. The covariance square root in prediction error shifts the threshold to $r=1/2$ for the source convention $f_*=T^r g$.
Extending the smoothness formula indefinitely. The bound $\lambda^{2r+1}$ for squared prediction bias is valid only through $r=1/2$. Above that point, the uniform bound is of order $\lambda^2$.
Interpreting saturation pointwise. Saturation is a limit on uniform guarantees over a source ball. Particular targets can have faster decay if their spectral coefficients have additional structure.
Ignoring the regularization scale. The bias bound alone does not determine a statistical rate. In a risk calculation, it must be balanced against a variance term under the relevant assumptions on the data and noise.

Saturation matters because a source condition is often used to describe target smoothness before choosing $\lambda$. Once the exponent exceeds the qualification threshold, asserting still greater smoothness does not justify a better worst-case prediction-bias power for kernel ridge. To obtain a faster guarantee, one needs additional structure or a different regularization method, rather than simply increasing the source exponent in the same bound.

Check Your Understanding

Use the covariance-scale source condition $f_*=T^r g$ and prediction norm throughout.

Why does the prediction-bias multiplier contain $\mu^{r+1/2}$ rather than $\mu^r$?
For which source exponents does the squared prediction-bias bound improve as $\lambda^{2r+1}$, and what order applies above that range?
If $T e=\mu e$ with $\mu>0$ and $g=e$, derive the prediction-bias norm and its order as $\lambda$ tends to zero.
Explain why a fixed target may have faster bias decay without contradicting a saturation result over a source ball.
Under a variance bound of order $\lambda^{-1/2}/n$, what regularization order balances variance and saturated squared bias?