Bias Bounds Under Source Smoothness

Why Source Smoothness Controls Bias

Population kernel ridge regression replaces a target function by a regularized version of it. The penalty reduces sensitivity to directions where the covariance operator has small eigenvalues, but this also leaves a systematic difference between the target and its regularized version: the population regularization bias. A source condition expresses how much of the target lies in directions of different covariance scales. The key question is how that smoothness translates into prediction error as the regularization parameter decreases.

We work in a reproducing kernel Hilbert space \(\mathcal H\), with feature functions \(K_x=K(x,\cdot)\). Assume \(E[K(X,X)]<\infty\), so the covariance operator \(T:\mathcal H\to\mathcal H\), defined by \(Th=E[h(X)K_X]\), is bounded, positive, and self-adjoint. Let \(f_*\in\mathcal H\) be the population regression function, and use squared loss with penalty \(\lambda\|f\|_{\mathcal H}^2\), where \(\lambda>0\). The population prediction seminorm is \(\|h\|_T^2=\langle h,Th\rangle_{\mathcal H}=E[h(X)^2]\).

The formula for \(f_\lambda\) follows by minimizing the population squared-error objective: its part depending on \(f\) is \(\langle f,Tf\rangle_{\mathcal H}-2\langle f,Tf_*\rangle_{\mathcal H}+\lambda\|f\|_{\mathcal H}^2\). Since \(T+\lambda I\) is strictly positive and has a bounded inverse, the unique minimizer satisfies \((T+\lambda I)f_\lambda=Tf_*\). This definition isolates regularization bias from sampling variability and noise.

The Spectral Form of the Bias

The bias has a particularly simple expression before taking its prediction norm. Rearranging the population normal equation gives \[ f_\lambda-f_*=-\lambda(T+\lambda I)^{-1}f_*. \] Under the source condition, the spectral theorem therefore represents the squared prediction bias as an integral of a scalar function of the eigenvalue. This representation is the step that makes the exponent range transparent.

Proof. The normal equation gives \(f_\lambda-f_*=-\lambda(T+\lambda I)^{-1}T^rg\). By the spectral theorem, \(T^r\), \(T\), and \((T+\lambda I)^{-1}\) are functions of the same operator. The squared \(T\)-seminorm is consequently the squared norm of \(T^{1/2}(f_\lambda-f_*)\), and its spectral multiplier applied to \(g\) is \(-\lambda t^{r+1/2}/(t+\lambda)\). Squaring and integrating against the spectral measure of \(g\) proves the identity. The integrand at \(t=0\) is zero, so the formula also covers a null space of \(T\).

For \(t>0\), put \(u=t/\lambda\). Then \[ \frac{\lambda^2t^{2r+1}}{(t+\lambda)^2} =\lambda^{2r+1}\frac{u^{2r+1}}{(1+u)^2}. \] If \(0\leq r\leq \tfrac12\), the function on the right is bounded over \(u\geq0\): near zero it tends to zero, and as \(u\) tends to infinity it behaves as \(u^{2r-1}\), which is bounded for this range of \(r\). Thus the integrand is at most \(C_r\lambda^{2r+1}\). Since the total spectral mass \(\int d\|E_tg\|^2\) equals \(\|g\|_{\mathcal H}^2\), integration proves the bound. \(\square\)

The endpoint \(r=\tfrac12\) is included: the scalar function is \(u^2/(1+u)^2\), bounded above by one. For \(0\leq r<\tfrac12\), its maximum occurs at \(u=(2r+1)/(1-2r)\). To check this, differentiate its logarithm for \(u>0\): \[ \frac{2r+1}{u}-\frac{2}{1+u}=0 \quad\Longleftrightarrow\quad u=\frac{2r+1}{1-2r}. \] This also shows why the exponent limit is not an arbitrary technical restriction: it is exactly where the large-\(u\) behavior of the multiplier changes from bounded to unbounded.

Worked Bias Calculations

What the Exponent Range Does—and Does Not—Say

For \(0\leq r\leq\tfrac12\), the source condition guarantees a uniform bound of order \(\lambda^{2r+1}\) for squared prediction bias. The exponent is determined by two factors: \(T^r\) contributes \(t^r\) to the error, while measuring prediction error contributes another factor \(t^{1/2}\). The regularization error contributes \(\lambda/(t+\lambda)\). Squaring their product gives the multiplier in the theorem.

The range comes from asking for a bound uniform over all spectral scales after factoring out \(\lambda^{2r+1}\). When \(r>\tfrac12\), the remaining scalar multiplier grows at large \(u=t/\lambda\), rather than staying bounded. The one-eigenspace example verifies that the asserted power can then fail even for a fixed bounded operator and a fixed source element. This conclusion concerns that particular uniform power bound; it does not say the bias cannot be controlled by other bounds under additional assumptions.

A common mistake is to see a factor \(\lambda^2\) in the squared bias and conclude that the bias is always of order \(\lambda^2\), or to infer an arbitrarily fast power by taking \(r\) large in the source condition. The exact spectral multiplier shows why neither inference is justified without checking its supremum. Smoothness affects the multiplier, but the regularization filter and the prediction norm jointly determine which claimed rate is valid.