Prediction Risk Decomposition for Kernel Ridge

Prediction Risk and Its Three Sources

A kernel ridge predictor can make prediction errors for three different reasons. Regularization may shrink the fitted function away from the regression function; the training responses may fluctuate because of noise; and even a predictor that knew the regression function exactly cannot predict a new noisy response perfectly. Separating these sources is useful because they respond differently to the choice of regularization parameter and sample size.

We work under squared loss. Let $(X_i,Y_i)$, $i=1,\ldots,n$, be independent observations from a common distribution, and let $(X_0,Y_0)$ be an independent test observation from that distribution. Assume $Y_i=f_*(X_i)+\varepsilon_i$, where $f_*(x)=E[Y\mid X=x]$, $E[\varepsilon_i\mid X_i]=0$, and $E[\varepsilon_i^2\mid X_i]=\tau^2(X_i)<\infty$. The function $f_*$ is the regression function, and $\tau^2(X)$ is the conditional noise variance.

Let $\mathcal H$ be a reproducing kernel Hilbert space with kernel $k$, and write $k_x=k(x,\cdot)$. For $\lambda>0$, kernel ridge regression (KRR) minimizes the average squared training loss plus a squared RKHS norm penalty. Assume $E[k(X,X)]<\infty$, and that the relevant second moments are finite. These conditions ensure the operators and prediction risks below are well-defined; the positive penalty gives a unique minimizer.

Definition: The KRR estimator and its noiseless-response counterpart are $$ \hat f_\lambda=\arg\min_{f\in\mathcal H} \left\{\frac{1}{n}\sum_{i=1}^n(Y_i-f(X_i))^2+\lambda\|f\|_{\mathcal H}^2\right\}, \qquad f^0_{\lambda,X_{1:n}}=\arg\min_{f\in\mathcal H} \left\{\frac{1}{n}\sum_{i=1}^n(f_*(X_i)-f(X_i))^2+\lambda\|f\|_{\mathcal H}^2\right\}. $$ The second fit uses the same training inputs but replaces each noisy response by its conditional mean. It is therefore a useful center for separating response-noise variation from prediction bias.

For a function $g$, write $\|g\|_{L^2(P_X)}^2=E[g(X)^2]$, where $X$ is a fresh covariate. The prediction risk considered here is $E[(\hat f_\lambda(X_0)-Y_0)^2]$, averaging over the training data and the independent test observation. The exact decomposition centers the predictor at $f^0_{\lambda,X_{1:n}}$, which depends on the training inputs.

The Exact Finite-Sample Decomposition

Theorem: Under the conditions above, $$ E[(\hat f_\lambda(X_0)-Y_0)^2] = E_{X_{1:n}}\!\left[ \|f^0_{\lambda,X_{1:n}}-f_*\|_{L^2(P_X)}^2 + E_{\varepsilon_{1:n}}\!\left[ \|\hat f_\lambda-f^0_{\lambda,X_{1:n}}\|_{L^2(P_X)}^2 \,\middle|\,X_{1:n} \right]\right] + E[\tau^2(X)]. $$ The three terms are, respectively, sample-dependent regularization bias, estimation variance from the training responses, and irreducible test noise.

Here $E_{X_{1:n}}$ averages over the training inputs. In the inner expectation, the inputs are fixed and only the training errors vary. The name “regularization bias” is apt because $f^0_{\lambda,X_{1:n}}$ is the penalized fit to the noiseless training targets. It can also reflect which parts of the input space the sample covers.

Write the normal equations.
Define $T_n=\frac{1}{n}\sum_{i=1}^n k_{X_i}\otimes k_{X_i}$, where $(k_x\otimes k_x)f=f(x)k_x$. The KRR normal equation is $(T_n+\lambda I)\hat f_\lambda=\frac{1}{n}\sum_iY_i k_{X_i}$. The noiseless fit satisfies the same equation with $Y_i$ replaced by $f_*(X_i)$.

Isolate response noise.
Subtracting the two equations gives $$ \hat f_\lambda-f^0_{\lambda,X_{1:n}} =(T_n+\lambda I)^{-1}\frac{1}{n}\sum_{i=1}^n\varepsilon_i k_{X_i}. $$ Conditional on the inputs, this difference has mean zero.

Expand squared prediction error.
For fixed inputs, the conditional mean-zero difference makes the cross term in the squared $L^2(P_X)$ norm vanish. Thus the conditional prediction error for estimating $f_*$ is the squared distance from $f^0_{\lambda,X_{1:n}}$ to $f_*$, plus the conditional expected squared distance of $\hat f_\lambda$ from $f^0_{\lambda,X_{1:n}}$.

Add test noise.
Conditional on $X_0$, the test error $\varepsilon_0$ has mean zero and is independent of the training data. Its cross term with $\hat f_\lambda(X_0)-f_*(X_0)$ vanishes, while $E[\varepsilon_0^2]=E[\tau^2(X)]$.

For completeness, the vanishing cross term in Step 3 follows by expanding the square. Let $d=f^0_{\lambda,X_{1:n}}-f_*$ and $u=\hat f_\lambda-f^0_{\lambda,X_{1:n}}$. Given the inputs, $d$ is fixed and $E[u(X)\mid X_{1:n}]=0$ for $P_X$-almost every $X$. Therefore $E[\langle d,u\rangle_{L^2(P_X)}\mid X_{1:n}]=0$, and $E[\|d+u\|_{L^2(P_X)}^2\mid X_{1:n}]=\|d\|_{L^2(P_X)}^2+E[\|u\|_{L^2(P_X)}^2\mid X_{1:n}]$. Averaging over inputs and adding test noise proves the result. $\square$

Worked Example: Constant Kernel and Three Explicit Risk Terms

Let $X$ be arbitrary, use the constant kernel $k(x,x')=1$, and let $\mathcal H$ consist of constant functions $f(x)=a$, with $\|f\|_{\mathcal H}^2=a^2$. Suppose $Y=2+\varepsilon$, where $E[\varepsilon\mid X]=0$ and $\operatorname{Var}(\varepsilon\mid X)=9$. Take $n=4$ and $\lambda=1$.

The objective as a function of $a$ is $\frac14\sum_i(Y_i-a)^2+a^2$. Differentiating and setting the derivative to zero gives $-2(\bar Y-a)+2a=0$, so $\hat a=\bar Y/2$. The noiseless counterpart is $a^0=2/2=1$. Its squared prediction bias is $(1-2)^2=1$.

Since $\hat a-a^0=\bar\varepsilon/2$, its variance is $\operatorname{Var}(\bar\varepsilon)/4=(9/4)/4=9/16$. The irreducible test noise is $9$. Consequently the expected squared prediction error is $1+9/16+9=169/16$. The three contributions are all on the prediction scale: squared function error, variance of the fitted function, and test-response noise.

Population Regularization Bias

The population KRR solution is the minimizer of the expected penalized objective, rather than the objective formed from a finite training sample. It helps quantify the shrinkage due solely to regularization, but it is not generally the conditional mean of the finite-sample estimator.

Definition: Define the population covariance operator on $\mathcal H$ by $Tf=E[f(X)k_X]$. The population KRR solution is $$ f_\lambda=\arg\min_{f\in\mathcal H} \left\{E[(Y-f(X))^2]+\lambda\|f\|_{\mathcal H}^2\right\}. $$ When $f_*\in\mathcal H$, it satisfies $f_\lambda=(T+\lambda I)^{-1}Tf_*$. The population regularization bias is $f_\lambda-f_*$, measured for prediction by $\|f_\lambda-f_*\|_{L^2(P_X)}^2$.

The inverse in this expression exists as a bounded operator because $T$ is positive and $\lambda>0$. The normal equation follows by differentiating the population objective in any direction $g\in\mathcal H$. Since $E[Y\mid X]=f_*(X)$, the derivative condition is $\langle Tf_\lambda-Tf_*+\lambda f_\lambda,g\rangle_{\mathcal H}=0$ for every $g$. This implies $(T+\lambda I)f_\lambda=Tf_*$, and hence the stated formula. In particular, $f_\lambda-f_*=-\lambda(T+\lambda I)^{-1}f_*$ when $f_*\in\mathcal H$. This is the population shrinkage bias; it need not equal the first term in the finite-sample decomposition.

Worked Example: Population Shrinkage on Two Input Points

Let $X$ take values $x_1,x_2$ with probability $1/2$ each. Consider functions represented by $f(x_1)=a_1$, $f(x_2)=a_2$, with RKHS norm squared $a_1^2+a_2^2$. Suppose $f_*(x_1)=2$, $f_*(x_2)=1$, and $\lambda=1/2$.

Ignoring the constant noise contribution, the population objective is $\frac12(2-a_1)^2+\frac12(1-a_2)^2+\frac12(a_1^2+a_2^2)$. The two coordinates minimize separately. Differentiating the first coordinate gives $-(2-a_1)+a_1=0$, so $a_1=1$. The second gives $-(1-a_2)+a_2=0$, so $a_2=1/2$.

The squared population prediction bias is $\frac12(1-2)^2+\frac12(1/2-1)^2=\frac12+\frac18=\frac58$. If the conditional noise variance is $1$ at both points, the population solution’s expected squared error for a fresh response is $5/8+1=13/8$. This calculation concerns the population minimizer; a finite-sample KRR risk also includes sampling and response-noise variation.

When the Population Bias Is Not the Finite-Sample Bias

A tempting shortcut is to write finite-sample prediction risk as population regularization bias squared plus a variance term plus irreducible noise. That identity does not generally hold with $f_\lambda$ as the center. Even after averaging over response noise, the finite-sample noiseless fit depends on the random training inputs and need not equal $f_\lambda$. Expanding around $f_\lambda$ produces a cross term: $$ \|f^0_{\lambda,X_{1:n}}-f_*\|_{L^2(P_X)}^2 = \|f_\lambda-f_*\|_{L^2(P_X)}^2 +\|f^0_{\lambda,X_{1:n}}-f_\lambda\|_{L^2(P_X)}^2 +2\langle f_\lambda-f_*,f^0_{\lambda,X_{1:n}}-f_\lambda\rangle_{L^2(P_X)}. $$ The last term is not automatically zero. It can disappear under additional conditions, such as a suitable unbiasedness property, but it must not be discarded without justification.

Worked Example: Unobserved Inputs Create Additional Bias

Use the two-point input space from the preceding example, but suppose all $n=2$ training inputs equal $x_1$. The test distribution still assigns probability $1/2$ to each point, and the target values are $f_*(x_1)=1$ and $f_*(x_2)=3$. Use the same RKHS norm $a_1^2+a_2^2$, $\lambda=1$, and conditional noise variance $4$.

The noiseless empirical objective is $(1-a_1)^2+a_1^2+a_2^2$, because both training inputs equal $x_1$. Its minimizer is $a_1=1/2$, $a_2=0$. Thus the sample-dependent squared bias on the test distribution is $\frac12(1/2-1)^2+\frac12(0-3)^2=\frac18+\frac92=\frac{37}{8}$.

With noisy responses, the fitted first coordinate is $\hat a_1=\bar Y/2$, so its conditional variance is $(4/2)/4=1/2$; the second coordinate remains zero. Only half of the test inputs have a prediction affected by $a_1$, giving integrated estimation variance $\frac12\cdot\frac12=\frac14$. Adding irreducible noise $4$ gives total risk $\frac{37}{8}+\frac14+4=\frac{71}{8}$. The large bias includes the fact that $x_2$ was never observed, not merely shrinkage of an otherwise well-supported target.

Why the Decomposition Matters

The exact decomposition is useful because it identifies what a change in $\lambda$ or in the data can affect. Stronger regularization may reduce response-noise variation while increasing shrinkage bias. More training data can change the variance and the distribution of the training inputs, which in turn changes the sample-dependent bias. The irreducible term $E[\tau^2(X)]$ is a property of the response distribution, not a quantity that KRR can remove.

A common mistake is to call every difference between an estimated predictor and $f_*$ “variance,” or to treat the population regularization bias as the complete finite-sample bias. The correction is to specify what is being averaged and what function is used as the center. Conditional on the inputs, the noiseless KRR fit gives an exact zero-cross-term decomposition. The population solution describes regularization at the population level, but generally does not replace that sample-dependent center.

Check Your Understanding

Use the definitions and decomposition above to answer the following questions.

Why does the test-response noise contribute $E[\tau^2(X)]$ to squared prediction risk?
What is held fixed in the conditional estimation-variance term of the theorem?
Why can the noiseless sample fit differ from the population KRR solution?
In the constant-kernel example, which calculation gives the variance contribution $9/16$?
When expanding the squared bias around $f_\lambda$, what condition would be needed to discard the cross term after averaging?