Misspecified Targets and Approximation Residuals

When the Target Is Not Representable by the Kernel

Kernel ridge regression is often analyzed under a source condition that represents the regression function through the kernel operator. But a real target need not belong to the kernel range, and it may not have a representation with a suitably small norm. In that case, the regularization bias cannot be summarized by smoothness alone: it also depends on how well the target can be approximated at the chosen regularization level.

The central object here is a scale-dependent approximation residual. It records both the prediction error of an approximation and the cost of using that approximation in the kernel space. This makes it possible to distinguish three effects: approximation limitations of the kernel, regularization, and estimation from a finite sample.

Population Approximation and Its Residual

Let \(L_2(P_X)\) be the space of square-integrable functions of the predictor, with norm \(\|\cdot\|_{L_2(P_X)}\). Let \(\mathcal H\) be the reproducing kernel Hilbert space, and let \(S:\mathcal H\to L_2(P_X)\) be the inclusion map, assumed bounded. Write \(T=SS^*\) for the corresponding positive covariance operator on \(L_2(P_X)\). Let \(f_*(x)=E[Y\mid X=x]\), assumed to belong to \(L_2(P_X)\).

For \(\lambda>0\), the population kernel ridge solution \(g_\lambda\) minimizes the population squared prediction loss plus the RKHS penalty. Its prediction is denoted by \(f_\lambda=Sg_\lambda\). The approximation residual is the error \(f_*-Sg\) left by a candidate \(g\in\mathcal H\); a useful scale-dependent summary also charges for the candidate's RKHS norm.

The range \(S(\mathcal H)\) need not be closed. Thus a target can be outside the range but still be approximable arbitrarily well in \(L_2(P_X)\), usually at the cost of approximants with increasingly large RKHS norm. The parameter \(\lambda\) controls how much of that cost is acceptable.

Two Bounds for the Population Regularization Error

The population solution exists and is unique for \(\lambda>0\), because its objective is a continuous, strictly convex quadratic functional with a positive norm penalty. Its prediction has the operator representation \(f_\lambda=T(T+\lambda I)^{-1}f_*\). The following comparison shows directly how any candidate approximation controls its residual.

Proof. The conditional-mean property gives \(E[(Y-Sg(X))^2]=E[(Y-f_*(X))^2]+\|Sg-f_*\|_{L_2(P_X)}^2\). The first term does not depend on \(g\), so minimizing population squared loss plus \(\lambda\|g\|_{\mathcal H}^2\) is equivalent to minimizing the expression defining \(A_\lambda(f_*)\). At its minimizer, the residual term is no larger than the full objective. Hence \(\|f_\lambda-f_*\|_{L_2(P_X)}^2\leq A_\lambda(f_*)\).

For the second bound, write \(f_*=Sg+e\). The operator formula gives \(f_\lambda-f_*=-\lambda(T+\lambda I)^{-1}(Sg+e)\). Since \(T\) is positive, the spectral theorem implies \(\|\lambda(T+\lambda I)^{-1}\|_{\mathrm{op}}\leq1\). Also, using \(T=SS^*\), \(\|\lambda(T+\lambda I)^{-1}S\|_{\mathrm{op}}\leq\sqrt{\lambda}\): on a spectral value \(t\geq0\), the relevant multiplier is \(\lambda\sqrt{t}/(t+\lambda)\leq\sqrt{\lambda}\). The triangle inequality now yields the stated bound. \(\square\)

The first inequality is often more useful when minimizing over candidate approximations. The second makes the residual-norm tradeoff visible: a small residual is helpful, but an approximant with a very large RKHS norm can be expensive at a fixed \(\lambda\).

Estimation Error Is a Separate Term

The population approximation error describes what regularized population fitting can achieve. A finite-sample estimator has an additional discrepancy because it is fitted to random data. To state the separation precisely, suppose \((X_i,Y_i)_{i=1}^n\) are independent and identically distributed, \(E[Y^2]<\infty\), and \(\hat g\) is a measurable empirical kernel ridge solution with penalty \(\lambda>0\). Define its prediction risk above the regression function by \(E\|S\hat g-f_*\|_{L_2(P_X)}^2\), where the outer expectation is over the training sample.

Proof. For every realized training sample, insert and subtract \(Sg_\lambda\). The Hilbert-space inequality \(\|u+v\|^2\leq2\|u\|^2+2\|v\|^2\) gives \(\|S\hat g-f_*\|^2\leq2\|S\hat g-Sg_\lambda\|^2+2\|Sg_\lambda-f_*\|^2\). Taking expectations over the sample proves the first inequality. The population comparison already proved gives \(\|Sg_\lambda-f_*\|^2\leq A_\lambda(f_*)\), proving the second. \(\square\)

The first term is an estimation discrepancy; the second is population approximation error, including regularization. This inequality does not itself give a rate for the estimation term. Such rates require additional assumptions about the noise and the sample, which are distinct from the approximation question addressed here.

Approximation Floors and Targets Outside the Range

Let \(\overline{S(\mathcal H)}\) denote the closure of the kernel range in \(L_2(P_X)\), and let \(d=\operatorname{dist}(f_*,\overline{S(\mathcal H)})\). The limiting behavior of \(A_\lambda\) has a simple interpretation: only the component orthogonal to the closure of the range is permanently unapproximable.

Proof. For any \(g\in\mathcal H\), the squared residual is at least \(d^2\), so \(A_\lambda(f_*)\geq d^2\). For the reverse limit, let \(\epsilon>0\). By the definition of closure and distance, choose a fixed \(g_\epsilon\in\mathcal H\) such that \(\|f_*-Sg_\epsilon\|<d+\epsilon\). Consequently, \(A_\lambda(f_*)\leq(d+\epsilon)^2+\lambda\|g_\epsilon\|_{\mathcal H}^2\). Taking the upper limit as \(\lambda\downarrow0\) gives at most \((d+\epsilon)^2\). Letting \(\epsilon\downarrow0\) and combining with the lower bound proves the result. \(\square\)

Common Pitfall and Practical Interpretation

A common mistake is to label every target outside the RKHS as having a positive, irreducible approximation error. That conclusion is false when the range is not closed. Outside the range means that no finite-norm RKHS element represents the target exactly; it does not mean that the target is separated by positive \(L_2(P_X)\) distance from all kernel predictions. The relevant floor is the distance to the closure of the range.

A second mistake is to treat a decreasing approximation residual as if it were a complete risk bound. A sequence of increasingly accurate approximants may have rapidly growing RKHS norms, and finite-sample estimation error may also increase as regularization weakens. The functional \(A_\lambda(f_*)\) makes the first tradeoff explicit; the estimation term must still be controlled separately. The key question is therefore not simply whether the target belongs to the kernel range, but how residual size, RKHS norm, and sample estimation behave together at the chosen \(\lambda\).