Source-Exponent Conventions and Smoothness Pitfalls

Why Source Exponents Need a Convention

A source condition describes how a target aligns with the eigendirections of a covariance operator. The exponent is not meaningful by itself: it must be attached to a specified operator and a specified object. Kernel ridge regression makes this especially important because there are two natural covariance operators, one on the RKHS and one on the space of prediction functions. Their nonzero eigenvalues agree, but the corresponding target coordinates differ by a square root of an eigenvalue. That changes the source exponent.

The practical consequence is direct: a bias rate derived under one convention cannot be applied unchanged to an exponent stated under the other. We will define both conventions, prove their conversion, and calculate what happens when the conversion is omitted.

The Two Covariance Operators

Let \(P_X\) be the distribution of the input \(X\), let \(\mathcal H\) be a reproducing kernel Hilbert space, and let \(L^2(P_X)\) be the space of square-integrable prediction functions. Assume the inclusion operator \(S:\mathcal H\to L^2(P_X)\), defined by \(Sf=f\), is bounded. Its adjoint is \(S^*\). Define the RKHS covariance operator \(C\) and prediction-space operator \(T\) by

C=S^*S:\mathcal H\to\mathcal H, \qquad T=SS^*:L^2(P_X)\to L^2(P_X).

Both are positive self-adjoint operators. When the kernel has finite expected diagonal, they are also compact, and their nonzero eigenvalues agree, including multiplicities. For a kernel \(k\), \(C\) is commonly written \(C=E[k_X\otimes k_X]\), where \(k_X=k(X,\cdot)\). The operator \(T\) acts on prediction functions; in integral form, under the usual integrability conditions, \((Tg)(x)=\int k(x,x')g(x')\,dP_X(x')\).

Definition: A prediction-space source condition with exponent \(r\geq 0\) writes the target prediction function as \(g=T^r h\), for some \(h\in L^2(P_X)\). An RKHS source condition with exponent \(s\geq 0\) writes its RKHS representer as \(f_*=C^s v\), for some \(v\in\mathcal H\). These are distinct conditions: one applies a power of \(T\) to a prediction function, and the other applies a power of \(C\) to an RKHS element.

Here, a representer \(f_*\) means \(Sf_*=g\). If \(g\) is in the range of \(S\), its minimum-norm representer is unique in \((\ker S)^\perp\). The source-coordinate comparison below concerns this representer. Fractional operator powers are defined by the spectral theorem.

Converting Between the Conventions

The polar decomposition of \(S\) gives \(S=UC^{1/2}\), where \(U\) is a partial isometry from the closure of the range of \(C\) onto the closure of the range of \(T\). It also gives \(T=UCU^*\) on the relevant subspaces. Thus applying \(C^s\) to an RKHS element and then mapping it to prediction space adds one-half to the exponent.

Theorem: If \(f_*=C^s v\) for \(s\geq 0\), then \[ Sf_*=T^{s+1/2}Uv. \] In particular, an RKHS source exponent \(s\) corresponds to the prediction-space exponent \(r=s+1/2\) for this representer.

Proof. From \(S=UC^{1/2}\),

Sf_*=SC^s v =UC^{s+1/2}v.

The polar decomposition intertwines the positive powers of \(C\) and \(T\): \(UC^a=T^aU\) for \(a\geq 0\) on the relevant subspaces. This follows first for integer powers from \(UC=TU\), then for bounded continuous functions of the operators by the spectral calculus; fractional powers follow by applying that calculus on the positive spectral subspaces. Taking \(a=s+1/2\) gives

UC^{s+1/2}v=T^{s+1/2}Uv,

as claimed. \(\square\)

The relation is one-way without additional range conditions. For \(r\geq 1/2\), a prediction-space source \(g=T^r h\) can be represented by an RKHS source with exponent \(s=r-1/2\), using the corresponding partial-isometry coordinates. For \(r<1/2\), a prediction-space source condition alone does not guarantee that \(g\) has an RKHS representer.

Worked Example: Translating One Eigen-Coordinate

Suppose \(C e=\mu e\), with \(\mu=0.04\), and take \(v=e\). Under the RKHS source condition with \(s=1/2\), the representer is

f_*=C^{1/2}e=\sqrt{0.04}\,e=0.2e.

Since \(Se\) is a prediction-space eigenfunction of \(T\) with eigenvalue \(0.04\), the prediction function has coordinate \(\sqrt{\mu}\times 0.2=0.04\) along the normalized prediction eigenfunction. Equivalently,

Sf_*=T^{1}Ue.

So the prediction-space exponent is \(r=s+1/2=1\), not \(1/2\). Calling this target \(T^{1/2}\)-smooth would assign an eigen-coordinate \(0.04^{1/2}=0.2\), five times the actual coordinate \(0.04\).

Bias Rates Under the Prediction-Space Convention

Consider population kernel ridge regression for a target prediction function \(g\in L^2(P_X)\). Its population solution minimizes \(E[(f(X)-g(X))^2]+\lambda\|f\|_{\mathcal H}^2\) over \(f\in\mathcal H\), with \(\lambda>0\). The normal equation is \((C+\lambda I)f_\lambda=S^*g\). Applying \(S\) to the solution yields

Sf_\lambda=T(T+\lambda I)^{-1}g.

Therefore the prediction bias, defined as \(g-Sf_\lambda\), is \(\lambda(T+\lambda I)^{-1}g\). If \(g=T^r h\), its norm is controlled by a scalar spectral multiplier. For the range \(0<r\leq 1\), the following bound makes the exponent explicit.

Theorem: If \(g=T^r h\), \(h\in L^2(P_X)\), and \(0<r\leq 1\), then \[ \|g-Sf_\lambda\|_{L^2(P_X)} \leq c_r\lambda^r\|h\|_{L^2(P_X)}, \] where \(c_r=\sup_{x\geq 0}x^r/(1+x)\). For \(0<r<1\), \(c_r=r^r(1-r)^{1-r}\); for \(r=1\), \(c_1=1\).

Proof. By the spectral theorem, the operator \(\lambda(T+\lambda I)^{-1}T^r\) has spectral multiplier \(\lambda t^r/(t+\lambda)\) at eigenvalue \(t\geq 0\). With \(x=t/\lambda\), this multiplier is \(\lambda^r x^r/(1+x)\). Its supremum over the spectrum is no greater than its supremum over all \(x\geq0\). Hence

\|g-Sf_\lambda\|_{L^2(P_X)} \leq \lambda^r\sup_{x\geq0}\frac{x^r}{1+x}\,\|h\|_{L^2(P_X)}.

For \(0<r<1\), differentiating \(x^r/(1+x)\) for \(x>0\) gives derivative \(x^{r-1}[r-(1-r)x]/(1+x)^2\). Its maximum occurs at \(x=r/(1-r)\), where the value is \(r^r(1-r)^{1-r}\). For \(r=1\), \(x/(1+x)\leq1\). This proves the bound. \(\square\)

If instead the representer satisfies \(f_*=C^s v\), the conversion theorem gives prediction-space exponent \(s+1/2\). Thus, when \(0\leq s\leq 1/2\), the prediction bias bound has power \(\lambda^{s+1/2}\), not \(\lambda^s\). This conclusion is about prediction error; other norms may have different bias powers.

Worked Example: Computing a Prediction Bias

Let \(T\) have a normalized eigenfunction \(\phi\) with eigenvalue \(\mu=0.25\). Set \(r=1/2\), \(h=2\phi\), so that \(g=T^{1/2}h=2\sqrt{0.25}\phi=\phi\). At \(\lambda=0.05\), the prediction bias is

g-Sf_\lambda =\frac{\lambda}{\mu+\lambda}\phi =\frac{0.05}{0.25+0.05}\phi =\frac{1}{6}\phi.

Its norm is \(1/6\). The theorem’s bound gives \(c_{1/2}\lambda^{1/2}\|h\|=0.5\sqrt{0.05}\times2\), approximately \(0.224\), which is greater than \(1/6\). The calculation checks both the eigen-coordinate shrinkage and the exponent convention.

Worked Example: How a Half-Power Changes a Rate

Suppose the RKHS source exponent is \(s=1/4\), and its representer is \(f_*=C^{1/4}v\). The corresponding prediction-space exponent is \(r=3/4\). For \(\lambda=0.01\), the prediction-bias bound scales as \(\lambda^{3/4}\), which is approximately \(0.0316\), times the source norm and constant \(c_{3/4}\).

If one incorrectly carries the RKHS exponent across unchanged and claims a prediction-bias power \(\lambda^{1/4}\), the displayed scale is \(0.01^{1/4}\), approximately \(0.3162\). That is a different, looser power, not the rate supplied by the prediction-space source condition. The error is not in exponent arithmetic: it is treating \(C^{1/4}\) on an RKHS representer as though it were \(T^{1/4}\) on the prediction function.

A Reliable Convention Check

A rate statement should identify four things: the target object, the operator acting on it, the norm in which error is measured, and the parameter range for the bound. For example, “\(g=T^r h\) in \(L^2(P_X)\), with \(0<r\leq1\), gives prediction bias of order \(\lambda^r\)” is interpretable. “The target is \(r\)-smooth, so the bias is \(\lambda^r\)” is not, unless the convention and norm have already been fixed.

The square-root shift comes from mapping an RKHS vector into prediction space: the map \(S\) itself contributes \(C^{1/2}\). It does not mean that every smoothness notion in kernel learning should be shifted mechanically. The shift applies to these specific source conditions and this prediction-error calculation.

Common Mistake

Wrong reasoning: “The representer satisfies \(f_*=C^s v\), so the prediction function satisfies \(g=T^s h\); therefore its prediction bias is of order \(\lambda^s\).” This drops the \(C^{1/2}\) in \(S=UC^{1/2}\).
Correction: Convert the representer first: \(Sf_*=T^{s+1/2}Uv\). Then use the bias bound for that prediction-space exponent, provided it lies in the stated range \(0< s+1/2\leq1\).

Key takeaway: Always attach a source exponent to its operator and target. For the RKHS and prediction-space conventions above, \(f_*=C^s v\) maps to \(g=T^{s+1/2}Uv\); confusing the two exponents can produce an incorrect prediction-bias rate.

Check Your Understanding

For each question, keep the operator and the target object explicit.

If \(f_*=C^{0.2}v\), what is the corresponding prediction-space exponent for \(Sf_*\)?
State the prediction-bias bound when \(g=T^{0.6}h\), including the norm and the condition on the exponent.
Why does a prediction-space source condition with exponent \(r<1/2\) not by itself guarantee an RKHS representer?
For an eigenvalue \(\mu\), what multiplier maps the target coordinate to its population kernel ridge prediction bias?
Name the four items that should be specified when interpreting a source-based bias rate.