Source Conditions in the Covariance Scale

Why Describe a Regression Target on the Covariance Scale?

Kernel methods analyze functions through directions that the input distribution and kernel make visible. Some directions correspond to large eigenvalues of the kernel operator; others have very small eigenvalues. A source condition describes how much of the regression target lies in each direction by requiring the target to be a fractional power of that operator applied to a square-integrable function.

The exponent in this representation is a smoothness index relative to the covariance operator. Increasing it requires the target's coefficients in small-eigenvalue directions to decay more quickly. This is a structural assumption on the target, not a consequence of using a kernel or fitting a kernel ridge estimator.

The Covariance-Scale Source Condition

Let \(X\) have distribution \(P_X\), and suppose the kernel \(k\) is measurable and positive semidefinite, with \(E[k(X,X)]<\infty\). Work in the real Hilbert space \(L^2(P_X)\), whose elements are square-integrable functions identified when they agree \(P_X\)-almost surely. The kernel integral operator \(T:L^2(P_X)\to L^2(P_X)\) is

Under these assumptions, \(T\) is a bounded, positive, self-adjoint, compact operator. The terminology “covariance scale” refers to this operator, which is also called the kernel integral operator. It is an uncentered operator: the definition does not subtract the mean of the kernel feature map.

To make the definition explicit, write the positive eigenvalues of \(T\), repeated with multiplicity, as \(\mu_j>0\), with corresponding orthonormal eigenfunctions \(\phi_j\). The remaining subspace is \(\ker(T)\), the set of functions mapped to zero by \(T\). For \(r>0\),

Thus \(T^r\) annihilates the component of \(h\) in \(\ker(T)\). For \(r=0\), we use \(T^0=I\), the identity operator, so the order-zero condition allows every \(f_*\in L^2(P_X)\).

What the Exponent Says About Coefficients

Proof. Suppose first that \(f_*=T^r g\). Let \(g_j=\langle g,\phi_j\rangle_{L^2(P_X)}\). By the definition of the fractional power, \(f_j=\mu_j^r g_j\) for every positive eigenvalue. Also, \(T^r g\) has no component in \(\ker(T)\). Therefore \(f_*\perp\ker(T)\), and

Conversely, suppose the stated orthogonality and summability conditions hold. Define \(g\) by its positive-eigenvalue coefficients \(g_j=f_j/\mu_j^r\), and give it zero coefficient in \(\ker(T)\). The sum condition says these coefficients are square-summable, so they define an element of \(L^2(P_X)\). Its image under \(T^r\) has coefficients \(\mu_j^r g_j=f_j\) in every positive-eigenvalue direction and zero coefficient in the null space. These are exactly the coefficients of \(f_*\), so \(T^r g=f_*\). Any other source element can differ from this one only by a component in \(\ker(T)\), which \(T^r\) removes. Such a component cannot reduce the norm. The constructed source element therefore has minimum norm, equal to the square root of the displayed sum. \(\square\)

This characterization makes the smoothness interpretation precise. For the same target coefficient \(f_j\), dividing by \(\mu_j^r\) is more demanding when \(\mu_j\) is small and \(r\) is larger. A higher-order source condition requires enough decay of the target's coefficients along those small-eigenvalue directions to keep the weighted sum finite. It is smoothness relative to \(T\), not a claim about differentiability in an unrelated coordinate system.

How Source Order Relates to the RKHS

There is a useful connection with the RKHS of \(k\). Let \(J\) denote the inclusion map from that RKHS into \(L^2(P_X)\), and suppose the kernel assumptions above make this inclusion well-defined. Its associated operator on \(L^2(P_X)\) is \(JJ^*=T\). As \(L^2\) functions, the image of the RKHS under \(J\) is exactly the range of \(T^{1/2}\). In particular, a source condition of order \(1/2\) describes an RKHS target, viewed as an \(L^2\) equivalence class.

Justification. For a positive eigenvalue \(\mu_j\), define \(e_j=J^*\phi_j/\sqrt{\mu_j}\). The identity \(JJ^*=T\) gives \(\langle e_i,e_j\rangle_{\mathcal H}=1\) when \(i=j\) and \(0\) otherwise. If \(h\in\mathcal H\), then the coefficient of \(Jh\) along \(\phi_j\) is

Consequently, the squared \(L^2\) coefficients of \(Jh\), divided by \(\mu_j\) and summed, are bounded by \(\|h\|_{\mathcal H}^2\). Conversely, if \(f\) has coefficients satisfying \(\sum_j f_j^2/\mu_j<\infty\) and has no null-space component, the series \(h=\sum_j (f_j/\sqrt{\mu_j})e_j\) converges in \(\mathcal H\), and the coefficient identity gives \(Jh=f\). The squared norm of this constructed \(h\) is \(\sum_j f_j^2/\mu_j\). The coefficient characterization of the order-\(1/2\) source condition gives the same sum and the same minimum source norm, proving the result. \(\square\)

Nested Source Classes and the Meaning of Smoothness

Higher source orders are nested inside lower ones. This follows from an operator identity and does not require a particular eigenvalue decay rate.

Proof. When \(r=s\), the inclusion is equality. When \(r<s\), take any \(f\in\operatorname{Ran}(T^s)\), so \(f=T^s g\) for some \(g\in L^2(P_X)\). Fractional powers of the same positive operator satisfy \(T^s=T^rT^{s-r}\), as follows by multiplying their eigenvalue powers in each spectral direction. Since \(T\) is bounded, \(T^{s-r}g\in L^2(P_X)\). Therefore

This proves the inclusion. \(\square\)

The inclusion says that every target satisfying a higher-order source condition also satisfies each lower-order one. The reverse inclusion need not hold, as the preceding infinite-series example demonstrates. Nor does the numerical value of \(r\), by itself, assert a universal amount of ordinary differentiability: it measures regularity against the eigenstructure induced jointly by the kernel and \(P_X\).

Common Pitfall

A common mistake is to treat a source condition as a property of the kernel alone: “this kernel is smooth, so the regression target has a high source order.” The condition is about the target. A kernel determines \(T\) and its eigen-directions, but the target must also have coefficients that satisfy the required weighted summability condition. A second mistake is to ignore \(\ker(T)\). For \(r>0\), a component in that null space cannot be produced by \(T^r g\), regardless of how \(g\) is chosen. Check both the coefficient sum and the null-space component before asserting source membership.