RKHS Norms in Mercer Coordinates

From Kernel Expansions to Function Coordinates

A Mercer expansion expresses a kernel through eigenfunctions of its integral operator. That expansion also gives coordinates for the functions in the associated reproducing kernel Hilbert space (RKHS). The important feature is that the squared RKHS norm does not simply sum squared coefficients: each coefficient is weighted by the reciprocal of its eigenvalue. Directions with small positive eigenvalues are therefore expensive in RKHS norm.

We work in the usual Mercer setting: \(\mathcal X\) is a compact metric space, \(\mu\) is a finite Borel measure with support \(\mathcal X\), and \(k\) is a continuous, symmetric, positive semidefinite kernel. Write the positive eigenvalues of its integral operator, with multiplicity, as \(\lambda_j>0\), and choose corresponding continuous, \(L^2(\mu)\)-orthonormal eigenfunctions \(\phi_j\). The index \(j\) runs over a finite or countable set. Mercer’s theorem gives

with uniform convergence on \(\mathcal X\times\mathcal X\). The list here contains only positive eigenvalues. The integral operator may also have a nontrivial zero-eigenvalue space in \(L^2(\mu)\); those zero-eigenvalue directions are not additional coordinates in the RKHS expansion.

Why the Eigenvalue Weights Appear

The kernel expansion can be read as an inner product of feature vectors. Define \(\Phi(x)=(\sqrt{\lambda_j}\phi_j(x))_j\), an element of the sequence space \(\ell^2\). Indeed, \(\|\Phi(x)\|_{\ell^2}^2=k(x,x)<\infty\), and

A vector \(w=(w_j)_j\in\ell^2\) therefore defines a function \(f_w(x)=\langle w,\Phi(x)\rangle_{\ell^2}\). Its eigenfunction coefficient is \(a_j=\sqrt{\lambda_j}w_j\). Consequently, the squared length of \(w\) is

This feature-space calculation suggests the norm formula. The next result establishes both directions of the characterization, including convergence and uniqueness.

Proof. The Mercer expansion gives the feature map \(\Phi\) above. The RKHS construction identifies its elements with functions \(f_w(x)=\langle w,\Phi(x)\rangle_{\ell^2}\), using vectors in the closed linear span of \(\{\Phi(x):x\in\mathcal X\}\). This closed span is all of \(\ell^2\): if \(w\) is orthogonal to every \(\Phi(x)\), then \(f_w(x)=0\) for all \(x\). The series defining \(f_w\) converges uniformly, since for any tail of indices \(J\),

The second factor is bounded by \(\sup_x\sqrt{k(x,x)}<\infty\), and the first tends to zero as the tail recedes. Thus \(f_w\) is continuous and, since \(\mu\) is finite, lies in \(L^2(\mu)\). Integrating its uniformly convergent series against \(\phi_r\) gives \(\int f_w\phi_r\,d\mu=\sqrt{\lambda_r}w_r\), by orthonormality. If \(f_w\) is identically zero, every such integral is zero, so \(w_r=0\) for every \(r\). The vectors representing functions are therefore unique, and the closed span is all of \(\ell^2\).

For any \(w\in\ell^2\), set \(a_j=\sqrt{\lambda_j}w_j\). The function has the asserted series, \(\sum_j a_j^2/\lambda_j=\sum_j w_j^2<\infty\), and its RKHS norm squared is \(\|w\|_{\ell^2}^2\). Conversely, if \(\sum_j a_j^2/\lambda_j<\infty\), take \(w_j=a_j/\sqrt{\lambda_j}\). Then \(w\in\ell^2\), so \(f_w\in\mathcal H\) and has exactly the proposed series and norm. Finally, integrating that series against each \(\phi_r\) recovers \(a_r\), which proves uniqueness. This establishes the characterization. \(\square\)

Convergence and the Role of the Diagonal

For a coefficient sequence with finite weighted sum, convergence is stronger than convergence in \(L^2(\mu)\): the function series converges uniformly. To see this directly, let \(J\) be a tail of the indices. Cauchy–Schwarz gives

The first factor tends to zero because the weighted coefficient series is summable. The square of the second factor is bounded by \(\sup_x k(x,x)\), which is finite. Hence the tails tend uniformly to zero. In fact, uniform convergence of the Mercer expansion on the diagonal implies that its diagonal tails tend uniformly to zero as well.

The same reasoning shows that the partial sums converge in RKHS norm: the squared norm of the omitted part is exactly \(\sum_{j\in J}a_j^2/\lambda_j\), which tends to zero. Thus the coordinate description controls both the function values and the Hilbert-space approximation.

Zero Eigenvalues Are Not Extra RKHS Coordinates

The integral operator \(T_k g(x)=\int k(x,y)g(y)\,d\mu(y)\) can have a nullspace. A function in this nullspace is an \(L^2(\mu)\) function \(g\) with \(T_k g=0\); it is not a new Mercer coordinate to which one can assign an RKHS coefficient and divide by an eigenvalue of zero. The RKHS expansion uses only the positive-eigenvalue eigenfunctions. In particular, a nonzero \(L^2\) function can lie in the nullspace without belonging to \(\mathcal H\).

Infinite Expansions and a Norm Calculation

Common Mistake: Using Unweighted Coefficients

A frequent error is to conclude that a function belongs to the RKHS whenever its eigenfunction coefficients are square-summable, or to identify the RKHS norm with the ordinary \(L^2\) norm. The required condition is instead \(\sum_j a_j^2/\lambda_j<\infty\). For example, in the trigonometric kernel above, take \(a_{m,c}=2^{-m}\) and all other coefficients zero. The coefficients are square-summable because \(\sum_{m\geq1}4^{-m}=1/3\), but the proposed RKHS squared norm is \(\sum_{m\geq1}4^{-m}/4^{-m}=\sum_{m\geq1}1=\infty\). So square-summability alone does not establish membership in this RKHS.

The reciprocal eigenvalue weight explains the mistake: small eigenvalues penalize their coordinates more heavily. Two functions can have similar \(L^2\) size but very different RKHS norms if one places more coefficient mass in low-eigenvalue directions. The coordinate formula makes that distinction precise without confusing the RKHS with the full space \(L^2(\mu)\).