Reproducing Kernel Hilbert Spaces and Evaluation Functionals

From Function Values to Hilbert-Space Representers

A function space used for statistical learning has two structures: its elements are functions, and its inner product measures how those functions relate geometrically. A key question is whether asking for a function’s value at an input is compatible with that geometry. If evaluation at each input is a bounded linear functional, the Riesz representation theorem supplies a particular function in the space that represents evaluation. These representing functions are the kernel sections of a reproducing kernel Hilbert space.

This connection is more than notation. It turns a pointwise prediction into an inner product with a fixed element of the function space. The definition below makes explicit which property is required and what the reproducing property means.

Definition: Let $\mathcal X$ be a set, and let $\mathcal H$ be a real Hilbert space whose elements are functions $f:\mathcal X\to\mathbb R$. For $x\in\mathcal X$, the evaluation functional at $x$ is the linear map $\delta_x:\mathcal H\to\mathbb R$ given by $\delta_x(f)=f(x)$. It is bounded if there is a finite constant $C_x$ such that $|f(x)|\le C_x\|f\|_{\mathcal H}$ for every $f\in\mathcal H$. The space $\mathcal H$ is a reproducing kernel Hilbert space (RKHS) on $\mathcal X$ if every evaluation functional $\delta_x$ is bounded.

The constant in the boundedness condition may depend on $x$. The condition is required at every point of the domain, but it does not assert that one common constant works for all points. The fact that $\mathcal H$ is a Hilbert space is also essential: boundedness of a linear functional alone does not guarantee a representer inside an arbitrary normed space.

The Reproducing Property

The Riesz representation theorem states that, for a real Hilbert space $\mathcal H$, every bounded linear functional $L:\mathcal H\to\mathbb R$ has a unique element $h_L\in\mathcal H$ such that $L(f)=\langle f,h_L\rangle_{\mathcal H}$ for every $f\in\mathcal H$. Apply this theorem to evaluation at each $x$. The resulting element is denoted $K(\cdot,x)$, or $k_x$, and is called the kernel section at $x$.

Theorem: If $\mathcal H$ is an RKHS on $\mathcal X$, then for each $x\in\mathcal X$ there is a unique $k_x\in\mathcal H$ such that $f(x)=\langle f,k_x\rangle_{\mathcal H}$ for every $f\in\mathcal H$. Define $K(x,y)=k_y(x)$. Then $K(\cdot,y)=k_y$, and $$ f(x)=\langle f,K(\cdot,x)\rangle_{\mathcal H} $$ for all $f\in\mathcal H$ and $x\in\mathcal X$. This identity is the reproducing property.

The theorem follows point by point from Riesz representation. For fixed $x$, evaluation $f\mapsto f(x)$ is linear because $(af+bg)(x)=af(x)+bg(x)$. It is bounded by the definition of an RKHS. Riesz representation therefore gives a unique $k_x\in\mathcal H$ with $f(x)=\langle f,k_x\rangle_{\mathcal H}$ for every $f$. Now define $K(x,y)=k_y(x)$. As a function of $x$, $K(x,y)$ is exactly $k_y$, so $K(\cdot,y)=k_y\in\mathcal H$. Substituting $x$ for the evaluation point gives the stated reproducing identity.

This definition of $K$ also ensures that it is a symmetric positive definite kernel. Symmetry follows from the reproducing identity and symmetry of the real inner product:

K(x,y)=k_y(x)=\langle k_y,k_x\rangle_{\mathcal H} =\langle k_x,k_y\rangle_{\mathcal H}=k_x(y)=K(y,x).

For positive definiteness, take any integer $m\ge 1$, points $x_1,\ldots,x_m\in\mathcal X$, and coefficients $a_1,\ldots,a_m\in\mathbb R$. Then

\begin{aligned} \sum_{i=1}^m\sum_{j=1}^m a_i a_j K(x_i,x_j) &=\sum_{i=1}^m\sum_{j=1}^m a_i a_j \langle k_{x_j},k_{x_i}\rangle_{\mathcal H}\\ &=\left\langle\sum_{j=1}^m a_j k_{x_j}, \sum_{i=1}^m a_i k_{x_i}\right\rangle_{\mathcal H}\\ &=\left\|\sum_{i=1}^m a_i k_{x_i}\right\|_{\mathcal H}^2 \ge 0. \end{aligned}

Thus the matrix with entries $K(x_i,x_j)$ is positive semidefinite for every finite collection of inputs, in agreement with the usual definition of a positive definite kernel. This result does not require the kernel sections to be linearly independent; the displayed quadratic form can be zero for nonzero coefficients.

Uniqueness: For a fixed Hilbert space of functions with its fixed inner product, the reproducing kernel is unique. At each $x$, its section must be the unique Riesz representer of $\delta_x$.

To verify uniqueness directly, suppose $u_x,v_x\in\mathcal H$ both represent evaluation at $x$. Then $\langle f,u_x-v_x\rangle_{\mathcal H}=0$ for every $f\in\mathcal H$. Taking $f=u_x-v_x$ yields $\|u_x-v_x\|_{\mathcal H}^2=0$, so $u_x=v_x$. The sections, and hence the kernel values, are therefore determined by the space and inner product.

Worked Examples

Worked Example: A Weighted Function Space on Two Points

Let $\mathcal X=\{a,b\}$, and let $\mathcal H$ consist of all real-valued functions on $\mathcal X$, with inner product $\langle f,g\rangle_{\mathcal H}=f(a)g(a)+4f(b)g(b)$. Find the kernel sections and check reproduction.

Write a function as the pair $(f(a),f(b))$. For evaluation at $a$, seek a section $k_a$ such that $\langle f,k_a\rangle_{\mathcal H}=f(a)$ for every $f$. If $k_a=(u,v)$, then $\langle f,k_a\rangle=f(a)u+4f(b)v$. Matching the coefficients of $f(a)$ and $f(b)$ gives $u=1$ and $v=0$. Thus $k_a=(1,0)$.

For evaluation at $b$, the same coefficient matching requires $u=0$ and $4v=1$, so $k_b=(0,1/4)$. Consequently the kernel values are $K(a,a)=1$, $K(b,a)=0$, $K(a,b)=0$, and $K(b,b)=1/4$. For $f=(r,s)$, the reproducing checks are $\langle f,k_a\rangle=r=f(a)$ and $\langle f,k_b\rangle=4s(1/4)=s=f(b)$.

Worked Example: Affine Functions on the Unit Interval

Let $\mathcal X=[0,1]$, and let $\mathcal H=\{f(t)=a+bt:a,b\in\mathbb R\}$ with $\langle a+bt,c+dt\rangle_{\mathcal H}=ac+bd$. Find the section at $x$ and verify the kernel formula.

For fixed $x\in[0,1]$, the function $k_x(t)=1+xt$ belongs to $\mathcal H$. If $f(t)=a+bt$, then $\langle f,k_x\rangle_{\mathcal H}=a\cdot1+b\cdot x=a+bx=f(x)$. Thus $k_x$ represents evaluation at $x$, and the kernel is $K(t,x)=1+tx$.

For a numerical check, take $f(t)=2-3t$ and $x=1/2$. Direct evaluation gives $f(1/2)=2-3/2=1/2$. The inner product with the section $k_{1/2}(t)=1+t/2$ gives $\langle f,k_{1/2}\rangle=2\cdot1+(-3)\cdot(1/2)=1/2$, as required.

Worked Example: Bounded Evaluation Does Not Replace the Hilbert Requirement

Consider the continuous functions on $[0,1]$ with the supremum norm $\|f\|_\infty=\sup_{t\in[0,1]}|f(t)|$. Each evaluation is bounded, since for every $x\in[0,1]$, $|f(x)|\le\|f\|_\infty$. But this norm does not come from an inner product.

To check the latter claim, take $f(t)=1$ and $g(t)=t$. Then $\|f\|_\infty=\|g\|_\infty=1$, $\|f+g\|_\infty=2$, and $\|f-g\|_\infty=1$. The parallelogram identity required of every inner-product norm would give $\|f+g\|_\infty^2+\|f-g\|_\infty^2=2\|f\|_\infty^2+2\|g\|_\infty^2$. Here the left side is $4+1=5$, while the right side is $2+2=4$. So this norm is not an inner-product norm, and the space with this norm is not an RKHS. Bounded evaluations are necessary, but the Hilbert-space structure is also required.

Why the Conditions Matter

The reproducing property expresses a prediction as an inner product with a section fixed by the input: $f(x)=\langle f,K(\cdot,x)\rangle_{\mathcal H}$. It identifies the geometric object associated with asking for a function’s value at $x$. That is why RKHS methods can work with functions through kernel evaluations while retaining an underlying Hilbert-space interpretation.

A common mistake is to treat a formula resembling an $L^2$ inner product as sufficient to make point evaluation bounded. For example, on continuous functions on $[0,1]$, use the inner product $\langle f,g\rangle=\int_0^1 f(t)g(t)\,dt$. For a positive integer $m$, define $f_m(t)=\sqrt{3m}(1-mt)$ on $0\le t\le 1/m$, and $f_m(t)=0$ on $1/m<t\le1$. Each $f_m$ is continuous and

\|f_m\|_2^2 =3m\int_0^{1/m}(1-mt)^2\,dt =3\int_0^1(1-u)^2\,du =1, \qquad f_m(0)=\sqrt{3m}.

The substitution $u=mt$ gives the second equality, and $\int_0^1(1-u)^2\,du=1/3$ gives the last norm calculation. The functions have unit norm while their values at zero grow without bound. Thus evaluation at zero is not bounded in this inner-product geometry, so this space is not an RKHS. The correction to the mistaken reasoning is to check the evaluation inequality itself, not to infer it merely from having an inner product.

Key takeaway: An RKHS is a Hilbert space of actual functions in which every point evaluation is bounded. Riesz representation then gives a unique section for each input, and the inner product with that section reproduces the function’s value.

Check Your Understanding

Answer each question using the definitions and results above.

What two conditions distinguish an RKHS from a general function space?
For fixed $x$, what property characterizes the section $K(\cdot,x)$ uniquely?
Why is the quadratic form associated with a finite Gram matrix nonnegative?
In the two-point example, why is the section at $b$ equal to $(0,1/4)$, rather than $(0,1)$?
What does the unit-norm sequence of continuous functions show about evaluation at zero under the integral inner product?

Reproducing Kernel Hilbert Spaces and Evaluation Functionals

What You'll Learn