Representers for Derivative and Integral Data

Why Represent Derivative and Integral Data?

Many learning problems do not observe a function only through its values at selected points. A sensor may report a local slope, an experiment may report a weighted average, or a constraint may specify the total amount of a function over an interval. These are all linear observations of the unknown function. In a reproducing kernel Hilbert space (RKHS), each bounded linear observation has a representer: a particular function that converts the observation into an inner product.

This matters because a finite collection of such observations still leads to a finite-dimensional span in which to search for a minimum-norm fit. But the representer is not automatically the ordinary kernel section $k(\cdot,x)$. Its form depends on the observation, and derivative measurements require genuine smoothness conditions.

Bounded Linear Observations and Their Representers

Definition: Let $\mathcal H$ be a real RKHS with inner product $\langle\cdot,\cdot\rangle_{\mathcal H}$. A bounded linear observation is a linear functional $L:\mathcal H\to\mathbb R$ for which there is a finite constant $C$ such that $|Lf|\le C\|f\|_{\mathcal H}$ for every $f\in\mathcal H$. Its representer is the unique $r_L\in\mathcal H$ satisfying $$ Lf=\langle f,r_L\rangle_{\mathcal H}\qquad\text{for every }f\in\mathcal H. $$

The existence and uniqueness of $r_L$ follow from the Riesz representation theorem for Hilbert spaces. For point evaluation, $L_xf=f(x)$, the RKHS reproducing property says that the representer is $k(\cdot,x)$. For a different observation, the same principle applies, but its representer must be derived.

For finitely many bounded observations $L_1,\ldots,L_m$, write $r_j$ for the representer of $L_j$. Their observation Gram matrix is $$ G_{ij}=L_i r_j=\langle r_j,r_i\rangle_{\mathcal H},\qquad 1\le i,j\le m. $$ It is positive semidefinite because, for any $a\in\mathbb R^m$, $$ a^\mathsf{T}Ga=\left\|\sum_{j=1}^m a_jr_j\right\|_{\mathcal H}^2\ge 0. $$ Thus the usual Gram-matrix construction extends directly from point evaluations to arbitrary bounded linear observations.

Derivative Measurements

A formal derivative of a kernel section is not enough by itself to guarantee that derivative evaluation is a bounded observation. The following condition supplies what is needed. Let the input domain be an open interval $D\subset\mathbb R$, and let $x\in D$. Suppose $k$ has a continuous mixed partial derivative $\partial_1\partial_2 k(s,t)$ on a neighborhood of $(x,x)$. Then the derivative observation $L_x f=f'(x)$ is bounded on the RKHS, and its representer is the derivative of the kernel section with respect to its second argument.

Theorem: Under the condition just stated, every $f\in\mathcal H$ is differentiable at $x$, and $$ r_x^{(1)}=\partial_2 k(\cdot,x),\qquad f'(x)=\langle f,r_x^{(1)}\rangle_{\mathcal H}. $$ For derivative observations at $x$ and $y$ where the same local conditions hold, $$ \langle r_x^{(1)},r_y^{(1)}\rangle_{\mathcal H} =\partial_1\partial_2 k(y,x). $$

To prove the theorem, let $k_x=k(\cdot,x)$, and for nonzero $h$ sufficiently small define the difference quotient $q_h=(k_{x+h}-k_x)/h$, an element of $\mathcal H$. The reproducing property gives $$ \langle q_h,q_g\rangle_{\mathcal H} =\frac{k(x+h,x+g)-k(x+h,x)-k(x,x+g)+k(x,x)}{hg}. $$ By applying the fundamental theorem of calculus in each coordinate, this is the average of $\partial_1\partial_2 k$ over the rectangle with corners $(x,x)$ and $(x+h,x+g)$. Continuity of the mixed partial therefore implies that $\langle q_h,q_g\rangle_{\mathcal H}\to c=\partial_1\partial_2k(x,x)$ as $h,g\to0$.

Consequently, $$ \|q_h-q_g\|_{\mathcal H}^2 =\|q_h\|_{\mathcal H}^2+\|q_g\|_{\mathcal H}^2-2\langle q_h,q_g\rangle_{\mathcal H}\longrightarrow c+c-2c=0. $$ So $q_h$ converges in the complete space $\mathcal H$ to some $r\in\mathcal H$. For any $f\in\mathcal H$, its difference quotient at $x$ is $$ \frac{f(x+h)-f(x)}{h}=\langle f,q_h\rangle_{\mathcal H}\longrightarrow\langle f,r\rangle_{\mathcal H}. $$ The scalar limit exists, so $f'(x)=\langle f,r\rangle_{\mathcal H}$. This proves boundedness, since $|f'(x)|\le\|f\|_{\mathcal H}\|r\|_{\mathcal H}$. Taking inner products with kernel sections identifies $r=\partial_2k(\cdot,x)$; applying the same derivative argument at $y$ gives the stated cross-inner-product formula. $\square$

The continuity assumption is local, but it is substantive: it makes the section difference quotients converge in RKHS norm, not merely pointwise. For a domain in higher dimensions, the corresponding argument applies to a partial derivative in a specified coordinate when the relevant mixed partial is continuous near the input pair.

Worked Example: Two Derivative Observations with a Gaussian Kernel

Consider $k(s,t)=\exp(-(s-t)^2/2)$ on $\mathbb R$. This kernel has continuous mixed partial derivatives, so derivative evaluation is bounded. Differentiating with respect to the second argument gives $$ r_a^{(1)}(t)=\partial_2 k(t,a)=(t-a)\exp(-(t-a)^2/2). $$ Differentiating in both arguments gives the Gram kernel for derivative observations: $$ \langle r_a^{(1)},r_b^{(1)}\rangle_{\mathcal H} =\bigl(1-(a-b)^2\bigr)\exp(-(a-b)^2/2). $$

Use observations $L_1f=f'(0)$ and $L_2f=f'(1/2)$. Each representer has squared norm $1$, while their inner product is $$ \rho=\left(1-\frac14\right)e^{-1/8}=\frac34e^{-1/8}. $$ Thus $G=\begin{pmatrix}1&\rho\\ \rho&1\end{pmatrix}$. Suppose the required derivative values are $1$ and $0$. A minimum-norm interpolant in the span of the representers has the form $f=c_1r_0^{(1)}+c_2r_{1/2}^{(1)}$. The constraints give $Gc=(1,0)^\mathsf{T}$, so $$ c_1=\frac{1}{1-\rho^2},\qquad c_2=-\frac{\rho}{1-\rho^2}. $$ The denominator is positive because $0<\rho<1$. Substitution gives $L_1f=c_1+\rho c_2=1$ and $L_2f=\rho c_1+c_2=0$, as required.

Integral Measurements

An integral observation can also be bounded. Let $D=[a,b]$ be compact, let $k$ be continuous on $D\times D$, and let $w\in L^1(D)$. Define $$ Lf=\int_a^b f(t)w(t)\,dt. $$ The continuous kernel makes every $f\in\mathcal H$ continuous. Moreover, the map $t\mapsto k(\cdot,t)$ is continuous in RKHS norm, since $$ \|k(\cdot,t)-k(\cdot,s)\|_{\mathcal H}^2=k(t,t)+k(s,s)-2k(t,s). $$ It is bounded on the compact interval, so the following integral exists as a Bochner integral in $\mathcal H$.

Theorem: Under these conditions, $L$ is bounded and has representer $$ r_L=\int_a^b k(\cdot,t)w(t)\,dt. $$ In particular, $|Lf|\le\|f\|_{\mathcal H}\int_a^b\sqrt{k(t,t)}\,|w(t)|\,dt$.

To verify the formula, take the inner product with any $f\in\mathcal H$. The defining property of the Bochner integral and the reproducing property yield $$ \langle f,r_L\rangle_{\mathcal H} =\int_a^b\langle f,k(\cdot,t)\rangle_{\mathcal H}w(t)\,dt =\int_a^b f(t)w(t)\,dt =Lf. $$ The bound follows from $|f(t)|\le\|f\|_{\mathcal H}\sqrt{k(t,t)}$ and integration of the absolute value. This proves both the representer identity and boundedness. $\square$

Worked Example: An Integral Representer for a Linear Kernel

On $[0,1]$, take $k(s,t)=1+st$ and the observation $Lf=\int_0^1 f(t)\,dt$. The RKHS consists of functions $f(t)=a+bt$, with inner product $\langle a+bt,c+dt\rangle_{\mathcal H}=ac+bd$. The kernel section at $t$, viewed as a function of $x$, is $1+xt$. Integrating these sections gives $$ r_L(x)=\int_0^1(1+xt)\,dt=1+\frac{x}{2}. $$ For $f(t)=a+bt$, the inner product with this representer is $$ \langle f,r_L\rangle_{\mathcal H} =\left\langle a+bt,1+\frac{x}{2}\right\rangle_{\mathcal H} =a+\frac b2 =\int_0^1(a+bt)\,dt. $$ Thus the representer identity holds for every function in this RKHS. Its squared norm is $1+(1/2)^2=5/4$, which also gives the exact squared operator norm of this nonzero functional.

Why the Representers Give a Finite-Dimensional Fit

For bounded observations $L_1,\ldots,L_m$, let $V=\operatorname{span}\{r_1,\ldots,r_m\}$. If a function $f$ satisfies prescribed observations $L_i f=y_i$, decompose it orthogonally as $f=v+u$, where $v\in V$ and $u\perp V$. Then $L_i u=\langle u,r_i\rangle_{\mathcal H}=0$, so $v$ satisfies the same observations. Also $\|f\|_{\mathcal H}^2=\|v\|_{\mathcal H}^2+\|u\|_{\mathcal H}^2$. Therefore, whenever the constraints are feasible, a minimum-norm interpolant lies in $V$.

Writing $v=\sum_jc_jr_j$ reduces the constraints to $Gc=y$. If the representers are linearly dependent, the coefficient vector need not be unique; what matters is the function in their span. Feasibility ensures that at least one such function exists, because the orthogonal projection preserves every observation.

Worked Example: A Derivative Representer for a Polynomial Kernel

Take $k(s,t)=(1+st)^2$ on $[-1,1]$. Expanding the kernel shows that its RKHS consists of $f(t)=c_0+c_1t+c_2t^2$, with squared norm $c_0^2+c_1^2/2+c_2^2$. The derivative representer at $a$ is $$ r_a^{(1)}(t)=\partial_2 k(t,a)=2t(1+at). $$ At $a=1/2$, this is $2t+t^2$. For $f(t)=1+3t+2t^2$, the ordinary derivative is $f'(t)=3+4t$, so $f'(1/2)=5$. The RKHS inner product confirms the result: $$ \langle f,r_{1/2}^{(1)}\rangle_{\mathcal H} =\frac{3\cdot2}{2}+2\cdot1=5. $$ The squared norm of the representer is $2^2/2+1^2=3$, which is the squared norm of derivative evaluation at $1/2$ as a bounded functional.

Common Pitfall and Practical Checks

A common mistake is to use $k(\cdot,x)$ as the representer for every observation associated with $x$. That section represents $f(x)$, not $f'(x)$ or an integral of $f$. For a derivative, differentiate the kernel in the observation’s argument and verify the regularity condition that makes derivative evaluation bounded. For an integral, integrate the kernel section against the same weight used in the observation.

A second check is to verify the representer identity on an arbitrary $f$, rather than relying on the appearance of a formula. For any proposed $r_L$, the defining test is $\langle f,r_L\rangle_{\mathcal H}=Lf$ for every $f\in\mathcal H$. For a collection of observations, also check that the resulting observation Gram matrix is positive semidefinite.

Key takeaway: Every bounded linear observation has a unique RKHS representer. Derivative observations require suitable mixed-partial smoothness of the kernel; integral observations are represented by weighted integrals of kernel sections under continuity and integrability conditions.

Check Your Understanding

Use the representer identity, rather than guessing from the observation’s name.

What condition on a linear functional guarantees that it has a representer in the RKHS?
For derivative evaluation, why is continuity of the mixed partial used in the difference-quotient proof?
State the representer for $Lf=\int_a^b f(t)w(t)\,dt$ under the conditions in this tutorial.
Why does projecting a feasible function onto the span of the observation representers preserve all its observed values?
For the polynomial kernel example, verify the derivative representer identity for $f(t)=1+3t+2t^2$ at $a=1/2$.