From a Kernel to an Operator
A kernel \(k(x,y)\) can be viewed not only as a rule for comparing two inputs, but also as a way to transform a function into another function. At each \(x\), the transformed value averages the input function \(f\) against the section \(k(x,\cdot)\). This construction is useful because operator properties such as boundedness, symmetry, and positivity describe how the kernel acts on whole functions, not just on finite collections of points.
Let \((\mathcal X,\mathcal A,P)\) be a probability space. Work with the real Hilbert space \(L^2(P)\), whose elements are equivalence classes of measurable functions with finite squared integral; functions equal \(P\)-almost everywhere represent the same element. Its inner product is \(\langle f,g\rangle_{L^2(P)}=\int f(x)g(x)\,dP(x)\). The operator is defined by integrating with respect to \(P\), so both its domain and its output space depend on that measure.
The square-integrability assumption is a convenient general condition: it ensures that almost every section \(k(x,\cdot)\) is in \(L^2(P)\), and it controls the operator norm. It does not by itself guarantee positivity. Symmetry of the kernel will give self-adjointness, while positivity requires an additional condition.
Boundedness and Self-Adjointness
Proof. Since \(\int\!\int |k(x,y)|^2\,dP(y)dP(x)<\infty\), Tonelli's theorem implies that \(\int |k(x,y)|^2\,dP(y)<\infty\) for almost every \(x\). For such \(x\), Cauchy–Schwarz gives $$ \left|\int k(x,y)f(y)\,dP(y)\right|^2 \leq \left(\int |k(x,y)|^2\,dP(y)\right)\|f\|_{L^2(P)}^2. $$ Integrating this inequality in \(x\) yields the asserted norm bound. It also shows that the integral defines an \(L^2(P)\) function. If two representatives of \(f\) differ only on a \(P\)-null set, their integrals agree for almost every \(x\), so the definition is independent of the representative. Linearity follows from linearity of the integral.
For self-adjointness, the double integral below is absolutely integrable: Cauchy–Schwarz on the product space bounds its absolute integral by \(\|k\|_{L^2(P\otimes P)}\|g\|_{L^2(P)}\|f\|_{L^2(P)}\). Fubini's theorem therefore applies. Using symmetry and exchanging the variables gives $$ \langle g,T_k f\rangle =\int\!\int g(x)k(x,y)f(y)\,dP(y)dP(x) =\int\!\int f(y)k(y,x)g(x)\,dP(x)dP(y) =\langle T_k g,f\rangle. $$ This proves both claims. \(\square\)
Worked Example: A Two-Point Space
Let \(\mathcal X=\{a,b\}\), with \(P(\{a\})=P(\{b\})=1/2\), and let the kernel matrix be $$ \begin{pmatrix} k(a,a)&k(a,b)\\ k(b,a)&k(b,b) \end{pmatrix} = \begin{pmatrix} 2&1\\ 1&2 \end{pmatrix}. $$ For a function \(f\), write \(f_a=f(a)\) and \(f_b=f(b)\). The operator is $$ (T_k f)(a)=\tfrac12(2f_a+f_b)=f_a+\tfrac12 f_b,\qquad (T_k f)(b)=\tfrac12(f_a+2f_b)=\tfrac12 f_a+f_b. $$
The \(L^2(P)\) inner product is \(\langle f,g\rangle=(f_ag_a+f_bg_b)/2\). Directly, \(\langle f,T_k f\rangle=(f_a^2+f_af_b+f_b^2)/2\). This is nonnegative because \(f_a^2+f_af_b+f_b^2=(f_a+f_b/2)^2+3f_b^2/4\). The matrix is symmetric, so the operator is self-adjoint; its two eigenvalues are \(3/2\) and \(1/2\), corresponding to the vectors \((1,1)\) and \((1,-1)\). The example illustrates that the operator includes the probability weights: its matrix in function-value coordinates is not the original kernel matrix, but one half of that matrix.
Positivity and Its Assumptions
For a self-adjoint operator, positivity means \(\langle f,T_k f\rangle_{L^2(P)}\geq 0\) for every \(f\in L^2(P)\). A pointwise positive definite kernel has nonnegative finite Gram quadratic forms: for every finite set \(x_1,\ldots,x_m\) and real coefficients \(c_1,\ldots,c_m\), \(\sum_{i,j}c_ic_j k(x_i,x_j)\geq0\). To pass from finite Gram matrices to an operator on \(L^2(P)\), measurability and integrability must also be controlled.
Proof. The feature representation and Cauchy–Schwarz in \(\mathcal H\) imply \(|k(x,y)|^2\leq k(x,x)k(y,y)\). Consequently, $$ \int\!\int |k(x,y)|^2\,dP(x)dP(y) \leq \left(\int k(x,x)\,dP(x)\right)^2<\infty. $$ Thus the boundedness result applies, and \(k\) is symmetric since the real Hilbert-space inner product is symmetric.
Fix \(f\in L^2(P)\). The \(\mathcal H\)-valued function \(x\mapsto f(x)\Phi(x)\) is strongly measurable and integrable, because $$ \int |f(x)|\|\Phi(x)\|_{\mathcal H}\,dP(x) \leq \|f\|_{L^2(P)} \left(\int \|\Phi(x)\|_{\mathcal H}^2\,dP(x)\right)^{1/2}<\infty. $$ Its Bochner integral \(h=\int f(x)\Phi(x)\,dP(x)\) therefore exists. Moreover, the double integral of \(|f(x)f(y)k(x,y)|\) is finite: it is bounded above by \(\big(\int |f(x)|\|\Phi(x)\|_{\mathcal H}\,dP(x)\big)^2\). Fubini's theorem and the feature representation now give $$ \langle f,T_k f\rangle_{L^2(P)} =\int\!\int f(x)f(y)\langle\Phi(x),\Phi(y)\rangle_{\mathcal H}\,dP(x)dP(y) =\langle h,h\rangle_{\mathcal H} =\|h\|_{\mathcal H}^2\geq0. $$ This proves positivity and completes the result. \(\square\)
Worked Example: A Linear Kernel on a Bounded Interval
Let \(P\) be uniform on \([-1,1]\), so \(dP(y)=dy/2\), and take \(k(x,y)=xy\). This is a positive definite kernel because for any finite \(x_i,c_i\), \(\sum_{i,j}c_ic_jx_ix_j=(\sum_i c_ix_i)^2\geq0\). Its feature map is \(\Phi(x)=x\in\mathbb R\), and \(\int k(x,x)\,dP(x)=E[X^2]=1/3\), so the theorem's integrability condition holds.
For any \(f\in L^2(P)\), $$ (T_k f)(x)=x\int_{-1}^1 yf(y)\,\frac{dy}{2}=xE[Yf(Y)]. $$ Thus the output is always a scalar multiple of \(x\). For the specific function \(f(y)=1+y\), \(E[Yf(Y)]=E[Y]+E[Y^2]=0+1/3=1/3\), and hence \(T_k f(x)=x/3\). Its quadratic form is $$ \langle f,T_k f\rangle=E[(1+X)X/3] =\tfrac13\{E[X]+E[X^2]\} =\tfrac19. $$ In general, the quadratic form is \((E[Xf(X)])^2\), which is nonnegative. This calculation also shows that the operator has rank one: its range consists of multiples of the function \(x\mapsto x\).
Worked Example: A Gaussian Kernel on a Compact Domain
Let \(P\) be uniform on \([-1,1]\), and use \(k(x,y)=\exp(-(x-y)^2/2)\). This kernel is bounded by one and has \(k(x,x)=1\), so \(\int k(x,x)\,dP(x)=1\). Its feature coordinates can be written \(\phi_j(x)=e^{-x^2/2}x^j/\sqrt{j!}\), \(j=0,1,\ldots\), since $$ \sum_{j=0}^{\infty}\phi_j(x)\phi_j(y) =e^{-x^2/2-y^2/2}\sum_{j=0}^{\infty}\frac{(xy)^j}{j!} =e^{-(x-y)^2/2}. $$ The series converges absolutely, and \(\sum_j\phi_j(x)^2=1\), giving a square-summable feature vector in \(\ell^2\). The coordinates are measurable, so this is a strongly measurable feature map into the separable Hilbert space \(\ell^2\).
For \(f=1\), the operator value is explicitly $$ (T_k1)(x)=\frac12\int_{-1}^{1}e^{-(x-y)^2/2}\,dy =\frac{\sqrt{2\pi}}{2} \left[ \Phi_{\!N}(1-x)-\Phi_{\!N}(-1-x) \right], $$ where \(\Phi_{\!N}\) is the standard normal distribution function. Positivity can also be checked from the feature series: uniform absolute convergence on \([-1,1]^2\) permits termwise integration, giving $$ \langle 1,T_k1\rangle =\sum_{j=0}^{\infty}\frac{1}{j!} \left(E[e^{-X^2/2}X^j]\right)^2>0. $$ Every summand is nonnegative, and the \(j=0\) summand is strictly positive because \(e^{-X^2/2}>0\) throughout the interval. The theorem guarantees nonnegativity for every \(L^2(P)\) function, not only for this constant one.
When the Assumptions Matter
Positive definiteness is a statement about finite collections of points. Boundedness of \(T_k\), by contrast, is a statement about its action on all of \(L^2(P)\). Integrability links these two settings. For instance, the linear kernel \(k(x,y)=xy\) is positive definite on the real line, but if \(P\) is Cauchy, then \(\int k(x,x)\,dP(x)=E[X^2]=\infty\). The feature map \(x\mapsto x\) is not square-integrable, and the theorem above does not apply. In fact, for \(f(y)=y/(1+|y|)\), which is bounded and hence in \(L^2(P)\), the defining integral would require \(E[Y^2/(1+|Y|)]\), which diverges under the Cauchy distribution. Thus the integral formula need not define a finite value even for an \(L^2(P)\) input.
A common mistake is to reason that a positive definite kernel automatically gives a bounded, positive operator on any \(L^2(P)\). That skips the measure-dependent conditions. A safe sequence is to verify that the kernel is measurable, establish a sufficient integrability condition such as \(k\in L^2(P\otimes P)\) for boundedness, check symmetry for self-adjointness, and justify positivity with an appropriate feature-integrability condition. The probability measure is part of the operator, not a disposable detail.
Check Your Understanding
Answer these questions using the definitions and assumptions above.
- Why does \(k\in L^2(P\otimes P)\) ensure that the integral defining \(T_k f\) exists for almost every \(x\) when \(f\in L^2(P)\)?
- Which assumption gives the norm bound for \(T_k\), and which additional assumption gives self-adjointness?
- For the linear kernel \(k(x,y)=xy\), write the quadratic form \(\langle f,T_k f\rangle\) as a square.
- Why is finite Gram-matrix positive definiteness not, by itself, a guarantee that the integral operator is bounded on \(L^2(P)\)?
- In the Gaussian-kernel example, why is the quadratic form for \(f=1\) strictly positive rather than merely nonnegative?