Why Restricting the Domain Changes the Norm
A kernel is often defined on a broad input domain even when the observations or predictions of interest lie in a smaller set. For instance, a kernel may be defined on all of \(\mathbb{R}\), while a study uses only inputs from an interval or a finite collection of points. Restricting the kernel to those inputs certainly restricts which function values can be evaluated, but the resulting RKHS norm also has a precise interpretation: it measures the cost of the least-norm function on the original domain that agrees with the specified restricted function.
This distinction matters because a function on a subset can have many extensions, with different norms, or may not be the restriction of any function in the original RKHS. The restricted RKHS contains exactly the extendable functions. Its norm accounts for the best possible extension rather than assigning a norm based on an arbitrary choice.
The Restricted RKHS and Its Norm
The key result identifies both the function space and its norm. In the theorem, an extension of \(g:A\to\mathbb{R}\) means a function \(f:\mathcal{X}\to\mathbb{R}\) satisfying \(f|_A=g\). The infimum is over extensions that belong to the original RKHS; if there are none, the infimum is \(+\infty\).
Here is a proof that also explains why the infimum defines a Hilbert-space norm. Let \(R\) be the restriction map \(Rf=f|_A\), and let \(N=\{f\in\mathcal{H}_K(\mathcal{X}):Rf=0\}\). Each evaluation \(f\mapsto f(a)\) is continuous in an RKHS: the reproducing property and Cauchy–Schwarz give \(|f(a)|\leq\|f\|_{\mathcal{H}_K}\sqrt{K(a,a)}\). Thus \(N\), an intersection of the closed sets \(\{f:f(a)=0\}\), is a closed linear subspace.
Two functions have the same restriction exactly when their difference belongs to \(N\). Therefore the restrictions can be identified with the quotient space \(\mathcal{H}_K(\mathcal{X})/N\). Give this quotient its usual Hilbert-space inner product: each equivalence class has a unique representative in \(N^\perp\), and the quotient norm is the norm of that representative. Equivalently, the quotient norm of the class of \(f\) is \(\inf_{n\in N}\|f+n\|_{\mathcal{H}_K}\), which is exactly the infimum over extensions of \(Rf\).
It remains to verify the reproducing kernel of this quotient. For \(a\in A\), let \(K_a=K(\,\cdot\,,a)\). If \(n\in N\), then \(\langle n,K_a\rangle_{\mathcal{H}_K}=n(a)=0\), so \(K_a\in N^\perp\). For any restriction \(g=Rf\), with quotient representative \(P_{N^\perp}f\), we have \[ \langle g, RK_a\rangle_{\mathcal{H}_K/N} =\langle P_{N^\perp}f,K_a\rangle_{\mathcal{H}_K} =\langle f,K_a\rangle_{\mathcal{H}_K} =f(a)=g(a). \] Thus \(RK_a\) reproduces evaluation at \(a\). The inner product of the two restricted sections at \(a,b\in A\) is \(\langle K_a,K_b\rangle_{\mathcal{H}_K}=K(a,b)=K_A(a,b)\). Hence the quotient space has reproducing kernel \(K_A\), so by uniqueness of the RKHS associated with a kernel it is \(\mathcal{H}_{K_A}(A)\). \(\square\)
The minimum-norm extension follows directly from the same decomposition. If \(f_0\) is any extension of \(g\), every other extension is \(f_0+n\) for some \(n\in N\). Write \(f_0=P_{N^\perp}f_0+P_Nf_0\). Orthogonality gives \[ \|f_0+n\|_{\mathcal{H}_K}^2 =\|P_{N^\perp}f_0\|_{\mathcal{H}_K}^2+\|P_Nf_0+n\|_{\mathcal{H}_K}^2. \] This is minimized uniquely when \(n=-P_Nf_0\), leaving the extension \(P_{N^\perp}f_0\). Moreover, \[ N^\perp=\overline{\operatorname{span}}\{K_a:a\in A\}. \] Indeed, a function is orthogonal to every \(K_a\) exactly when \(f(a)=0\) for every \(a\in A\), which is exactly membership in \(N\); taking orthogonal complements proves the identity. Thus the least-norm extension lies in the closed span of the kernel sections indexed by the subset. \(\square\)
Worked Examples
Worked Example: Restricting the Identity Kernel on a Finite Set
Let \(\mathcal{X}=\{1,2,3\}\) and \(K(i,j)=1\) if \(i=j\), and \(0\) otherwise. Take \(A=\{1,2\}\). The three kernel sections are the coordinate vectors, so an RKHS function has values \((y_1,y_2,y_3)\) and squared norm \(y_1^2+y_2^2+y_3^2\).
A function \(g\) on \(A\) has values \((c_1,c_2)\). Its extensions have values \((c_1,c_2,t)\), where \(t\) is arbitrary, and their squared norms are \(c_1^2+c_2^2+t^2\). The minimum occurs at \(t=0\), giving \(\|g\|_{\mathcal{H}_{K_A}}^2=c_1^2+c_2^2\). The restricted kernel is again the identity kernel, now on \(\{1,2\}\), in agreement with this norm.
Worked Example: Restricting a Linear Kernel
On \(\mathcal{X}=\mathbb{R}\), take \(K(x,z)=xz\) and \(A=\{1,2\}\). The original RKHS consists of functions \(f(x)=\alpha x\), with \(\|f\|_{\mathcal{H}_K}=|\alpha|\): the feature map is \(x\mapsto x\), so the function is the inner product with the scalar \(\alpha\).
Restriction gives \(g(1)=\alpha\) and \(g(2)=2\alpha\). Consequently, the restricted RKHS contains only pairs of values of the form \((c,2c)\), not all functions on the two-point set. Such a pair has only one possible original extension, \(f(x)=cx\), and its restricted norm is \(|c|\). Equivalently, the restricted Gram matrix is \[ \begin{pmatrix}1&2\\2&4\end{pmatrix}, \] which has rank one, reflecting the one-dimensional restricted function space.
Worked Example: A Correlated Kernel and a Nontrivial Minimum Extension
Let \(\mathcal{X}=\{u,v\}\) and specify the kernel by its Gram matrix \[ K=\begin{pmatrix}2&1\\1&2\end{pmatrix}. \] Its eigenvalues are \(3\) and \(1\), so it is positive definite. For a value vector \(y=(c,t)^\mathsf{T}\), the squared RKHS norm is \(y^\mathsf{T}K^{-1}y\). Since \[ K^{-1}=\frac{1}{3}\begin{pmatrix}2&-1\\-1&2\end{pmatrix}, \] this squared norm is \((2c^2-2ct+2t^2)/3\).
Now restrict to \(A=\{u\}\), and prescribe \(g(u)=c\). The value \(t=f(v)\) is free when choosing an extension. Completing the square gives \[ \frac{2c^2-2ct+2t^2}{3} =\frac{c^2}{2}+\frac{2}{3}\left(t-\frac{c}{2}\right)^2. \] Therefore the minimum occurs at \(t=c/2\), and the restricted squared norm is \(c^2/2\). This agrees with the restricted kernel \(K_A(u,u)=2\), whose one-point RKHS assigns squared norm \(c^2/2\) to the value \(c\).
What the Norm Measures—and a Common Pitfall
The theorem gives a useful interpretation: the restricted norm depends on how easily a function on \(A\) can be extended to the original domain within the original RKHS. It is not generally the norm of a chosen extension, nor can one freely assign arbitrary values on \(A\). The extension with least norm is the relevant one.
A common mistake is to take any extension \(f\) of \(g\) and declare \(\|g\|_{\mathcal{H}_{K_A}}=\|f\|_{\mathcal{H}_K}\). This fails whenever two extensions have different norms. In the correlated-kernel example, both \((c,0)\) and \((c,c/2)\) extend the same value at \(u\), but their squared norms are \(2c^2/3\) and \(c^2/2\), respectively. The correction is to minimize over all extensions; the theorem guarantees that this minimum exists and is unique.
Another useful consequence is that restricting a domain does not necessarily mean taking an arbitrary subspace of functions with the inherited norm. It forms a quotient by functions invisible on the subset: any two original functions agreeing on \(A\) represent the same restricted function. The quotient norm removes the part of an extension that vanishes on \(A\).
Check Your Understanding
Use the theorem and examples to answer the following questions.
- Why is the set of original RKHS functions that vanish on \(A\) a closed subspace?
- For the identity kernel on \(\{1,2,3\}\) restricted to \(\{1,2\}\), what value at \(3\) gives the minimum-norm extension, and why?
- For the linear kernel \(K(x,z)=xz\) restricted to \(\{1,2\}\), can the restricted function take values \((1,3)\)? Explain.
- In the correlated-kernel example, what value at \(v\) minimizes the extension norm for a prescribed value \(c\) at \(u\)?
- Explain why the restricted norm is not defined by selecting an arbitrary extension and using its original RKHS norm.