Geometry of RKHS Norm Balls

Introduction

An RKHS norm measures more than the size of a function in an abstract Hilbert space. Through the reproducing property, it also controls the function’s values at individual inputs and the differences between values at nearby or strongly related inputs. The geometry of a norm ball therefore translates directly into constraints on possible predictions.

The key objects are the kernel sections: for an input \(x\), the function \(k_x=k(\cdot,x)\). Evaluation at \(x\) is the inner product with \(k_x\), so Cauchy–Schwarz turns a norm constraint on a predictor into a bound on its evaluation. Comparing two sections gives a corresponding bound on differences. These facts also reveal why separate bounds at each input do not, by themselves, describe all the joint restrictions on predictions.

Norm Balls and Evaluation Bounds

The value \(k(x,x)\) is nonnegative, since it is the squared norm of \(k_x\). Applying Cauchy–Schwarz to the reproducing property gives a bound on every member of the ball.

Indeed, \(|f(x)|=|\langle f,k_x\rangle_{\mathcal H}|\leq\|f\|_{\mathcal H}\|k_x\|_{\mathcal H}\leq R\sqrt{k(x,x)}\). For sharpness, the specified function has norm \(R\), and its value at \(x\) is \(R\|k_x\|_{\mathcal H}=R\sqrt{k(x,x)}\). If \(k_x=0\), every function in the space has value zero at \(x\), so the bound is also exact.

Thus the radius alone does not determine the same prediction bound everywhere: the diagonal \(k(x,x)\) matters. When the diagonal is constant, the same bound applies at every input. When it varies, the ball permits larger absolute values at inputs with larger diagonal values.

Distances Between Kernel Sections

The inner product between two sections is itself a kernel evaluation: \(\langle k_x,k_y\rangle_{\mathcal H}=k(x,y)\). Expanding the squared Hilbert-space distance therefore expresses it entirely in terms of the kernel.

To prove the identity, expand the squared norm using the inner product: $$ \|k_x-k_y\|_{\mathcal H}^2 =\langle k_x,k_x\rangle_{\mathcal H} +\langle k_y,k_y\rangle_{\mathcal H} -2\langle k_x,k_y\rangle_{\mathcal H} =k(x,x)+k(y,y)-2k(x,y). $$ For the bound, use reproduction at both inputs and then Cauchy–Schwarz: $$ |f(x)-f(y)| =|\langle f,k_x-k_y\rangle_{\mathcal H}| \leq \|f\|_{\mathcal H}\|k_x-k_y\|_{\mathcal H} \leq R\|k_x-k_y\|_{\mathcal H}. $$ Substituting the identity proves the result. When \(k_x\neq k_y\) and \(R>0\), equality is attained by taking \(f=R(k_x-k_y)/\|k_x-k_y\|_{\mathcal H}\). If the sections are equal, the difference in predictions is zero for every \(f\in\mathcal H\). \(\square\)

This is a geometric statement about the kernel sections, not necessarily about ordinary distance between inputs. Inputs whose sections are close in the RKHS have similar predictions for every function in a bounded norm ball. The kernel determines which inputs are close in this sense.

Joint Constraints on Predictions

A finite set of evaluations can be treated as one inner product. Let \(x_1,\ldots,x_m\) be inputs, let \(a_1,\ldots,a_m\) be real coefficients, and define the Gram matrix \(K\) by \(K_{ij}=k(x_i,x_j)\). The linear combination of predictions is the inner product with the corresponding linear combination of kernel sections.

Set \(g=\sum_i a_i k_{x_i}\). By reproduction and linearity, $$ \sum_{i=1}^m a_i f(x_i)=\left\langle f,\sum_{i=1}^m a_i k_{x_i}\right\rangle_{\mathcal H} =\langle f,g\rangle_{\mathcal H}. $$ Cauchy–Schwarz bounds the absolute value by \(\|f\|_{\mathcal H}\|g\|_{\mathcal H}\leq R\|g\|_{\mathcal H}\). Expanding the squared norm gives $$ \|g\|_{\mathcal H}^2 =\sum_{i=1}^m\sum_{j=1}^m a_i a_j\langle k_{x_i},k_{x_j}\rangle_{\mathcal H} =\sum_{i=1}^m\sum_{j=1}^m a_i a_j k(x_i,x_j) =a^\mathsf{T}Ka. $$ When \(g\neq0\), the proposed \(f\) has norm \(R\) and \(\langle f,g\rangle_{\mathcal H}=R\|g\|_{\mathcal H}\), proving sharpness. When \(g=0\), the linear combination is zero for every \(f\). \(\square\)

This result gives a sharp bound for each chosen linear combination of predictions. It also explains why predictions at several inputs are jointly constrained: their combined bound depends on the Gram matrix, including off-diagonal kernel values, not just on separate diagonal entries.

Why the Geometry Matters

An RKHS norm ball can be understood through the support of its predictions: the maximum possible value of a chosen linear combination of evaluations is the radius times the norm of its representer, a linear combination of kernel sections. This viewpoint is useful whenever a norm constraint is intended to control prediction magnitude or prediction variation. It shows precisely which kernel quantities determine those controls.

A common mistake is to treat pointwise bounds as independent. For example, \(|f(x_i)|\leq R\sqrt{k(x_i,x_i)}\) at every input does not mean that every vector of values satisfying those coordinatewise inequalities is attainable. Off-diagonal entries of the Gram matrix constrain combinations of values. To check a proposed joint pattern, apply the linear-combination bound to relevant coefficients; for the sum in the preceding example, the proposed values fail that test.

A second pitfall is to use \(k(x,y)\) alone as the distance between inputs or sections. The squared section distance is \(k(x,x)+k(y,y)-2k(x,y)\). The diagonal terms matter, and the expression as a whole is nonnegative because it is a squared norm. Keeping the section-distance formula intact avoids sign and scaling errors.