Kernel Bandwidth as a Boundary-Resolution Trade-Off

Bandwidth Sets a Spatial Scale

A kernel classifier makes a prediction from the sign of a score function \(f\). Its decision boundary is where that score is zero. For a smooth kernel, the bandwidth controls how quickly scores can vary over space without using a large RKHS norm. This matters when the Bayes decision changes sharply: a broad kernel can make a narrow transition expensive, while a narrower kernel can resolve changes over a shorter distance.

We use the Gaussian kernel on \(\mathbb{R}^d\), \(k_h(x,z)=\exp(-\|x-z\|^2/(2h^2))\), where \(h>0\) is the bandwidth. The corresponding reproducing kernel Hilbert space is denoted \(\mathcal{H}_h\), with norm \(\|\cdot\|_{\mathcal{H}_h}\). The reproducing property says \(f(x)=\langle f,k_h(\cdot,x)\rangle_{\mathcal{H}_h}\) for \(f\in\mathcal{H}_h\). A kernel section \(k_h(\cdot,x)\) is a smooth function concentrated around \(x\), with spatial scale proportional to \(h\).

The bandwidth does not determine classification quality by itself. The relevant quantities include both the norm of the score and where the feature distribution places probability mass. The following exact calculation makes the dependence of score variation on \(h\) precise.

A Norm Bound for Nearby Scores

For any \(x,y\in\mathbb{R}^d\), the reproducing property turns a difference in scores into an inner product with a difference of kernel sections. The norm of that section difference is available in closed form for the Gaussian kernel.

Proof. Write \(k_x=k_h(\cdot,x)\) and \(k_y=k_h(\cdot,y)\). By the reproducing property and Cauchy–Schwarz,

The squared norm of the section difference is

This gives the first inequality. For \(u\geq 0\), \(1-e^{-u}\leq u\). Apply this with \(u=\|x-y\|^2/(2h^2)\); then the square root in the first bound is at most \(\|x-y\|/h\). This proves the second inequality. \(\square\)

In particular, if \(\|f\|_{\mathcal{H}_h}\leq B\), the score cannot change by more than \(B\|x-y\|/h\) between two nearby points. Equivalently, requiring a contrast of magnitude at least \(2a\) across points separated by distance \(\delta\) forces the lower bound

When \(\delta\) is small relative to \(h\), the denominator is approximately \(\delta/h\), so the required norm is approximately \(2ah/\delta\). This is the precise sense in which a broad kernel makes a fixed-amplitude, rapid score change expensive. The bound is not saying that every narrow boundary requires a large norm for every bandwidth: when \(h\) is itself small relative to \(\delta\), the exact bound is close to a constant.

An Explicit Smooth Boundary Score

A finite combination of Gaussian kernel sections provides a direct example of a smooth score with a sharp sign boundary. In one dimension, place sections at \(-h\) and \(h\) and subtract them. This construction has a zero exactly at the origin; its transition scale tracks the bandwidth.

That qualification is important. Correct signs alone do not ensure that a score is a good approximation to a target real-valued score, nor do they ensure small logistic loss. A smooth score can have the right sign but very small magnitude, and a norm calculation at two points does not control its values everywhere. In classification, one must connect score behavior to probability mass near the decision boundary.

Boundary Resolution and Classification Risk

For binary labels \(Y\in\{-1,+1\}\), a classifier induced by \(f\) predicts \(+1\) when \(f(X)\geq0\) and \(-1\) otherwise. The Bayes classifier minimizes the probability of misclassification. In a noiseless example with \(Y=\operatorname{sign}(X)\), errors can occur only where the classifier disagrees with that sign. If the disagreement is confined to a narrow band around zero, the probability mass of that band gives an immediate risk bound.

In settings with label noise, the probability of a boundary layer alone is not enough to determine excess risk. What matters also is how likely the label is to favor one class strongly at each location. Margin conditions formalize that relationship. The bandwidth calculation supplies a geometric ingredient—how finely the score can change—while the distribution and noise determine the risk cost of errors near the boundary.

What the Trade-Off Does and Does Not Say

The exact section-difference bound yields a useful interpretation. If \(h\) is large compared with the distance over which a score must change by a fixed amount, the required RKHS norm can be large. Choosing a smaller bandwidth makes the kernel sections more localized and can make that change representable with a smaller norm. But the bandwidth alone does not specify the score's norm, the location of its zero set, or the risk of the resulting classifier.

The reverse direction also matters. At a fixed norm budget \(B\), the theorem implies that a score contrast between points separated by \(\delta\) is bounded above by \(B\sqrt{2(1-e^{-\delta^2/(2h^2)})}\). When \(\delta\ll h\), this is approximately \(B\delta/h\). A broad kernel and a small norm budget therefore limit how rapidly the score can move across a narrow region. A smaller \(h\) relaxes this particular restriction, but does not guarantee that the data support the resulting fine-scale boundary.

This tutorial concerns the representational trade-off for a fixed bandwidth. Choosing a bandwidth from data, and specifying how it should change with sample size to obtain consistency, are separate questions. Likewise, a fixed kernel can fail to represent the relevant decision rule well even when its score optimization is carried out accurately.

Common Pitfall

A related mistake is to use the score-contrast bound as if it were an equality for every score. It is an upper bound on possible variation at a given norm, and its rearrangement is a necessary lower bound on the norm for a specified contrast. The explicit section-difference example supplies a particular score and its exact norm; it does not show that the lower bound is universally attained.