Canonical Feature Maps and Kernel Sections

From Kernel Sections to Feature Vectors

A positive definite kernel can be viewed as a rule for comparing inputs. Its value \(k(x,y)\) is not merely a similarity score: in the reproducing kernel Hilbert space (RKHS) associated with \(k\), it is the inner product of two particular functions. Those functions are the kernel sections, and treating them as feature vectors gives a canonical way to represent each input.

This viewpoint is useful because it turns kernel calculations into ordinary Hilbert-space geometry without requiring coordinates for the RKHS. The map is canonical once the kernel and its RKHS are fixed, although other feature representations of the same kernel may use different spaces or coordinates.

The defining reproducing property of \(\mathcal{H}_k\) states that \(f(x)=\langle f,k_x\rangle_{\mathcal{H}_k}\) for every \(f\in\mathcal{H}_k\) and \(x\in X\). In particular, each section is a function on the input set: its value at \(z\) is \(k_x(z)=k(z,x)\). It is not just the number \(k(x,x)\).

The Feature-Inner-Product Identity

The fundamental identity follows directly from reproduction. Apply the reproducing property to the function \(k_y\), evaluated at \(x\). Symmetry of a real positive definite kernel then gives the result.

Proof. Since \(k_y\in\mathcal{H}_k\), the reproducing property gives

The real Hilbert-space inner product is symmetric, so \(\langle k_y,k_x\rangle_{\mathcal{H}_k}=\langle k_x,k_y\rangle_{\mathcal{H}_k}\). Using \(\Phi(x)=k_x\) and \(\Phi(y)=k_y\) proves the identity. \(\square\)

Thus a kernel value is exactly an inner product between canonical feature vectors. This also gives immediate formulas for their lengths and separation. These formulas follow by expanding the squared norm and applying the identity just proved.

Linear Combinations of Canonical Features

The identity applies not only to single feature vectors but also to every finite linear combination of them. The resulting calculation is often the practical reason to use the canonical map: inner products and norms of combinations can be computed entirely from kernel values.

To derive this, expand the inner product using bilinearity:

In particular, taking the two combinations to be the same gives $$ \left\|\sum_{i=1}^m\alpha_i\Phi(x_i)\right\|_{\mathcal{H}_k}^2 =\sum_{i=1}^m\sum_{j=1}^m\alpha_i\alpha_j k(x_i,x_j). $$

Other Feature Maps and the Canonical Representation

A feature map need not take values in \(\mathcal{H}_k\). More generally, a map \(\psi:X\to G\), where \(G\) is a real Hilbert space, is a feature representation of \(k\) if \(k(x,y)=\langle\psi(x),\psi(y)\rangle_G\) for every pair \(x,y\). The coordinates and even the ambient space can differ from those of the canonical map.

Proof. On finite linear combinations, define \(U(\sum_i\alpha_i\Phi(x_i))=\sum_i\alpha_i\psi(x_i)\). The formula for squared norms of combinations shows that both sides have the same squared norm:

This equality proves that \(U\) is well-defined. Indeed, if one representation of a vector has sum zero in \(\mathcal{H}_k\), its squared norm is zero, so the corresponding sum in \(G\) also has squared norm zero and is the zero vector. Applying the same argument to differences of representations shows that the assigned value does not depend on the chosen representation. The norm equality also shows that \(U\) is linear and preserves norms on the span.

Because \(U\) preserves distances, it extends uniquely by limits to an isometry on the closed span of the canonical features: a Cauchy sequence of finite combinations is sent to a Cauchy sequence in the complete space \(G\), and equivalent sequences have the same limit. The image of this extension is closed, since an isometric image of a complete space is complete and hence closed in \(G\). It contains every \(\psi(x)=U\Phi(x)\), so it contains their closed linear span. It is also contained in that closed span by construction; therefore it equals the closed span of the \(\psi(x)\). Finally, any such isometry must agree with \(U\) on every \(\Phi(x)\), and hence on their span and its closure by continuity. This proves uniqueness. \(\square\)

Common Pitfall and Interpretation

A common mistake is to identify the feature vector \(\Phi(x)\) with the scalar \(k(x,x)\). These are different objects. The section \(\Phi(x)=k(\,\cdot\,,x)\) is a function in an RKHS; \(k(x,x)\) is a number, equal to its squared norm. For instance, in the three-point example, \(\Phi(c)\) is the function with values \((1,1,2)\), whereas \(k(c,c)=2\).

Another tempting but incorrect inference is that different inputs must have different canonical features. The distance identity shows the precise condition:

The forward implication follows because the difference has norm zero. Conversely, a vector in a Hilbert space with squared norm zero is the zero vector, so a zero value on the right implies \(\Phi(x)-\Phi(y)=0\). Consequently, a kernel can assign identical canonical features to distinct inputs; injectivity is not automatic.

The canonical map is valuable because it gives a precise geometric interpretation of kernel values while avoiding a choice of coordinates. Any other feature representation produces the same geometry on the span of its feature vectors, up to the isometry proved above. This is the basis for carrying out feature-space inner-product calculations using only kernel evaluations.