Feature Covariance and Kernel Integral Operators

Why Two Operators Describe One Kernel

A positive definite kernel can be viewed either through functions in its reproducing kernel Hilbert space or through an integral operator acting on square-integrable functions. These viewpoints use different spaces and have different eigenfunctions, but their positive eigenvalues agree. The link is the feature map: it sends an RKHS function to its values as an element of $L^2(P)$.

We make the integrability assumptions explicit so that each operator is well-defined and compact. Let $(\mathcal X,\mathcal A,P)$ be a probability space, let $k:\mathcal X\times\mathcal X\to\mathbb R$ be a measurable, symmetric, positive definite kernel, and let $\mathcal H$ be its RKHS. Assume $\mathcal H$ is separable and

\int_{\mathcal X} k(x,x)\,dP(x)<\infty.

For the canonical feature map, $\phi(x)=k(x,\cdot)\in\mathcal H$, the reproducing property gives $k(x,y)=\langle\phi(x),\phi(y)\rangle_{\mathcal H}$ and $\|\phi(x)\|_{\mathcal H}^2=k(x,x)$. In particular, the assumption says that the feature map is square-integrable.

The Two Operators

The feature-space operator below is sometimes called a covariance operator. Here it is the uncentered second-moment operator: it uses $\phi(X)$, not $\phi(X)-E[\phi(X)]$. For $u,v\in\mathcal H$, write $(u\otimes v)h=\langle h,v\rangle_{\mathcal H}u$.

Definition: The feature second-moment operator $C:\mathcal H\to\mathcal H$ is $$ C=E[\phi(X)\otimes\phi(X)],\qquad Ch=E[\langle h,\phi(X)\rangle_{\mathcal H}\phi(X)]. $$ The kernel integral operator $T:L^2(P)\to L^2(P)$ is $$ (Tg)(x)=\int_{\mathcal X} k(x,y)g(y)\,dP(y), $$ where the result is understood as an equivalence class modulo equality $P$-almost everywhere.

The expectation defining $C$ exists as a Bochner integral: by Cauchy–Schwarz, $E[\|\phi(X)\|_{\mathcal H}]\leq (E[\|\phi(X)\|_{\mathcal H}^2])^{1/2}<\infty$, and for fixed $h$, the vector-valued integrand has norm at most $\|h\|_{\mathcal H}\|\phi(X)\|_{\mathcal H}^2$, which is integrable. The operator $C$ is self-adjoint and positive because $\langle Ch,h\rangle_{\mathcal H}=E[\langle h,\phi(X)\rangle_{\mathcal H}^2]\geq0$.

For the integral operator, positive definiteness implies $|k(x,y)|^2\leq k(x,x)k(y,y)$: this is the Cauchy–Schwarz inequality for the two feature vectors. Therefore $\int\int |k(x,y)|^2\,dP(x)dP(y)\leq(\int k(x,x)\,dP(x))^2<\infty$. Thus the kernel is square-integrable and defines a Hilbert–Schmidt, hence compact, operator on $L^2(P)$. The factorization below will also establish that $T$ is positive and self-adjoint.

Worked Example: A Two-Point Distribution and a Linear Kernel

Let $P(X=-1)=1/4$, $P(X=1)=3/4$, and take $k(x,y)=1+xy$. A feature map is $\phi(x)=(1,x)^\top$ in $\mathbb R^2$, with the usual inner product. The required moments are $E[X]=(-1)(1/4)+(1)(3/4)=1/2$ and $E[X^2]=1$. Hence

C=E[\phi(X)\phi(X)^\top] =\begin{pmatrix}1&E[X]\\E[X]&E[X^2]\end{pmatrix} =\begin{pmatrix}1&1/2\\1/2&1\end{pmatrix}.

The characteristic polynomial is $(1-\lambda)^2-1/4$, so the eigenvalues are $3/2$ and $1/2$. Now represent $g\in L^2(P)$ by $(g(-1),g(1))^\top$. Since $k(-1,1)=k(1,-1)=0$ and $k(-1,-1)=k(1,1)=2$,

(Tg)(-1)=\tfrac14(2)g(-1)=\tfrac12g(-1),\qquad (Tg)(1)=\tfrac34(2)g(1)=\tfrac32g(1).

Thus the two coordinate indicator functions are eigenfunctions of $T$, with eigenvalues $1/2$ and $3/2$. The $L^2(P)$ inner product weights the coordinates by $1/4$ and $3/4$, but that weighting does not change these eigenvalue equations. This calculation also checks directly that the positive eigenvalues match those of $C$.

Worked Example: A Constant Kernel

Let $P$ be any probability measure and set $k(x,y)=2$. A feature map is the constant $\phi(x)=\sqrt{2}\in\mathbb R$, so $C$ acts on $\mathbb R$ as multiplication by $E[(\sqrt 2)^2]=2$. The integral operator is

(Tg)(x)=2\int_{\mathcal X}g(y)\,dP(y).

The constant function $1$ has $L^2(P)$ norm one and satisfies $T1=2\cdot1$. If $\int g\,dP=0$, then $Tg=0$. Every $g\in L^2(P)$ decomposes as $g=(\int g\,dP)1+[g-(\int g\,dP)1]$, where the second term has mean zero. Therefore $T$ has just one positive eigenvalue, $2$, matching $C$.

Worked Example: A Rank-One Linear Kernel Under a Normal Distribution

Suppose $X\sim N(0,1)$, take $k(x,y)=xy$, and let $\phi(x)=x\in\mathbb R$. The diagonal integral is $E[X^2]=1$, and $C$ is multiplication by $1$. For $g\in L^2(P)$,

$$ (Tg)(x)=xE[Xg(X)]. $$

The function $x\mapsto x$ is in $L^2(P)$, has squared norm $E[X^2]=1$, and satisfies $T(x)=xE[X^2]=x$. If $E[Xg(X)]=0$, then $Tg=0$. More generally, $g=E[Xg(X)]X+[g-E[Xg(X)]X]$; the bracketed term is orthogonal to $X$ and is mapped to zero. Hence $T$ has positive eigenvalue $1$ and otherwise acts as zero, as does the feature-space operator.

Factorization Through the Feature Map

Define $S:\mathcal H\to L^2(P)$ by $(Sh)(x)=h(x)=\langle h,\phi(x)\rangle_{\mathcal H}$. This is well-defined on $L^2(P)$ equivalence classes. Indeed,

\|Sh\|_{L^2(P)}^2 =E[\langle h,\phi(X)\rangle_{\mathcal H}^2] \leq \|h\|_{\mathcal H}^2 E[\|\phi(X)\|_{\mathcal H}^2].

Its adjoint is the feature average $S^*:L^2(P)\to\mathcal H$, given by $S^*g=E[g(X)\phi(X)]$. The Bochner integral exists since $E[|g(X)|\|\phi(X)\|_{\mathcal H}]\leq\|g\|_{L^2(P)}(E\|\phi(X)\|_{\mathcal H}^2)^{1/2}$. For $h\in\mathcal H$, the adjoint identity follows directly: $\langle Sh,g\rangle_{L^2(P)}=E[\langle h,\phi(X)\rangle g(X)]=\langle h,S^*g\rangle_{\mathcal H}$.

Theorem: Under the assumptions above, $C=S^*S$ and $T=SS^*$. In particular, both operators are positive, self-adjoint, and compact.

Proof. For $h\in\mathcal H$, the adjoint formula gives \[ S^*Sh=E[(Sh)(X)\phi(X)] =E[\langle h,\phi(X)\rangle_{\mathcal H}\phi(X)] =Ch. \] Thus $C=S^*S$. For $g\in L^2(P)$, $SS^*g$ is represented, almost everywhere, by \[ (SS^*g)(x)=\left\langle E[g(Y)\phi(Y)],\phi(x)\right\rangle_{\mathcal H} =\int_{\mathcal X}g(y)\langle\phi(y),\phi(x)\rangle_{\mathcal H}\,dP(y) =(Tg)(x). \] The inner product may be passed through the Bochner integral because it is a bounded linear functional; the resulting integral is absolutely finite for $P$-almost every $x$, as is also implied by the square-integrability of the kernel and Cauchy–Schwarz on $L^2(P)$. Hence $T=SS^*$. Both products are positive and self-adjoint: for example, $\langle S^*Sh,h\rangle=\|Sh\|^2\geq0$, and the same argument applies to $SS^*$. Finally, $S$ is Hilbert–Schmidt, since for any orthonormal basis $(e_j)$ of $\mathcal H$, Tonelli’s theorem and Parseval’s identity give \[ \sum_j\|Se_j\|_{L^2(P)}^2 =E\left[\sum_j|\langle e_j,\phi(X)\rangle_{\mathcal H}|^2\right] =E[\|\phi(X)\|_{\mathcal H}^2]<\infty. \] Thus $S$ is compact, and both $S^*S$ and $SS^*$ are compact. $\square$

Why the Positive Eigenvalues Agree

Theorem: The positive eigenvalues of $C$ and $T$ agree, including their multiplicities. For each $\lambda>0$, the map $S$ sends the $\lambda$-eigenspace of $C$ bijectively onto the $\lambda$-eigenspace of $T$; its inverse on that eigenspace is $\lambda^{-1}S^*$.

Proof. Suppose $Cu=\lambda u$ for $u\in\mathcal H$ and $\lambda>0$. Then $Su\neq0$, since $Su=0$ would imply $Cu=S^*Su=0$, contradicting $\lambda u\neq0$. Using the factorization, \[ T(Su)=SS^*Su=S(Cu)=\lambda Su. \] Thus $S$ maps the $\lambda$-eigenspace of $C$ into that of $T$. Conversely, if $Tv=\lambda v$, then $S^*v\neq0$, because $S^*v=0$ would give $Tv=SS^*v=0$. Also, \[ C(S^*v)=S^*SS^*v=S^*(Tv)=\lambda S^*v. \] On a $C$-eigenvector $u$, $\lambda^{-1}S^*Su=\lambda^{-1}Cu=u$; on a $T$-eigenvector $v$, $\lambda^{-1}SS^*v=\lambda^{-1}Tv=v$. These maps are therefore inverse bijections between the two eigenspaces. Their dimensions, which are the eigenvalue multiplicities, agree. Since the operators are compact and positive, their nonzero spectrum consists of positive eigenvalues, so this proves the claim. $\square$

The eigenfunctions themselves need not be the same objects: an eigenvector of $C$ lies in $\mathcal H$, while an eigenfunction of $T$ lies in $L^2(P)$. If $Cu=\lambda u$ with $\lambda>0$, then $Su$ is an $L^2(P)$ eigenfunction. In the other direction, an eigenfunction $v$ of $T$ produces the RKHS eigenvector $S^*v$. The positive eigenvalues match because both operators are compositions of the same map and its adjoint in opposite orders.

The finite-diagonal-integral assumption also ensures $C$ is trace-class. For an orthonormal basis of $\mathcal H$, the same Tonelli and Parseval calculation gives $\sum_j\langle Ce_j,e_j\rangle_{\mathcal H}=E[\|\phi(X)\|_{\mathcal H}^2]=\int k(x,x)\,dP(x)$. This identifies the total feature-space second moment with the trace of $C$; it does not require an eigenfunction expansion of the kernel.

Common Pitfall: Centering Changes the Operator

The word “covariance” is sometimes reserved for a centered quantity. If $\mu=E[\phi(X)]$, the centered feature covariance is $E[(\phi(X)-\mu)\otimes(\phi(X)-\mu)]=C-\mu\otimes\mu$, not $C$. The factorization proved here uses the uncentered $C=E[\phi(X)\otimes\phi(X)]$ and the original kernel integral operator $T$. Substituting the centered covariance while leaving $T$ unchanged breaks the identity $C=S^*S$. Centering can be handled by changing the feature map or kernel operator consistently, but it is a different construction.

A related mistake is to claim that $C$ and $T$ have identical eigenvectors. They generally do not even act on the same space. What is preserved is the positive spectrum and the dimension of each positive eigenspace, with $S$ and $S^*$ translating between the corresponding eigenvectors.

Key takeaway: With a square-integrable feature map, $C=S^*S$ on the RKHS and $T=SS^*$ on $L^2(P)$. These are compact positive operators with matching positive eigenvalues, counted with multiplicity, even though their eigenvectors belong to different spaces.

Check Your Understanding

Use the definitions and operator identities above to answer the following.

Why does $\int k(x,x)\,dP(x)<\infty$ imply that the feature map is square-integrable?
For $C=S^*S$, explain why $Cu=\lambda u$ with $\lambda>0$ forces $Su\neq0$.
In the two-point example, calculate $E[X]$ and explain why the off-diagonal entries of $C$ are positive.
For the constant kernel $k(x,y)=2$, describe the null space of $T$.
What changes if the feature covariance is centered but the integral operator still uses the original, uncentered kernel?