Population and Sample Kernel Ridge Operators

Why Two Kernel Ridge Solutions Appear

Kernel ridge regression has a population version and a sample version. The population version is defined using expectations under the data-generating distribution; it is a fixed target for a given regularization level. The sample version replaces those expectations with averages from the observed data, so it is random under random sampling. Keeping these objects distinct is essential: the sample solution is not simply the population solution written in different notation.

We work in a reproducing kernel Hilbert space $\mathcal H$ with measurable positive definite kernel $k$. Write $k_x=k(x,\cdot)$, so that $f(x)=\langle f,k_x\rangle_{\mathcal H}$. Suppose $(X,Y)$ has the population distribution, $\mathcal H$ is separable, $x\mapsto k_x$ is measurable as an $\mathcal H$-valued map, $E\|k_X\|_{\mathcal H}^2<\infty$, and $E[Y^2]<\infty$. Let $(X_i,Y_i)_{i=1}^n$ be independent observations with this same distribution. These assumptions ensure the expectations below exist; in particular, $E[|Y|\|k_X\|_{\mathcal H}]<\infty$ by Cauchy–Schwarz.

Population and Empirical Operators

Definition: The population feature covariance operator $C:\mathcal H\to\mathcal H$ and the population response-feature vector $g\in\mathcal H$ are $$ C=E[k_X\otimes k_X], \qquad g=E[Yk_X], $$ where $(u\otimes v)h=u\langle v,h\rangle_{\mathcal H}$. Given a sample, define the empirical covariance operator $C_n$ and empirical response-feature vector $g_n$ by $$ C_n=\frac1n\sum_{i=1}^n k_{X_i}\otimes k_{X_i}, \qquad g_n=\frac1n\sum_{i=1}^n Y_i k_{X_i}. $$

For every $h\in\mathcal H$, the population operator satisfies $\langle h,Ch\rangle_{\mathcal H}=E[h(X)^2]\geq 0$. Thus $C$ is positive and self-adjoint. It is bounded, and in fact trace class, because $E\|k_X\|_{\mathcal H}^2<\infty$. The empirical operator is also positive and self-adjoint; its range is contained in the span of $k_{X_1},\ldots,k_{X_n}$.

The population and empirical operators act on functions in the RKHS, not on vectors of observed responses. Their connection to prediction is given by $$ \langle f,Cf\rangle_{\mathcal H}=E[f(X)^2], \qquad \langle f,g\rangle_{\mathcal H}=E[Yf(X)]. $$ These identities follow from the reproducing property and the definitions of $C$ and $g$.

The Regularized Operator Equations

For $\lambda>0$, define the population kernel ridge objective $J_\lambda(f)=E[(Y-f(X))^2]+\lambda\|f\|_{\mathcal H}^2$. Expanding the square gives $J_\lambda(f)=E[Y^2]-2\langle f,g\rangle_{\mathcal H}+\langle f,Cf\rangle_{\mathcal H}+\lambda\|f\|_{\mathcal H}^2$. Its minimizer is characterized by a regularized operator equation.

Theorem: Under the assumptions above, for every $\lambda>0$, there is a unique minimizer $f_\lambda\in\mathcal H$ of $J_\lambda$. It is the unique solution of $$ (C+\lambda I)f_\lambda=g, \qquad f_\lambda=(C+\lambda I)^{-1}g. $$ The empirical objective $J_{n,\lambda}(f)=n^{-1}\sum_{i=1}^n(Y_i-f(X_i))^2+\lambda\|f\|_{\mathcal H}^2$ also has a unique minimizer $\hat f_\lambda$, satisfying $$ (C_n+\lambda I)\hat f_\lambda=g_n, \qquad \hat f_\lambda=(C_n+\lambda I)^{-1}g_n. $$

Proof. The directional derivative of $J_\lambda$ at $f$ in direction $h\in\mathcal H$ is $2\langle (C+\lambda I)f-g,h\rangle_{\mathcal H}$. A minimizer must therefore solve $(C+\lambda I)f=g$, and any solution is a minimizer because the objective is quadratic with nonnegative quadratic part. To establish existence and uniqueness, let $A=C+\lambda I$. It is bounded and self-adjoint, and $\langle h,Ah\rangle_{\mathcal H}\geq\lambda\|h\|_{\mathcal H}^2$. Consequently $\|Ah\|_{\mathcal H}\geq\lambda\|h\|_{\mathcal H}$. This inequality makes the range of $A$ closed: if $Ah_m$ converges, then $h_m$ is Cauchy and its limit maps to the limit of $Ah_m$. Also, the orthogonal complement of the range is $\ker(A)$, since $A$ is self-adjoint; the same inequality gives $\ker(A)=\{0\}$. Thus the range is both dense and closed, so it equals $\mathcal H$. The equation has a unique solution, and $\|A^{-1}\|\leq 1/\lambda$. The empirical argument is identical with $C_n$ and $g_n$ in place of $C$ and $g$, since $C_n$ is positive and self-adjoint. $\square$

The population solution $f_\lambda$ depends on the distribution and on $\lambda$, but not on the realized sample. By contrast, $C_n$, $g_n$, and $\hat f_\lambda$ are random. Even though each empirical operator equation has a unique solution, it need not produce the same function as the population equation.

Worked Example: A Constant Kernel Shows the Difference

Let $k(x,x')=1$ for all $x,x'$. The RKHS consists of constant functions $f(x)=a$, with norm $\|f\|_{\mathcal H}=|a|$. Suppose $Y$ takes values $0,2,4$, each with probability $1/3$, and choose $\lambda=1$.

For any $x$, $k_x$ is the constant function $1$, so $C$ acts as the identity on this one-dimensional space. Also, $g=E[Y]\,1=2\cdot 1$. Therefore the population equation is $2f_\lambda=2\cdot1$, giving $f_\lambda(x)=1$.

Now consider a sample of size $n=2$ in which both observed responses are zero. This event has probability $1/9$. Again $C_n$ acts as the identity, while $g_n=0$, so the empirical equation gives $\hat f_\lambda(x)=0$. The population target is $1$, but this realized sample solution is $0$. The difference is due to replacing population expectations by sample averages, not to a change in the definition of the RKHS.

From the Sample Operator Equation to a Gram System

The empirical solution lies in the span of the observed kernel sections. Indeed, its equation can be rearranged as $\lambda\hat f_\lambda=g_n-C_n\hat f_\lambda$, whose right-hand side belongs to that span. Write a function in the span as $f=\sum_{j=1}^n\alpha_j k_{X_j}$, and let $K$ be the Gram matrix with entries $K_{ij}=k(X_i,X_j)$. The finite-dimensional system associated with the operator equation is

(K+n\lambda I_n)\alpha=y, \qquad y=(Y_1,\ldots,Y_n)^\top.

This matrix is positive definite for $\lambda>0$, even if $K$ is singular: for every nonzero $v\in\mathbb R^n$, $v^\top(K+n\lambda I_n)v=v^\top Kv+n\lambda\|v\|_2^2>0$. So the Gram system has a unique coefficient vector. Its represented function is the empirical operator solution.

Worked Example: A Two-Point Population Operator

Take $\mathcal H=\mathbb R^2$, with feature vectors $k_{x_1}=e_1$ and $k_{x_2}=e_2$, where $e_1=(1,0)^\top$ and $e_2=(0,1)^\top$. Let $P(X=x_1)=3/4$, $P(X=x_2)=1/4$, and let $Y=2$ at $x_1$ and $Y=-1$ at $x_2$.

The feature covariance and response-feature vector are $C=(3/4)e_1e_1^\top+(1/4)e_2e_2^\top=\operatorname{diag}(3/4,1/4)$ and $g=(3/4)(2)e_1+(1/4)(-1)e_2=(3/2,-1/4)^\top$. For $\lambda=1/2$, the population equation is diagonal:

\begin{pmatrix}5/4&0\\0&3/4\end{pmatrix} \begin{pmatrix}f_{\lambda,1}\\f_{\lambda,2}\end{pmatrix} = \begin{pmatrix}3/2\\-1/4\end{pmatrix}.

Dividing coordinatewise gives $f_{\lambda,1}=(3/2)/(5/4)=6/5$ and $f_{\lambda,2}=(-1/4)/(3/4)=-1/3$. Thus the population regularized function takes values $6/5$ at $x_1$ and $-1/3$ at $x_2$. The unregularized response values are $2$ and $-1$; here regularization shrinks each coordinate by a factor determined by its population feature frequency.

Worked Example: Duplicate Features and a Singular Gram Matrix

Suppose $n=2$, both observations have the same kernel section $k_{X_1}=k_{X_2}=k$, with $\|k\|_{\mathcal H}^2=1$, and $Y_1=Y_2=1$. Take $\lambda=1$. Then $K$ is the all-ones matrix, which is singular, and $C_n f=k\langle k,f\rangle_{\mathcal H}$, while $g_n=k$.

The operator solution is $\hat f_\lambda=k/2$, since $C_n(k/2)+k/2=k/2+k/2=k=g_n$. The Gram system is $\begin{pmatrix}3&1\\1&3\end{pmatrix}\alpha=(1,1)^\top$. Its unique solution is $\alpha=(1/4,1/4)^\top$, which represents $(1/4)k+(1/4)k=k/2$.

But $\alpha'=(1/2,0)^\top$ also represents the same function $k/2$. It does not solve the Gram system, because $\begin{pmatrix}3&1\\1&3\end{pmatrix}\alpha'=(3/2,1/2)^\top\neq(1,1)^\top$. Thus uniqueness of the Gram-system coefficients does not imply that every coefficient vector representing the operator solution solves that system.

Why the Gram-System Qualification Matters

For any coefficient vector $\alpha$, let $\Phi\alpha=\sum_j\alpha_j k_{X_j}$. Substituting $f=\Phi\alpha$ into the empirical operator equation shows that it is a solution exactly when

\Phi\big((K+n\lambda I_n)\alpha-y\big)=0.

The equation says that the residual coefficient vector is in the null space of the feature map, not necessarily that the coefficient vector is zero. If the Gram matrix is singular, distinct coefficient vectors can map to the same function. However, the Gram system itself always has a unique solution $\beta$, and the function $\Phi\beta$ satisfies the operator equation. By uniqueness of the operator solution, it equals $\hat f_\lambda$. Therefore an arbitrary representation of $\hat f_\lambda$ can be replaced by the Gram-system solution without changing the represented function.

This distinction is useful in both theory and computation. Operator equations express the estimator independently of coordinates; Gram systems provide a finite-dimensional way to calculate it from the sample. The regularization term ensures that the Gram system is invertible, even when observations have repeated inputs or linearly dependent kernel sections. It does not make representations of a function unique.

Common Mistake

Wrong reasoning: “The operator solution can be written as $\sum_j\alpha_jk_{X_j}$, so any such coefficient vector must satisfy $(K+n\lambda I_n)\alpha=y$.” This treats a function representation as if its coefficients were unique.
Correction: The operator equation implies only that $\Phi((K+n\lambda I_n)\alpha-y)=0$. When $K$ is singular, the feature map can have a nontrivial null space. The unique Gram-system solution represents the operator solution, but another coefficient vector may represent the same function without solving the system.

Key takeaway: The population solution solves $(C+\lambda I)f_\lambda=g$, while the random sample solution solves $(C_n+\lambda I)\hat f_\lambda=g_n$. The Gram system has unique coefficients for $\lambda>0$, but function representations in the observed kernel sections need not be unique.

Check Your Understanding

Answer each question using the operator and function distinctions developed above.

What integrability condition ensures that the population feature covariance operator is well-defined as a bounded operator?
Why is $C+\lambda I$ invertible for $\lambda>0$, even if $C$ has a nontrivial null space?
Which quantities in the empirical operator equation are random under independent sampling?
Why does adding $n\lambda I_n$ make the Gram matrix system uniquely solvable even when $K$ is singular?
In the duplicate-feature example, why does $(1/2,0)^\top$ represent the operator solution but fail to solve the Gram system?