Integrated Variance of Kernel Estimators

Why Integrated Variance Has a Dimension-Dependent Scale

A kernel density estimate averages local contributions from the observations. Narrowing its bandwidth makes each contribution more concentrated, which increases variability; increasing the dimension makes that concentration stronger still. The integrated variance quantifies this variability across the whole density domain, rather than at one fixed point.

The key calculation is precise: under standard square-integrability conditions, the exact integrated variance has a leading term $R(K)/(n h^d)$. Here $R(K)$ is the integral of the squared kernel. The exact expression also contains a finite-sample correction, so it is important to distinguish the leading order from an equality that holds for every bandwidth.

Definition: Let $X_1,\ldots,X_n$ be independent and identically distributed random vectors in $\mathbb{R}^d$ with density $f$. For a measurable kernel $K:\mathbb{R}^d\to\mathbb{R}$ and bandwidth $h>0$, define $K_h(u)=h^{-d}K(u/h)$ and the kernel density estimator $$ \hat f_h(x)=\frac{1}{n}\sum_{i=1}^n K_h(x-X_i) =\frac{1}{n h^d}\sum_{i=1}^n K\left(\frac{x-X_i}{h}\right). $$ If the integrals are finite, the integrated variance is $\int_{\mathbb{R}^d}\operatorname{Var}(\hat f_h(x))\,dx$.

For the results below, assume $f$ is a density in $L^2(\mathbb{R}^d)$, and $K$ is measurable, belongs to both $L^1(\mathbb{R}^d)$ and $L^2(\mathbb{R}^d)$, and satisfies $\int K(u)\,du=1$. Define the squared-kernel integral by

R(K)=\int_{\mathbb{R}^d}K(u)^2\,du.

The normalization ensures that the kernel has unit integral; it does not imply $R(K)=1$. In fact, $R(K)>0$, since a function with integral one cannot be zero almost everywhere. A common nonnegative kernel is a special case, but the calculations below also apply to signed kernels satisfying these integrability conditions.

The Exact Integrated-Variance Identity

Write $m_h(x)=E[\hat f_h(x)]$. Independence gives the pointwise variance as the variance of one kernel contribution divided by $n$. Integrating that variance yields an exact identity, not just an asymptotic approximation.

Theorem: Under the assumptions above, for every $h>0$, $$ \int_{\mathbb{R}^d}\operatorname{Var}(\hat f_h(x))\,dx = \frac{R(K)}{n h^d} -\frac{1}{n}\int_{\mathbb{R}^d}m_h(x)^2\,dx. $$ Equivalently, since $m_h=K_h*f$, the second term is $-\|K_h*f\|_2^2/n$.

Proof. Let $Z_i(x)=K_h(x-X_i)$, so that $\hat f_h(x)=n^{-1}\sum_i Z_i(x)$. For each $x$, the $Z_i(x)$ are independent and identically distributed, and therefore

\operatorname{Var}(\hat f_h(x)) =\frac{1}{n}\left(E[Z_1(x)^2]-(E[Z_1(x)])^2\right).

The nonnegative integrand in the first expectation permits Tonelli's theorem. Changing variables $v=(x-u)/h$ gives

\begin{aligned} \int E[Z_1(x)^2]\,dx &=\int\int K_h(x-u)^2 f(u)\,du\,dx\\ &=\int f(u)\left(\int h^{-2d}K\left(\frac{x-u}{h}\right)^2dx\right)du\\ &=\int f(u)\,du\; h^{-d}R(K) =h^{-d}R(K). \end{aligned}

Also, $E[Z_1(x)]=m_h(x)$. The convolution $K_h*f$ belongs to $L^2$: Young's inequality gives $\|K_h*f\|_2\leq \|K_h\|_1\|f\|_2=\|K\|_1\|f\|_2$. Thus the integral of $m_h^2$ is finite. Integrating the pointwise variance identity and substituting the two integrals proves the result. $\square$

Worked Example: Exact Correction for a Gaussian Density

Let $d=1$, let $f$ be the $N(0,1)$ density, and use the standard normal density as $K$. Then $R(K)=1/(2\sqrt{\pi})$. The expectation $m_h=K_h*f$ is the density of $N(0,1+h^2)$, because it is the distribution of the sum of independent $N(0,1)$ and $N(0,h^2)$ variables.

For a $N(0,v)$ density $g$, direct integration gives $\int g(x)^2dx=1/(2\sqrt{\pi v})$. Consequently,

\int m_h(x)^2dx=\frac{1}{2\sqrt{\pi}\sqrt{1+h^2}}, \qquad \operatorname{IV}(\hat f_h) = \frac{1}{2n\sqrt{\pi}} \left(\frac{1}{h}-\frac{1}{\sqrt{1+h^2}}\right).

For $n=100$ and $h=0.2$, the leading term is $1/(2\sqrt{\pi}\cdot20)$, approximately $0.01410$. The correction is $1/(200\sqrt{\pi}\sqrt{1.04})$, approximately $0.00277$. The exact integrated variance is therefore approximately $0.01134$. The correction is not negligible at this bandwidth, even though it is smaller than the leading term.

Leading Order as the Bandwidth Shrinks

The exact identity separates a term that grows like $h^{-d}/n$ from a correction that is at most of order $1/n$. This establishes why the squared-kernel integral and the dimension appear in the leading expression.

Theorem: Suppose $f\in L^2(\mathbb{R}^d)$, $K\in L^1(\mathbb{R}^d)\cap L^2(\mathbb{R}^d)$, and $\int K=1$. For any sequence $h=h_n\to0$, $$ \int\operatorname{Var}(\hat f_{h_n}(x))\,dx = \frac{R(K)}{n h_n^d}\{1+o(1)\}. $$ More specifically, the difference between the exact integrated variance and $R(K)/(n h_n^d)$ has absolute value at most $\|K\|_1^2\|f\|_2^2/n$.

Proof. The exact identity and Young's inequality imply

0\leq\frac{1}{n}\|K_h*f\|_2^2 \leq\frac{\|K\|_1^2\|f\|_2^2}{n}.

The correction is thus bounded by a constant times $1/n$. Relative to the positive leading term $R(K)/(n h^d)$, its magnitude is at most

\frac{h^d\|K\|_1^2\|f\|_2^2}{R(K)}.

Since $d\geq1$ and $h\to0$, this ratio tends to zero. Dividing the exact identity by its leading term proves the asserted relative approximation, and the same bound proves the stated absolute bound. $\square$

Worked Example: A Product Kernel in Three Dimensions

Consider the product uniform kernel $$ K(u_1,u_2,u_3)=\frac{1}{8}\mathbf{1}\{|u_j|\leq1\text{ for }j=1,2,3\}. $$ Its integral is $(1/8)(2^3)=1$. Its squared-kernel integral is

R(K)=\int_{[-1,1]^3}\frac{1}{64}\,du =\frac{8}{64}=\frac{1}{8}.

With $n=500$ and $h=0.1$, the leading integrated variance is $$ \frac{R(K)}{n h^3} =\frac{1/8}{500(0.1)^3} =\frac{1/8}{0.5} =0.25. $$ The cubic power of the bandwidth matters: $h^3=0.001$, not $h$. The exact integrated variance subtracts the nonnegative term $\|K_h*f\|_2^2/n$, so $0.25$ is the leading approximation, not necessarily the exact value.

What the Squared-Kernel Integral Contributes

At fixed $n$, $h$, and dimension, the leading variance is proportional to $R(K)$. Kernels with greater squared integral produce a larger leading integrated variance under this comparison. This is one reason the kernel's shape matters even when several kernels have unit integral.

Worked Example: Comparing Two One-Dimensional Kernels

The uniform kernel on $[-1,1]$ is $K_U(u)=\tfrac12\mathbf{1}\{|u|\leq1\}$, so $$ R(K_U)=\int_{-1}^1\frac14\,du=\frac12. $$ The Epanechnikov kernel is $K_E(u)=\tfrac34(1-u^2)\mathbf{1}\{|u|\leq1\}$. Its squared integral is

\begin{aligned} R(K_E) &=\frac{9}{16}\int_{-1}^1(1-2u^2+u^4)\,du\\ &=\frac{9}{16}\left(2-\frac{4}{3}+\frac{2}{5}\right) =\frac{9}{16}\cdot\frac{16}{15} =\frac35. \end{aligned}

For $n=100$ and $h=0.1$, the leading term is $R(K)/(nh)$. It equals $0.5/(10)=0.05$ for the uniform kernel and $0.6/(10)=0.06$ for the Epanechnikov kernel. This comparison concerns the leading variance term only; it does not by itself compare total estimation risk.

Common Pitfall and Interpretation

A frequent mistake is to treat $R(K)/(n h^d)$ as the exact integrated variance. The exact expression contains the subtraction $\|K_h*f\|_2^2/n$; omitting it is justified for the leading order as $h\to0$, but not automatically for a particular finite bandwidth. The Gaussian example showed this distinction numerically.

A second mistake is to use $h^{-1}$ in every dimension. The factor $h^{-d}$ follows from rescaling volume in $\mathbb{R}^d$: under $v=(x-u)/h$, the differential changes by $dx=h^d\,dv$, while the squared kernel contributes $h^{-2d}$. Their product is $h^{-d}$. Thus the leading integrated variance grows more rapidly as dimension increases, for the same scalar bandwidth.

The result isolates the variance contribution to integrated squared error. It does not, on its own, specify an appropriate bandwidth: that requires considering bias as well as variance. Nor does a smaller value of $R(K)$ alone establish that one kernel is preferable for every estimation objective.

Key takeaway: For a normalized square-integrable kernel and an $L^2$ density, the exact integrated variance is $R(K)/(nh^d)-\|K_h*f\|_2^2/n$. As $h\to0$, the second term is lower order, leaving the leading scale $R(K)/(nh^d)$.

Check Your Understanding

Use the definitions and derivations above to answer these questions.

Why does integrating the squared scaled kernel $K_h^2$ produce the factor $h^{-d}R(K)$?
For a two-dimensional kernel with $R(K)=0.4$, $n=200$, and $h=0.2$, what is the leading integrated-variance term?
What finite-sample term is subtracted from $R(K)/(nh^d)$ in the exact identity?
Why is the relative size of that correction negligible as $h\to0$, under the stated assumptions?
Two normalized kernels have squared-kernel integrals $0.3$ and $0.5$. With the same sample size, bandwidth, and dimension, which has the smaller leading integrated variance?