Kernel Bias Expansions

Why Kernel Moments Determine Bias

A kernel density estimator smooths the data over a bandwidth \(h\). This smoothing reduces random fluctuation in many settings, but it also shifts the estimator's expectation away from the target density. The size and shape of that shift are governed by two ingredients: how smoothly the density changes and which moments of the kernel vanish.

The key calculation is to expand the density near a point \(x\), then average that expansion against the kernel. If the kernel's first several moments are zero, the corresponding Taylor terms cancel. The first surviving moment determines the leading bias. Integrating the squared bias gives a related expansion, but its justification requires control of the remainder in an integrated norm, not just at each fixed point.

Pointwise Bias Expansion

For an independent sample \(X_1,\ldots,X_n\) from a density \(f\), the kernel density estimator is \(\hat f_h(x)=(nh)^{-1}\sum_{i=1}^n K((x-X_i)/h)\). Here \(K\) is an integrable kernel with unit integral, and \(h>0\) is the bandwidth. A change of variable in the expectation gives \(E[\hat f_h(x)]=\int K(u)f(x-hu)\,du\). Thus its pointwise bias is the difference between a local average of \(f\) and \(f(x)\).

The following pointwise result gives the leading bias and its sign. One convenient set of sufficient conditions is that \(f\) has bounded derivatives through order \(r\) on the real line and \(f^{(r)}\) is continuous at the point \(x\). The kernel must have order \(r\) and finite absolute \(r\)th moment.

Proof. Taylor's formula with an integral remainder gives \(f(x-hu)=\sum_{j=0}^{r-1}(-hu)^j f^{(j)}(x)/j!+R_r(x,h,u)\), where the remainder can be written \(R_r(x,h,u)=(-hu)^r\int_0^1(1-s)^{r-1}f^{(r)}(x-shu)\,ds/(r-1)!\). For each fixed \(u\), continuity of \(f^{(r)}\) at \(x\) implies \(R_r(x,h,u)/h^r\to (-u)^r f^{(r)}(x)/r!\). Boundedness of \(f^{(r)}\) also gives \(|R_r(x,h,u)|/h^r\leq C|u|^r\) for a constant \(C\) independent of \(h,u\). Since \(\int |u|^r|K(u)|\,du<\infty\), dominated convergence allows integration of the scaled remainder against \(K\). In the Taylor polynomial, the constant term integrates to \(f(x)\); each term of degrees \(1,\ldots,r-1\) vanishes because its kernel moment is zero. The degree-\(r\) contribution from the remainder limit is \((-1)^r\mu_r(K)f^{(r)}(x)/r!\). Multiplication by \(h^r\) proves the expansion. \(\square\)

The sign \((-1)^r\) comes from the argument \(x-hu\). For an even-order kernel it is positive, while for an odd-order kernel it is negative. The pointwise bias is generally of order \(h^r\), provided \(f^{(r)}(x)\) and \(\mu_r(K)\) are nonzero. At points where \(f^{(r)}(x)=0\), the leading term vanishes, so the bias may be smaller there.

Integrated Squared-Bias Expansion

The integrated squared bias (ISB) measures the squared bias across the whole real line: \(\operatorname{ISB}(h)=\int (E[\hat f_h(x)]-f(x))^2\,dx\). To obtain its leading order, a pointwise expansion alone is insufficient: a remainder that is small at every fixed \(x\) need not be small after integration over all \(x\). A useful sufficient smoothness condition is \(f\in W^{r,2}(\mathbb R)\): \(f\) and its weak derivatives through order \(r\) are square-integrable. Assume also that \(K\) has order \(r\) and finite absolute \(r\)th moment.

Proof. The expected estimate can be written as \(E[\hat f_h(\cdot)]=\int K(u)f(\cdot-hu)\,du\). For \(f\in W^{r,2}\), Taylor expansion of translations holds in \(L^2\): \(f(\cdot-t)=\sum_{j=0}^{r-1}(-t)^j f^{(j)}/j!+R_r(t)\), with \(R_r(t)=(-t)^r\int_0^1(1-s)^{r-1}f^{(r)}(\cdot-st)\,ds/(r-1)!\). Translations are continuous in \(L^2\), so for each fixed \(u\), \(R_r(hu)/h^r\to (-u)^r f^{(r)}/r!\) in \(L^2\). The integral formula also gives \(\|R_r(hu)\|_2/h^r\leq |u|^r\|f^{(r)}\|_2/r!\). The right-hand side is integrable against \(|K(u)|\). Dominated convergence for this \(L^2\)-valued integral therefore gives convergence of the integrated remainder in \(L^2\). Integrating the Taylor polynomial against \(K\), its terms of degrees \(1,\ldots,r-1\) vanish and its constant term cancels \(f\). This proves the stated \(L^2\) bias expansion.

Write \(c=(-1)^r\mu_r(K)/r!\). The expansion says \(B_h=h^r c f^{(r)}+q_h\), where \(\|q_h\|_2=o(h^r)\). Hence \(\|B_h\|_2^2=h^{2r}c^2\|f^{(r)}\|_2^2+2h^r c\langle f^{(r)},q_h\rangle+\|q_h\|_2^2\). By Cauchy–Schwarz, the middle term is \(o(h^{2r})\), and the last term is also \(o(h^{2r})\). Squaring removes the sign of \(c\), yielding the ISB formula. \(\square\)

Thus the pointwise bias has leading order \(h^r\), while the integrated squared bias has leading order \(h^{2r}\). These are bias orders, not statements about the estimator's total risk: variance contributes separately.

What “Integrated Bias” Means

The signed bias integrates to zero under the usual normalization assumptions; it is not the same quantity as integrated squared bias. If \(f\) is a density, \(K\) is absolutely integrable with \(\int K=1\), and the integrals are justified by Fubini's theorem, then \(\int E[\hat f_h(x)]\,dx=\int K(u)\int f(x-hu)\,dx\,du=1\). Since \(\int f(x)\,dx=1\), it follows that \(\int B_h(x)\,dx=0\). Positive and negative deviations cancel. By contrast, \(\int B_h(x)^2dx\) does not cancel and captures the integrated squared departure from the target.

A common mistake is to infer the ISB order by integrating the pointwise bias itself, or to square a pointwise remainder without checking its behavior across \(x\). The signed integral may be exactly zero, while the ISB is typically of order \(h^{2r}\). The \(L^2\) expansion above supplies the needed control of the remainder; pointwise convergence alone does not.