Kernel Density Estimators and Kernel Order

From Histograms to Kernel Smoothing

A histogram estimates a density by assigning observations to bins. Its shape can depend noticeably on where bin boundaries fall. Kernel density estimation replaces each observation’s bin contribution with a smooth, rescaled kernel, then adds the contributions. In multiple dimensions, each observation contributes a local surface over the feature space rather than a one-dimensional curve.

Two choices govern different aspects of the estimate. The bandwidth sets the spatial scale over which observations are spread. The kernel determines the shape of that spread and, through its moments, which low-degree patterns it averages away. These choices should not be conflated: changing bandwidth changes the scale, while changing kernel order changes moment cancellation.

The Multivariate Kernel Density Estimator

Let \(\mathbf X_1,\ldots,\mathbf X_n\) be independent observations from a density \(f\) on \(\mathbb R^d\). Let \(K:\mathbb R^d\to\mathbb R\) be a measurable kernel and let \(h>0\) be a scalar bandwidth. Write \(\mathbf x=(x_1,\ldots,x_d)\) for a point at which the density is estimated.

Definition: The multivariate kernel density estimator with kernel \(K\) and bandwidth \(h\) is

\hat f_h(\mathbf x) = \frac{1}{n h^d}\sum_{i=1}^n K\!\left(\frac{\mathbf x-\mathbf X_i}{h}\right).

The factor \(h^{-d}\) rescales the kernel in \(d\) dimensions so that its total integral is preserved. A kernel is normalized when \(\int_{\mathbb R^d}K(\mathbf u)\,d\mathbf u=1\).

For a normalized kernel, the contribution centered at \(\mathbf X_i\) is \(h^{-d}K((\mathbf x-\mathbf X_i)/h)\). Its spatial scale is \(h\): smaller \(h\) concentrates each contribution near its observation, while larger \(h\) spreads it over a wider region. If \(K\) is nonnegative, the estimate is nonnegative as well. Normalization and nonnegativity together make the estimate a density, provided the observations are fixed and the integral is taken over all of \(\mathbb R^d\).

Worked Example: A Two-Dimensional Box Kernel Estimate

Use the product box kernel \(K(u_1,u_2)=1/4\) when both coordinates lie in \([-1,1]\), and \(K(u_1,u_2)=0\) otherwise. Take \(n=4\), \(h=0.5\), and observations \((0.1,0.2)\), \((-0.4,0.3)\), \((0.7,0)\), and \((0,-0.6)\). Calculate the estimate at \(\mathbf x=(0,0)\).

The scaled kernel is nonzero when both coordinates of \((\mathbf x-\mathbf X_i)/h\) are between \(-1\) and \(1\). Equivalently, each observation must be within \(0.5\) of \(\mathbf x\) in both coordinates. The first two observations satisfy this condition; the third fails because \(0.7>0.5\), and the fourth fails because \(0.6>0.5\).

For each of the first two observations, the kernel argument is inside the box, so its contribution before averaging is \(h^{-2}K=0.5^{-2}(1/4)=1\). The other contributions are zero. Therefore

\hat f_{0.5}(0,0) = \frac{1}{4}(1+1+0+0) = 0.5.

The estimate is a density value, not a probability. Its value can exceed one; probabilities arise by integrating the estimate over regions.

Normalization and Integrability

Normalization is the condition that each observation contributes total mass one before the contributions are averaged. Nonnegativity is a separate property: a signed kernel can be normalized while taking negative values. Integrability conditions specify whether the relevant mass and moments are well-defined.

Kernel conditions: A normalized integrable kernel satisfies \(K\in L^1(\mathbb R^d)\) and \(\int K(\mathbf u)\,d\mathbf u=1\). A nonnegative normalized kernel also satisfies \(K(\mathbf u)\geq 0\) almost everywhere. For variance calculations, the usual additional condition is square integrability, \(\int K(\mathbf u)^2\,d\mathbf u<\infty\).

The \(L^1\) condition ensures absolute integrability of the kernel. The \(L^2\) condition is distinct: it controls the squared size of the kernel and is important in variance analysis. Neither condition alone implies the other for every possible function. In common bounded-support examples, both hold. For a nonnegative normalized kernel, integrating the estimate gives exactly one.

Result: If \(K\) is nonnegative, measurable, and normalized, then for any fixed observations and \(h>0\), \(\hat f_h\) is a density on \(\mathbb R^d\).

To verify the result, integrate each summand and use the change of variables \(\mathbf u=(\mathbf x-\mathbf X_i)/h\), so \(d\mathbf x=h^d\,d\mathbf u\). Nonnegativity permits summing and integrating by Tonelli’s theorem:

\begin{aligned} \int_{\mathbb R^d}\hat f_h(\mathbf x)\,d\mathbf x &= \frac{1}{n}\sum_{i=1}^n \int_{\mathbb R^d} h^{-d} K\!\left(\frac{\mathbf x-\mathbf X_i}{h}\right)d\mathbf x\\ &= \frac{1}{n}\sum_{i=1}^n \int_{\mathbb R^d}K(\mathbf u)\,d\mathbf u =1. \end{aligned}

Every summand is nonnegative, so the estimate is nonnegative, and the calculation shows its total integral is one. If a normalized kernel is signed, the same integral calculation remains valid when \(K\in L^1\), but the estimate need not be nonnegative and therefore need not be a density in the usual sense. \(\square\)

Kernel Moments and Kernel Order

A kernel averages the density around the target point. Its moments describe how that averaging responds to simple shapes. For a multi-index \(\boldsymbol\alpha=(\alpha_1,\ldots,\alpha_d)\) of nonnegative integers, write \(|\boldsymbol\alpha|=\sum_{j=1}^d\alpha_j\) and \(\mathbf u^{\boldsymbol\alpha}=\prod_{j=1}^d u_j^{\alpha_j}\). The corresponding moment is \(\int \mathbf u^{\boldsymbol\alpha}K(\mathbf u)\,d\mathbf u\), when it exists.

Definition: A normalized kernel has order \(q\), for a positive integer \(q\), if its absolute \(q\)th moment is finite, \(\int \|\mathbf u\|^q |K(\mathbf u)|\,d\mathbf u<\infty\); every moment \(\int \mathbf u^{\boldsymbol\alpha}K(\mathbf u)\,d\mathbf u\) with \(1\leq|\boldsymbol\alpha|<q\) is zero; and at least one moment with \(|\boldsymbol\alpha|=q\) is nonzero.

The absolute-moment condition makes the required moments finite, including those of lower degree. The normalization condition supplies the degree-zero moment, equal to one. A kernel of order two, for example, has zero first moments but at least one nonzero second moment. Those first-moment cancellations prevent local linear trends from shifting the average in the same way they would under an asymmetric kernel. In general, vanishing lower moments cancel the corresponding polynomial components in the local averaging operation.

Kernel order is not a measure of how wide the kernel is. A fixed kernel retains its order when \(h\) changes; bandwidth rescales the same kernel. Moreover, the usual nonnegative symmetric kernels have order two, while higher orders typically require signed kernels. The moment definition describes these cancellations without implying that a higher-order estimate must remain nonnegative.

Result: If \(K\) has finite absolute moments through degree \(q\), its scaled version \(K_h(\mathbf v)=h^{-d}K(\mathbf v/h)\) has moment

\int_{\mathbb R^d}\mathbf v^{\boldsymbol\alpha}K_h(\mathbf v)\,d\mathbf v = h^{|\boldsymbol\alpha|} \int_{\mathbb R^d}\mathbf u^{\boldsymbol\alpha}K(\mathbf u)\,d\mathbf u.

For the proof, substitute \(\mathbf v=h\mathbf u\), giving \(d\mathbf v=h^d\,d\mathbf u\), and use \(\mathbf v^{\boldsymbol\alpha}=h^{|\boldsymbol\alpha|}\mathbf u^{\boldsymbol\alpha}\):

\begin{aligned} \int \mathbf v^{\boldsymbol\alpha}h^{-d}K(\mathbf v/h)\,d\mathbf v &= \int (h\mathbf u)^{\boldsymbol\alpha}h^{-d}K(\mathbf u)h^d\,d\mathbf u\\ &= h^{|\boldsymbol\alpha|} \int\mathbf u^{\boldsymbol\alpha}K(\mathbf u)\,d\mathbf u. \end{aligned}

Thus scaling preserves which moments are zero, while multiplying a nonzero moment of degree \(r\) by \(h^r\). This is the precise sense in which bandwidth sets the scale of the kernel’s moment effects. \(\square\)

Worked Example: Checking the Order of a Product Box Kernel

For the two-dimensional box kernel from the earlier calculation, verify normalization, find its first and second moments, and identify its order.

The kernel is constant at \(1/4\) on the square \([-1,1]^2\), whose area is \(4\). Hence its integral is \((1/4)4=1\). Symmetry about each coordinate axis makes both first moments zero:

\int u_1K(\mathbf u)\,d\mathbf u = \int u_2K(\mathbf u)\,d\mathbf u =0.

For the first coordinate’s second moment, separate the integrals across the square:

\int u_1^2K(\mathbf u)\,d\mathbf u = \frac14 \left(\int_{-1}^1u_1^2\,du_1\right) \left(\int_{-1}^1du_2\right) = \frac14\left(\frac23\right)(2) = \frac13.

By symmetry, \(\int u_2^2K(\mathbf u)\,d\mathbf u=1/3\), so a second-degree moment is nonzero. The kernel is nonnegative, has bounded support and thus finite absolute moments of every order, and has zero first moments. It is therefore of order two. Its moments after scaling by bandwidth \(h\) are \(h^2/3\) for each coordinate’s second moment.

Worked Kernel Checks and the Smoothing Interpretation

Worked Example: A Gaussian Kernel and Its Smoothed Target

Take the product standard Gaussian kernel \(K(\mathbf u)=(2\pi)^{-1}\exp(-\|\mathbf u\|^2/2)\) in two dimensions. Check its normalization and order, then find the expectation of its estimator at the origin when \(f\) is the standard bivariate normal density and \(h=0.5\).

The kernel factors into two standard normal densities, each integrating to one, so its two-dimensional integral is one. Each coordinate has mean zero and variance one. The first moments vanish and the coordinate second moments are one, so the kernel has order two. It is nonnegative, and its Gaussian tails give finite absolute moments and a finite integral of \(K^2\).

The expectation of the estimator is the convolution of \(f\) with the scaled kernel. Here the scaled kernel is the density of \(N(0,h^2I_2)\); adding independent normal vectors gives a normal vector with covariance \((1+h^2)I_2\). At \(h=0.5\), this covariance is \(1.25I_2\), and the expected estimate at the origin is

E[\hat f_{0.5}(0,0)] = \frac{1}{2\pi(1+0.5^2)} = \frac{1}{2.5\pi} \approx 0.1273.

The true density at the origin is \(f(0,0)=1/(2\pi)\approx0.1592\). In this example, smoothing spreads the density and lowers its central value. This calculation illustrates the effect of bandwidth; it does not say that smoothing always lowers the estimate at every point.

Worked Example: A Compactly Supported Kernel in Two Dimensions

Consider \(K(\mathbf u)=(2/\pi)(1-\|\mathbf u\|^2)\) for \(\|\mathbf u\|\leq1\), and zero outside the unit disk. Verify normalization, a second moment, and square integrability.

Using polar coordinates, the integral of the kernel is

\int K(\mathbf u)\,d\mathbf u = \frac{2}{\pi}2\pi\int_0^1r(1-r^2)\,dr = 4\left(\frac12-\frac14\right) = 1.

Rotational symmetry gives zero first moments. For the first coordinate’s second moment, the angular integral of \(\cos^2(\theta)\) from \(0\) to \(2\pi\) is \(\pi\):

\begin{aligned} \int u_1^2K(\mathbf u)\,d\mathbf u &= \frac{2}{\pi}\pi\int_0^1r^3(1-r^2)\,dr\\ &= 2\left(\frac14-\frac16\right) = \frac16. \end{aligned}

The second moment is nonzero, so the kernel has order two. It is bounded and supported on a disk of finite area, which ensures finite absolute moments and square integrability. In particular,

\int K(\mathbf u)^2\,d\mathbf u = \frac{4}{\pi^2}2\pi\int_0^1r(1-r^2)^2\,dr = \frac{8}{\pi}\cdot\frac16 = \frac{4}{3\pi}.

Expectation as a Local Average

The connection between kernel moments and smoothing can be stated directly through the estimator’s expectation. If \(K\in L^1\), \(f\) is a density, and the integral below is finite at \(\mathbf x\), independence and the density formula give

E[\hat f_h(\mathbf x)] = \int_{\mathbb R^d}K(\mathbf u)f(\mathbf x-h\mathbf u)\,d\mathbf u.

Indeed, the expectation of one summand is \(\int h^{-d}K((\mathbf x-\mathbf z)/h)f(\mathbf z)\,d\mathbf z\). Substituting \(\mathbf u=(\mathbf x-\mathbf z)/h\) transforms this to the displayed local average; averaging \(n\) identical expectations leaves it unchanged. For signed kernels, absolute integrability is needed to justify the substitution and interchange. If \(f\) is bounded and \(K\in L^1\), the integral is absolutely finite.

This formula explains why normalization matters: a constant density remains unchanged by averaging with a normalized kernel. It also explains why moment conditions matter: they determine which polynomial trends are canceled in the local average. Precise bias expansions require additional smoothness assumptions and are a separate calculation. For the present purpose, the useful distinction is that bandwidth controls the radius of averaging, while order controls moment cancellation.

Common Pitfalls

Using \(h^{-1}\) in every dimension. The volume scaling is \(h^d\), so the estimator uses \(h^{-d}\). In two dimensions with \(h=0.5\), the factor is \(0.5^{-2}=4\), not \(2\).
Assuming normalization implies nonnegativity. The integral of a signed normalized kernel is one, but its estimate can be negative in some regions. Nonnegativity must be checked separately.
Calling every symmetric kernel “higher order.” Symmetry usually makes first moments zero, which gives order two when second moments exist and are nonzero. It does not by itself establish cancellation of all moments through a larger degree.
Treating bandwidth and order as interchangeable. A smaller bandwidth narrows the same kernel; it does not change which moments vanish. A kernel’s order is a property of its shape and moment conditions.
Ignoring integrability when discussing moments or variance. A moment must exist before it can be used in an order definition, and square integrability is a separate condition relevant to variance calculations.

Key takeaway: A multivariate kernel estimator averages rescaled contributions from the observations. Normalization preserves total mass, nonnegativity makes the estimate a density, integrability ensures the required calculations exist, and kernel moments determine order. Bandwidth sets the spatial scale of smoothing.

Check Your Understanding

Use the definitions and calculations above to check your understanding.

Why does a \(d\)-dimensional kernel estimate use the scaling factor \(h^{-d}\)?
What additional property, beyond normalization, ensures that a kernel estimate is nonnegative?
For a kernel of order two, which moments must vanish, and what must be true of at least one second-degree moment?
Does reducing the bandwidth change a kernel’s order? Explain the role of the bandwidth-scaling moment identity.
Why is square integrability a separate condition from normalization, and in what later calculation is it useful?