Nonparametric Density Estimation · Tutorial 608 of 989

Pointwise Minimax Rates for Density Values

See how local smoothness, kernel bias and variance, and a two-point lower bound combine to determine the optimal rate for estimating one density value.

Intermediate 12 min read

What You'll Learn

Define pointwise squared-error risk and the Hölder density class used for the rate calculation
Derive a kernel estimator’s pointwise bias and variance bounds
Balance squared bias against variance to obtain the optimal bandwidth and rate
Construct localized density alternatives that yield a matching minimax lower bound
Apply a two-point testing argument and identify a common rate-calculation error

Why Estimating One Density Value Is a Local Problem

Suppose $X_1,\ldots,X_n$ are independent observations from a density $f$ on $[0,1]$. The target here is one number, $f(x_0)$, where $x_0$ is fixed and lies strictly inside the interval. The loss is squared error at that point: an estimator $\hat T$ has risk $E_f[(\hat T-f(x_0))^2]$. Because the target concerns a single location, the useful information is concentrated near $x_0$; how much can be pooled from that neighborhood depends on the smoothness of $f$.

The goal is a minimax rate: the order of the smallest possible worst-case risk over a specified class of densities. We will show that, for a one-dimensional Hölder class of smoothness $\beta$, the rate is $n^{-2\beta/(2\beta+1)}$. The upper bound comes from balancing the squared bias and variance of a kernel estimator. The lower bound shows that no estimator can improve the rate uniformly over the class.

Risk, Smoothness, and the Target Class

Definition: For a class of densities $\mathcal F$, the pointwise minimax squared-error risk at $x_0$ is $$ R_n(\mathcal F,x_0) = \inf_{\hat T}\sup_{f\in\mathcal F} E_f\!\left[(\hat T-f(x_0))^2\right], $$ where the infimum is over measurable estimators based on $X_1,\ldots,X_n$.

For precision, take a noninteger smoothness $\beta>0$, write $\beta=k+\alpha$ with integer $k\geq 0$ and $0<\alpha<1$, and use the following Hölder condition. Each $f\in\mathcal F$ is a density on $[0,1]$, has continuous derivatives through order $k$ (when $k=0$, this just means $f$ is continuous), satisfies $\sup_x|f^{(j)}(x)|\leq L$ for $0\leq j\leq k$, and satisfies $|f^{(k)}(x)-f^{(k)}(y)|\leq L|x-y|^\alpha$. Assume $L>1$, and take a Hölder ball large enough to contain the small perturbations used below. Also assume $x_0\in(0,1)$ is fixed. These conditions make $f$ bounded, which will control the estimator’s variance.

Worked Example: What the Risk Measures

Suppose $f(x_0)=2$, and an estimator takes the value $\hat T=2.3$ on a particular sample. Its squared error on that sample is $(2.3-2)^2=0.09$. The risk is not this one realized error; it is the expectation of squared error over repeated samples from $f$.

The worst-case risk over $\mathcal F$ takes the largest such expected error among densities in the class. The minimax risk then asks how small that worst-case quantity can be made by choosing an estimator. Thus the rate is a statement about performance across the entire class, not just at one convenient density.

Kernel Upper Bound

Use a compactly supported kernel $K$ with support in $[-A,A]$, $\int K(u)\,du=1$, $\int u^jK(u)\,du=0$ for $1\leq j\leq k$, and $\int K(u)^2\,du<\infty$. Also assume $\int |u|^\beta|K(u)|\,du<\infty$, which follows from compact support and boundedness for the usual kernels. Kernels of order $k$ may take negative values when $k\geq 1$. Define $$ \hat f_h(x_0)=\frac{1}{nh}\sum_{i=1}^n K\!\left(\frac{x_0-X_i}{h}\right). $$

The bandwidth $h$ determines the width of the neighborhood used. Let $d=\min(x_0,1-x_0)>0$. When $Ah\leq d$, every point $x_0-hu$ for $u$ in the support of $K$ lies in $[0,1]$. This interior condition matters: it lets us use the full kernel integral without losing mass at the domain boundary.

Theorem: Under the stated Hölder and kernel conditions, for all sufficiently small $h$ such that $Ah\leq d$, there are constants $C_1,C_2$ independent of $n$ and $h$ for which $$ \left|E_f[\hat f_h(x_0)]-f(x_0)\right|\leq C_1h^\beta, \qquad \operatorname{Var}_f(\hat f_h(x_0))\leq \frac{C_2}{nh}. $$ Consequently, uniformly over $\mathcal F$, $$ E_f[(\hat f_h(x_0)-f(x_0))^2] \leq C_1^2h^{2\beta}+\frac{C_2}{nh}. $$

Proof of the bias bound. A change of variables in the expectation gives $$ E_f[\hat f_h(x_0)] = \int_{(x_0-1)/h}^{x_0/h}K(u)f(x_0-hu)\,du. $$ Since $Ah\leq d$, the integration interval includes $[-A,A]$, which contains the support of $K$. Thus this equals the integral over the full support of $K$. Hölder smoothness gives a Taylor expansion at $x_0$ through degree $k$, with remainder bounded by $C L|hu|^\beta$ for a constant $C$ depending only on $\beta$. For $k=0$, this is exactly the Hölder condition. For $k\geq1$, it follows by applying Taylor’s formula through degree $k-1$, writing the remainder as an integral involving $f^{(k)}$, and subtracting $f^{(k)}(x_0)$ inside that integral; the difference is bounded by $L|hu|^\alpha$.

The polynomial terms of degrees $1,\ldots,k$ integrate to zero by the kernel moment conditions, while the degree-zero term integrates to $f(x_0)$. The remainder therefore yields $$ \left|E_f[\hat f_h(x_0)]-f(x_0)\right| \leq CLh^\beta\int |u|^\beta|K(u)|\,du. $$ This proves the bias bound with a constant uniform over the class.

Proof of the variance bound. Independence gives $$ \operatorname{Var}_f(\hat f_h(x_0)) =\frac{1}{n}\operatorname{Var}_f\!\left[ \frac{1}{h}K\!\left(\frac{x_0-X_1}{h}\right)\right] \leq \frac{1}{nh^2}E_f\!\left[K^2\!\left(\frac{x_0-X_1}{h}\right)\right]. $$ Changing variables in this expectation and using $\sup_x f(x)\leq L$ gives $$ \operatorname{Var}_f(\hat f_h(x_0)) \leq \frac{L}{nh}\int K(u)^2\,du. $$ Finally, mean squared error equals variance plus squared bias, proving the risk bound. $\square$

Worked Example: A Hölder-One-Half Density

Let $\beta=1/2$, $x_0=0.4$, and use $K(u)=\tfrac12\mathbf 1\{|u|\leq1\}$. This kernel integrates to one and has $\int |u|^{1/2}K(u)\,du=2/3$, while $\int K(u)^2\,du=1/2$. For a density satisfying $|f(x)-f(y)|\leq L|x-y|^{1/2}$ and $\sup f\leq M$, the bias is bounded by $$ Lh^{1/2}\int |u|^{1/2}K(u)\,du =\frac{2L}{3}h^{1/2}. $$ The variance is at most $M/(2nh)$. The support condition holds when $h\leq0.4$, since $d=\min(0.4,0.6)=0.4$.

Taking $h=n^{-1/2}$ (for $n$ large enough that $h\leq0.4$) gives $$ E_f[(\hat f_h(x_0)-f(x_0))^2] \leq \frac{4L^2}{9}n^{-1/2}+\frac{M}{2}n^{-1/2}. $$ Both terms have order $n^{-1/2}$, the claimed pointwise rate for $\beta=1/2$.

Choosing the Bandwidth

The bound has two competing terms. A smaller $h$ reduces the squared bias $h^{2\beta}$, but increases the variance $1/(nh)$. Set the orders equal: $$ h^{2\beta}=\frac{1}{nh} \quad\Longleftrightarrow\quad h^{2\beta+1}=\frac{1}{n} \quad\Longleftrightarrow\quad h=n^{-1/(2\beta+1)}. $$

At this bandwidth, $h^{2\beta}=n^{-2\beta/(2\beta+1)}$, and $1/(nh)=n^{-2\beta/(2\beta+1)}$ as well. The kernel bound therefore gives a uniform upper bound of order $n^{-2\beta/(2\beta+1)}$. Since $x_0$ is fixed in the interior, the condition $Ah\leq d$ holds for all sufficiently large $n$.

Worked Example: Smoothness Three-Halves

Let $\beta=3/2$. Then $k=1$, so use a kernel with zero first moment. The Epanechnikov kernel $K(u)=\tfrac34(1-u^2)\mathbf 1\{|u|\leq1\}$ has $\int K=1$, $\int uK(u)\,du=0$, and $\int K^2=3/5$. Also, $$ \int |u|^{3/2}K(u)\,du =\frac32\int_0^1(u^{3/2}-u^{7/2})\,du =\frac32\left(\frac25-\frac29\right) =\frac4{15}. $$

For a $C^{1,1/2}$ density with Hölder constant $L$ for $f'$, the Taylor remainder is bounded by $\tfrac23L|hu|^{3/2}$. Thus the bias is at most $\tfrac23Lh^{3/2}(4/15)=8Lh^{3/2}/45$. If $\sup f\leq M$, the variance is at most $3M/(5nh)$. Balancing gives $h=n^{-1/4}$, and the two risk terms are bounded by $64L^2n^{-3/4}/2025$ and $3Mn^{-3/4}/5$. The rate is $n^{-3/4}$, as $2\beta/(2\beta+1)=3/4$.

Why the Rate Cannot Be Improved

An upper bound alone does not establish a minimax rate. To prove a matching lower bound, construct two densities that are hard to distinguish from $n$ observations but whose values at $x_0$ differ by the relevant amount. The perturbation is localized: it changes the density near $x_0$, integrates to zero, and has height proportional to its width raised to $\beta$.

Choose a smooth function $\eta(t)=\exp[-1/(1-t^2)]$ for $|t|<1$, and $\eta(t)=0$ otherwise. Let $\psi(t)=\eta(t)-2\eta(2(t-2))$. The second term has integral $2\cdot\frac12\int\eta=\int\eta$, so $\int\psi=0$; its support is disjoint from the first term’s support, and $\psi(0)=e^{-1}$. For sufficiently small $h$, the support of $x\mapsto\psi((x-x_0)/h)$ lies inside $[0,1]$.

Theorem: Let $\mathcal F$ be a Hölder-$\beta$ density class on $[0,1]$, with radius $L>1$ large enough to contain the perturbations below. For fixed $x_0\in(0,1)$, there is a constant $c_0>0$ such that, for all sufficiently large $n$, $$ R_n(\mathcal F,x_0)\geq c_0 n^{-2\beta/(2\beta+1)}. $$ Together with the kernel upper bound, this gives $R_n(\mathcal F,x_0)\asymp n^{-2\beta/(2\beta+1)}$.

Proof. Take the base density $f_0(x)=1$ and define $$ f_1(x)=1+c h^\beta\psi\!\left(\frac{x-x_0}{h}\right), $$ where $c>0$ is a small fixed constant. Since $\int\psi=0$, the perturbation integrates to zero, so $\int_0^1 f_1=1$. For small enough $c$, $f_1\geq0$. Its derivatives of order $j\leq k$ have perturbation size proportional to $c h^{\beta-j}$; the $k$-th derivative has an $\alpha$-Hölder seminorm bounded by a constant times $c$, by rescaling. Thus, choosing $c$ sufficiently small puts both $f_0$ and $f_1$ in the stated class for all sufficiently small $h$.

The separation between the target values is $$ \Delta=|f_1(x_0)-f_0(x_0)| =c h^\beta|\psi(0)|=c e^{-1}h^\beta. $$ For one observation under $f_0$, the chi-squared divergence of $f_1$ from $f_0$ is $$ \int_0^1\frac{(f_1(x)-f_0(x))^2}{f_0(x)}\,dx =c^2h^{2\beta+1}\int\psi(t)^2\,dt. $$ The power $h$ in this expression comes from the change of variables $t=(x-x_0)/h$. For $n$ independent observations, the product chi-squared divergence is $(1+\chi^2)^n-1$, which is at most $\exp(n\chi^2)-1$. Set $h=a n^{-1/(2\beta+1)}$, where $a>0$ is small enough that this divergence is at most $1$. The total variation distance between the two $n$-sample distributions is then at most $\tfrac12\sqrt{1}=\tfrac12$, using $\operatorname{TV}\leq\tfrac12\sqrt{\chi^2}$.

For any estimator $\hat T$, classify the sample as coming from whichever of $f_0(x_0)$ and $f_1(x_0)$ is closer to $\hat T$. If the classification is wrong, the estimation error has magnitude at least $\Delta/2$. The sum of the two testing error probabilities is at least $1-\operatorname{TV}$. Therefore, $$ \frac12\left(E_{f_0}[(\hat T-f_0(x_0))^2]+E_{f_1}[(\hat T-f_1(x_0))^2]\right) \geq \frac{\Delta^2}{8}(1-\operatorname{TV}) \geq \frac{\Delta^2}{16}. $$ Since $\Delta^2=c^2e^{-2}a^{2\beta}n^{-2\beta/(2\beta+1)}$, the worst-case risk over the class is at least a positive constant times this rate. Taking the infimum over estimators proves the lower bound. $\square$

Worked Example: Checking the Lower-Bound Scaling

For $\beta=1/2$, choose $h=a n^{-1/2}$. The target separation is $\Delta=ce^{-1}a^{1/2}n^{-1/4}$, so $\Delta^2=c^2e^{-2}a n^{-1/2}$. Meanwhile, the one-observation chi-squared divergence is $c^2h^2\int\psi^2$; multiplying by $n$ gives $c^2a^2\int\psi^2$, which does not depend on $n$. Choose $a$ small enough to keep the two experiments close in total variation. The testing bound then forces risk of order $n^{-1/2}$, matching the kernel upper bound.

Common Mistake and Practical Meaning

A frequent rate-calculation error is to balance the bias $h^\beta$ directly against the variance $1/(nh)$. These are not terms in the same risk decomposition: bias must be squared. The correct balance is $h^{2\beta}\asymp1/(nh)$, yielding $h\asymp n^{-1/(2\beta+1)}$ and risk $n^{-2\beta/(2\beta+1)}$. Balancing the unsquared bias instead gives the wrong bandwidth and rate.

The result quantifies how local smoothness controls the difficulty of estimating a density value: smoother densities allow a wider neighborhood to be averaged without incurring as much bias. The interior-point assumption is substantive. Near a boundary, the full-support calculation used above fails for an uncorrected symmetric kernel, so the bias argument must be modified. The compact-support condition in the proof also ensures that the expectation integral truly covers the full kernel support; without it, omitted tails would need to be bounded separately.

Check Your Understanding

Use the definitions and rate arguments above to answer these questions.

For $\beta=2$, what bandwidth and pointwise squared-error rate follow from balancing the kernel bias and variance?
Why does the interior condition $Ah\leq d$ allow the expectation integral to be written over the full support of $K$?
In the lower-bound construction, why must the localized perturbation integrate to zero?
Explain why the squared bias term is $h^{2\beta}$, rather than $h^\beta$, in the risk calculation.
What role does choosing the constant $a$ small play in the two-point lower-bound argument?

Back to Tutorials