Bandwidth Optimization for Integrated Risk

Why Bandwidth Is a Trade-off

A kernel density estimator averages observations over a neighborhood whose width is controlled by the bandwidth $h$. A small bandwidth uses very local information: this limits smoothing bias but leaves more sampling variability. A large bandwidth reduces variability by averaging over a wider region, but can smooth away features of the density. Bandwidth optimization makes this trade-off quantitative.

Here the target is integrated squared-error risk. The central calculation balances an integrated squared bias of order $h^{2\beta}$ against an integrated variance of order $1/(nh^d)$, where $d$ is the dimension and $\beta$ describes the order of the bias. The resulting bandwidth shrinks as the sample grows, but not so quickly that the variance remains large.

Risk and the Leading-Order Assumptions

Let $X_1,\ldots,X_n$ be independent observations with density $f$ on $\mathbb{R}^d$. For a kernel $K$ with $\int_{\mathbb{R}^d}K(u)\,du=1$, define $K_h(u)=h^{-d}K(u/h)$ and the kernel density estimator

\hat f_h(x)=\frac{1}{n}\sum_{i=1}^n K_h(x-X_i) =\frac{1}{nh^d}\sum_{i=1}^n K\!\left(\frac{x-X_i}{h}\right).

Its integrated squared-error risk is $R_n(h)=E\!\left[\int_{\mathbb{R}^d}(\hat f_h(x)-f(x))^2\,dx\right]$. The integrated bias and variance decomposition gives

R_n(h)=\int_{\mathbb{R}^d}\big(E[\hat f_h(x)]-f(x)\big)^2\,dx +\int_{\mathbb{R}^d}\operatorname{Var}(\hat f_h(x))\,dx.

The bandwidth calculation uses the following leading-order conditions. They summarize the bias and integrated-variance behavior needed here; in particular, the bias condition must apply to the integrated squared bias, not just to the pointwise bias at a selected $x$.

Definition: Suppose, as $h\to0$ and $nh^d\to\infty$, the integrated squared bias is $A h^{2\beta}(1+o(1))$ and the integrated variance is $\frac{B}{nh^d}(1+o(1))$, where $\beta>0$ and $A,B>0$. The leading integrated-risk approximation is $$ R_{0,n}(h)=A h^{2\beta}+\frac{B}{nh^d}. $$

The constants $A$ and $B$ depend on the density and kernel, as well as on the particular bias order. For example, when a symmetric second-order kernel and sufficient smoothness give bias of order $h^2$, the usual value is $\beta=2$. The optimization below does not require that particular value.

Deriving the Optimal Bandwidth

The leading-risk-optimal bandwidth minimizes $R_{0,n}(h)$ over $h>0$. The first term increases with $h$; the second decreases. Differentiating gives a unique balance point.

Theorem: For $A,B,\beta,d>0$ and $n>0$, the function $R_{0,n}(h)=A h^{2\beta}+B/(nh^d)$ has a unique global minimizer $$ h_{0,n}=\left(\frac{dB}{2\beta A n}\right)^{1/(2\beta+d)}. $$ At this bandwidth, $h_{0,n}$ is of order $n^{-1/(2\beta+d)}$, and the minimized leading risk is of order $n^{-2\beta/(2\beta+d)}$.

Proof. Differentiate for $h>0$:

R_{0,n}'(h)=2\beta A h^{2\beta-1}-\frac{dB}{n}h^{-d-1}.

The derivative is zero exactly when $2\beta A h^{2\beta+d}=dB/n$. The left side is strictly increasing from zero to infinity as $h$ ranges from zero to infinity, so this equation has exactly one positive solution, namely the stated $h_{0,n}$. Moreover, $R_{0,n}'(h)<0$ below that solution and $R_{0,n}'(h)>0$ above it. Thus the solution is the unique global minimum.

At the stationary point, multiplying the derivative equation by $h$ shows

2\beta A h_{0,n}^{2\beta} =d\frac{B}{n h_{0,n}^{d}}, \qquad \frac{B}{n h_{0,n}^{d}} =\frac{2\beta}{d}A h_{0,n}^{2\beta}.

Substituting the bandwidth into the first term yields

R_{0,n}(h_{0,n}) =\left(1+\frac{2\beta}{d}\right)A \left(\frac{dB}{2\beta A}\right)^{2\beta/(2\beta+d)} n^{-2\beta/(2\beta+d)}.

This proves both rates and shows that the two risk terms need not be equal: at the optimum, their ratio depends on $\beta$ and $d$. $\square$

When the Leading-Order Calculation Describes the Risk

The theorem exactly optimizes $R_{0,n}$. To interpret it as an asymptotic optimization of the actual risk, the expansions must hold in the bandwidth range being considered. One precise sufficient formulation is that, for every compact interval $[c_1,c_2]\subset(0,\infty)$, uniformly over $c\in[c_1,c_2]$,

n^{2\beta/(2\beta+d)} R_n\!\left(c n^{-1/(2\beta+d)}\right) \longrightarrow A c^{2\beta}+B c^{-d}.

The limit on the right has a unique minimizer $c_*=(dB/(2\beta A))^{1/(2\beta+d)}$, by the same derivative calculation. If exact risk minimizers exist and their rescaled bandwidths $n^{1/(2\beta+d)}h_n$ stay bounded and bounded away from zero, uniform convergence on compact intervals implies that these rescaled minimizers converge to $c_*$. To see why, take any neighborhood of $c_*$ inside such a compact interval. The continuous limiting function has a strictly larger minimum on the compact set outside that neighborhood, since its minimizer is unique. Uniform convergence preserves a positive part of this gap for all sufficiently large $n$, so an exact minimizer cannot remain outside the neighborhood.

This condition matters: leading-order expansions at each fixed $h$ do not, by themselves, rule out a better exact-risk value at a bandwidth outside the range where the expansions apply. The rate is the usual asymptotic bandwidth choice under the stated regularity, not a guarantee that the displayed formula is an exact finite-sample minimizer.

Worked Gaussian Examples

Worked Example: One-Dimensional Normal Density

Suppose $X_i\sim N(0,1)$, and use the standard normal kernel $K$ in one dimension. For this kernel, the leading bias is $h^2 f''/2$, so $\beta=2$ and $A=\frac14\int_{\mathbb{R}}(f''(x))^2\,dx$. The integrated variance constant is $B=\int K(u)^2\,du$, because $f$ integrates to one.

For the standard normal density, $f''(x)=(x^2-1)f(x)$. Since $f(x)^2=(2\pi)^{-1}e^{-x^2}$, integration using the moments of a $N(0,1/2)$ variable gives

\int (f'')^2 =\frac{\sqrt{\pi}}{2\pi} E\!\left[(Z^2-1)^2\right] =\frac{1}{2\sqrt{\pi}}\left(\frac34-1+1\right) =\frac{3}{8\sqrt{\pi}}, \qquad Z\sim N(0,1/2).

Therefore $A=3/(32\sqrt{\pi})$. Also $B=\int K^2=1/(2\sqrt{\pi})$. With $d=1,\ \beta=2$, the leading-risk-optimal bandwidth is

h_{0,n} =\left(\frac{B}{4An}\right)^{1/5} =\left(\frac{4}{3n}\right)^{1/5}.

For $n=1000$, this is approximately $0.266$. At the optimum, the variance term is four times the squared-bias term because $2\beta/d=4$. The leading risk is approximately $5A h_{0,n}^4$, or $0.00133$. These are leading-order values, not exact finite-sample risk calculations.

Worked Example: Two-Dimensional Normal Density

Now let $X_i\sim N_2(0,I_2)$ and use the standard bivariate normal kernel. In dimension $d$, the corresponding normal target and kernel give $B=(4\pi)^{-d/2}$. The second-order bias has leading term $h^2\Delta f/2$, where $\Delta f$ is the sum of the second partial derivatives, and hence $A=\frac14\int(\Delta f)^2$.

For $f=N_d(0,I_d)$, $\Delta f(x)=(\|x\|^2-d)f(x)$. Under the density proportional to $f^2$, $x$ has distribution $N_d(0,I_d/2)$, and $\int f^2=(4\pi)^{-d/2}$. The squared norm has mean $d/2$ and variance $d/2$, so

\int(\Delta f)^2 =(4\pi)^{-d/2} E\!\left[(\|Z\|^2-d)^2\right] =(4\pi)^{-d/2}\left(\frac d2+\frac{d^2}{4}\right) =(4\pi)^{-d/2}\frac{d(d+2)}{4}.

Thus $A=(4\pi)^{-d/2}d(d+2)/16$. For $d=2$, this gives $A=1/(8\pi)$ and $B=1/(4\pi)$. With $\beta=2$ and $n=1000$,

h_{0,1000} =\left(\frac{2B}{4A(1000)}\right)^{1/6} =1000^{-1/6} \approx 0.316.

Since $h^4=0.01$ and $h^2=0.1$, the leading squared bias is $Ah^4=1/(800\pi)\approx0.000398$, while the leading variance is $B/(1000h^2)=1/(400\pi)\approx0.000796$. The variance is twice the bias term, in agreement with $2\beta/d=2$, and the total leading risk is $3/(800\pi)\approx0.00119$.

Worked Example: Sample Size Scaling in Two Dimensions

Keep the two-dimensional normal model and kernel from the previous calculation. The leading bandwidth is $h_{0,n}=n^{-1/6}$, and the leading risk is $3/(800\pi)$ times $(n/1000)^{-2/3}$, using the value at $n=1000$ as the reference.

At $n=64$, $h_{0,64}=64^{-1/6}=1/2$. The leading squared bias is

\frac{1}{8\pi}\left(\frac12\right)^4=\frac{1}{128\pi}\approx0.00249,

and the leading variance is

\frac{1/(4\pi)}{64(1/2)^2} =\frac{1}{64\pi} \approx0.00497.

The total is $3/(128\pi)\approx0.00746$, which is $6.25$ times the $n=1000$ leading risk. This agrees with the rate prediction: $(1000/64)^{2/3}=15.625^{2/3}=6.25$. The comparison illustrates that the dimension and smoothness determine the exponents; the constants still determine the numerical scale.

Interpretation and a Common Pitfall

The bandwidth rate $n^{-1/(2\beta+d)}$ reflects the competition between smoothing bias and sampling variability. Greater smoothness, represented by larger $\beta$, allows a larger bandwidth while still controlling bias. Greater dimension makes the variance term more sensitive to shrinking bandwidth through $h^{-d}$, and the resulting risk rate becomes slower for fixed $\beta$.

Common mistake: setting the two risk terms equal. One might reason that the optimum must satisfy $Ah^{2\beta}=B/(nh^d)$. That equality is generally false. Differentiating the risk shows instead that $2\beta Ah^{2\beta}=dB/(nh^d)$. The equality of terms happens only when $d=2\beta$. For example, in the one-dimensional normal example the variance term is four times the squared-bias term at the optimum, not equal to it.

The calculation also relies on the assumed integrated bias expansion. It does not establish that the same expansion holds at every location or for every support and kernel. In particular, a setting where boundary behavior changes the bias requires its own analysis; the interior bandwidth balance cannot be used to settle that issue.

Key takeaway: Under integrated squared bias $A h^{2\beta}$ and integrated variance $B/(nh^d)$, optimize their sum to obtain $h_{0,n}=(dB/(2\beta A n))^{1/(2\beta+d)}$ and leading risk of order $n^{-2\beta/(2\beta+d)}$. Keep the constants and the scope of the bias assumptions in view.

Check Your Understanding

Use the definitions and leading-risk calculation above to answer the following.

For $d=1$ and $\beta=2$, what are the bandwidth and integrated-risk exponents in $n$?
At the leading-risk optimum, what is the ratio of the variance term to the integrated squared-bias term?
If $A$ and $B$ are fixed, how does the optimal bandwidth change when the sample size is multiplied by $16$?
Why does pointwise convergence of the risk approximation alone not necessarily establish the behavior of an exact risk minimizer?
In the two-dimensional Gaussian example, verify that the stated leading variance is twice the leading squared bias at the optimal bandwidth.