Spatially Varying Bandwidths Under Random Design

Why the Bandwidth Rule Matters

In local polynomial regression, a bandwidth determines which observations contribute meaningfully to the estimate at a target point. Under random design, however, the number of observations near that point depends on the predictor density. The same fixed bandwidth can therefore use many observations in a densely sampled region and few in a sparse one. A fixed-neighbor rule responds to this imbalance by changing the radius until it contains a specified number of observations.

This tutorial compares those rules for an interior point in one-dimensional regression. The main distinction is between a fixed spatial scale and a fixed local sample size. Neither rule eliminates the bias–variance tradeoff: they redistribute smoothing in different ways across the predictor space.

Setup and Bandwidth Rules

Assume throughout the derivations that \(x\) is an interior point of the predictor domain, \(g\) is continuous and strictly positive near \(x\), and \(\sigma^2\) is continuous there. We use a symmetric kernel \(K\) supported on \([-1,1]\), with \(\int K(u)\,du=1\), and bandwidths small enough that the kernel window remains inside the domain. The bandwidth is a radius: an observation at \(X_i\) has kernel argument \((X_i-x)/h\).

For a degree-\(p\) local polynomial fit, the fitted polynomial is obtained by weighted least squares using the kernel weights \(K((X_i-x)/h)\). Under the usual nonsingularity and moment conditions, its leading variance has the form

where \(L_p\) is the equivalent kernel for the local polynomial fit at an interior point. This is the random-design variance approximation for local polynomial estimators. The quantity \(n g(x)h\) is the local sample-size scale; the factor \(R(L_p)\) accounts for the weights, including any negative weights in higher-order fits.

Fixed Bandwidth: A Common Radius, Unequal Local Counts

For small \(h\), the probability that one predictor falls within distance \(h\) of \(x\) is

Thus the expected number of observations in the window is approximately \(2n g(x)h\). With \(h\) fixed, this count is proportional to the design density. The leading variance is proportional to \(1/[n g(x)h]\), so it is larger in sparse regions. The smoothing radius itself does not change with \(x\).

Fixed Neighbors: A Density-Dependent Radius

Let \(H_k(x)\) be the distance to the \(k\)-th nearest predictor to \(x\), with ties having probability zero under a continuous design density. For a small deterministic radius \(r\), the number of observations within \(r\) is approximately \(2n g(x)r\). Equating this local count to \(k\) suggests

This approximation becomes precise in relative probability under standard local conditions. Specifically, suppose \(k\to\infty\), \(k/n\to0\), and \(g\) is continuous and positive at the interior point \(x\). Then

To justify the radius result, write \(N_x(r)=\sum_{i=1}^n\mathbf{1}\{|X_i-x|\le r\}\). For fixed \(\epsilon\in(0,1)\), take \(r_{\pm}=(1\pm\epsilon)k/[2n g(x)]\). Since \(k/n\to0\), both radii tend to zero. Continuity of \(g\) gives \[ E[N_x(r_{\pm})] =n\int_{x-r_{\pm}}^{x+r_{\pm}}g(t)\,dt =(1\pm\epsilon)k\{1+o(1)\}. \] Each count is binomial, so its variance is at most its mean, which is of order \(k\). Chebyshev's inequality implies that the count divided by its mean converges in probability to one. Therefore, with probability tending to one, \(N_x(r_-)<k\) and \(N_x(r_+)\ge k\): the count means differ from \(k\) by order \(\epsilon k\), while the count fluctuations are only order \(\sqrt{k}\). These inequalities imply \(r_-<H_k(x)\le r_+\). Since this holds for every fixed \(\epsilon>0\), the stated ratio converges in probability to one. \(\square\)

Substituting the radius approximation into the leading local polynomial variance gives

The design density cancels from this leading variance expression. This cancellation describes the first-order behavior at a fixed interior point; it does not mean the complete finite-sample variance is exactly constant across \(x\). Nor does it mean the effective weights or the regression function are the same at every point.

For the variance conclusion, start from the fixed-radius expression with \(h=H_k(x)\). The radius result gives \(n g(x)H_k(x)/k\xrightarrow{p}1/2\), so its reciprocal is asymptotic in probability to \(2/k\). Multiplying by \(\sigma^2(x)R(L_p)\) yields the displayed leading expression. This substitution uses the local variance approximation in a shrinking window with an increasing number of observations; it is not an exact conditional-variance identity for a random bandwidth. \(\square\)

How the Bias Changes

A local polynomial bias is governed primarily by the physical radius, not just by the number of observations. To make the comparison concrete, consider a local linear estimator of \(m(x)\), with \(m\) twice continuously differentiable near \(x\) and a symmetric kernel with finite second moment. Its leading interior bias is

The local linear fit removes the first-order effect of a local slope. At an interior point, its leading bias does not include the first derivative of the design density. Nevertheless, changing the bandwidth changes the bias through \(h^2\). Since \(H_k(x)\) is larger where \(g(x)\) is smaller, a fixed-neighbor rule generally has larger local bias in sparse regions. More generally, when the leading bias for a degree-\(p\) fit is of order \(h^{p+1}\), substituting \(H_k(x)\) makes that scale proportional to \(\{k/[n g(x)]\}^{p+1}\), subject to the relevant smoothness and moment conditions.

Interpreting the Comparison

The rules answer different design questions. Fixed bandwidth keeps the physical scale of smoothing comparable across the domain, so the estimator targets the same-sized neighborhood everywhere. Its local precision varies with \(g(x)\). Fixed neighbors keeps the local sample count comparable, so its leading variance is less sensitive to \(g(x)\), but its spatial resolution varies. This can be useful when stable local sample sizes matter, while fixed bandwidth can be preferable when a consistent physical scale is essential.

These statements concern pointwise interior behavior. They do not by themselves establish which rule has smaller integrated risk or is preferable for a particular scientific goal. The conditional noise variance \(\sigma^2(x)\), the curvature or higher derivatives of \(m\), boundary behavior, kernel choice, and the way tuning parameters are selected all matter. In particular, equalizing the leading variance does not equalize mean squared error, because mean squared error also includes squared bias.

Common Pitfall

• Wrong reasoning: “A fixed-neighbor bandwidth has the same variance everywhere because it always uses \(k\) observations.” This ignores both the weights and the conditional noise variance. Even the leading expression is \(2\sigma^2(x)R(L_p)/k\), which can vary with \(\sigma^2(x)\); finite-sample variance can also vary for additional reasons.
• Correction: Under the stated local conditions, the design-density factor cancels from the leading variance because the radius expands or contracts approximately as \(1/g(x)\). The same radius change affects bias, and the leading noise variance need not be constant if the conditional variance changes with \(x\).
• Another error: Treating \(H_k(x)\) as if it were a fixed bandwidth in an exact variance formula. It is random and depends on the observed predictors. The substitution into the fixed-radius expression is a local asymptotic approximation, justified when the neighbor radius shrinks while \(k\) grows.