Anisotropic Bandwidths for Unequal Coordinate Scales

Why Use a Separate Bandwidth for Each Coordinate?

In multivariate regression, predictor coordinates need not vary on comparable scales, and the regression function need not bend equally in every direction. A single bandwidth imposes the same smoothing distance on all coordinates. A diagonal bandwidth matrix instead allows the fit to look farther along some coordinates and less far along others. Its consequences are specific: each derivative in the bias expansion is multiplied by the corresponding coordinate bandwidths, while the leading variance depends on the product of the bandwidths.

We focus on an interior point $x\in\mathbb{R}^d$ and a local polynomial of total degree $p$. Write $Y_i=m(X_i)+\varepsilon_i$, where $E[\varepsilon_i\mid X_i]=0$. The diagonal bandwidth matrix rescales distances before the polynomial fit; it does not change which polynomial degree is fitted.

Definition: The diagonal bandwidth matrix is $H=\operatorname{diag}(h_1,\ldots,h_d)$, where every $h_j>0$. The scaled coordinate for an observation $X_i$ near $x$ is $u_i=H^{-1}(X_i-x)$, so $u_{ij}=(X_{ij}-x_j)/h_j$. Write $|H|=\prod_{j=1}^d h_j$ for its determinant and $h_{\max}=\max_j h_j$. The local fit weights observations according to a kernel $K(u_i)$ and fits a polynomial in the scaled coordinates.

For precision, let $r_p(u)$ be the vector of all monomials $u^\alpha=\prod_{j=1}^d u_j^{\alpha_j}$ with total degree $|\alpha|=\sum_j\alpha_j\leq p$, in a fixed order with the constant first. The local polynomial estimate is the fitted constant at $u=0$. The entries of $H$ determine the scale of each coordinate in this fit.

Equivalent-Kernel Moments and the Bias Expansion

The leading bias can be described using the equivalent kernel of the local polynomial fit. Assume $K$ is bounded, has compact support, and has enough finite moments for the expressions below. Define the moment matrix $M=\int K(u)r_p(u)r_p(u)^\mathsf{T}\,du$, and assume it is nonsingular. Let $e_0$ select the constant coefficient. The equivalent kernel is

L_p(u)=K(u)e_0^\mathsf{T}M^{-1}r_p(u).

This function may be signed, even if $K$ is nonnegative. Its key property is reproduction of polynomials through degree $p$: $\int L_p(u)\,du=1$, and $\int u^\beta L_p(u)\,du=0$ for every multi-index $\beta$ with $1\leq|\beta|\leq p$. These identities express the fact that a local polynomial fit reproduces those polynomials at its center.

Bias expansion: Suppose $m$ has continuous derivatives through order $p+2$ near $x$. For a regular interior design, the leading smoothing bias is $$ B_h(x)=\sum_{|\alpha|=p+1}\frac{h^\alpha}{\alpha!}D^\alpha m(x)\int u^\alpha L_p(u)\,du, \qquad h^\alpha=\prod_{j=1}^d h_j^{\alpha_j}. $$ The next Taylor terms are of order $O(h_{\max}^{p+2})$. A displayed leading coefficient can be zero; in that case the actual bias can be of smaller order.

To see why the coordinate scales appear as $h^\alpha$, expand $m(x+Hu)$ around $x$. Its term indexed by $\alpha$ is $D^\alpha m(x)(Hu)^\alpha/\alpha!=D^\alpha m(x)h^\alpha u^\alpha/\alpha!$. Integrating against $L_p$ removes all terms of degree at most $p$; the first terms that can remain have degree $p+1$.

Worked Example: A Two-Dimensional Local Linear Bias

Use a product of normalized Epanechnikov kernels, $K(u)=k(u_1)k(u_2)$, where $k(t)=\tfrac34(1-t^2)$ for $|t|\leq1$ and zero otherwise. At an interior point, fit a local linear polynomial ($p=1$). The kernel is symmetric, so its first moments and its mixed second moment vanish. For this fit the equivalent kernel is $K$, and $\int u_j^2K(u)\,du=1/5$.

Take $m(x_1,x_2)=x_1^2+3x_2^2+2x_1x_2$ at $x=(0,0)$, with $h_1=0.2$ and $h_2=0.1$. The second derivatives are $m_{11}=2$, $m_{22}=6$, and $m_{12}=2$. The quadratic Taylor contribution is

\frac{1}{2}h_1^2m_{11}\int u_1^2K(u)\,du +h_1h_2m_{12}\int u_1u_2K(u)\,du +\frac{1}{2}h_2^2m_{22}\int u_2^2K(u)\,du.

The mixed integral is zero by symmetry. Substituting the remaining values gives $\tfrac12(0.2)^2(2)(1/5)+\tfrac12(0.1)^2(6)(1/5)=0.008+0.006=0.014$. The two diagonal curvatures contribute separately, each scaled by the square of its own bandwidth.

Why the Product of Bandwidths Controls Variance

The observations receiving appreciable weight lie in a region whose coordinate widths are proportional to $h_1,\ldots,h_d$. Its volume is proportional to $|H|$. If the design density at $x$ is $f(x)>0$, the expected number of observations in that region is of order $n f(x)|H|$. Thus, narrowing any one bandwidth reduces the local data available, even if the others are unchanged.

Leading variance: If $\operatorname{Var}(Y\mid X=z)=\sigma^2(z)$ is continuous at $x$, then the leading variance of the local polynomial estimate is $$ \operatorname{Var}\{\hat m(x)\}\ \sim\ \frac{\sigma^2(x)}{n f(x)|H|}\int L_p(u)^2\,du. $$ The anisotropic bandwidths enter this expression through their product, while their separate values enter the bias expansion through $h^\alpha$.

Here is a precise asymptotic version, together with the main justification. Assume the $n$ observations are independent and identically distributed; $f$ is continuous and positive at $x$; $m$ has continuous derivatives through order $p+2$; and the conditional error variance is continuous and positive at $x$. Assume also bounded compactly supported $K$, nonsingular $M$, uniformly bounded conditional $2+\delta$ error moments near $x$ for some $\delta>0$, and $h_{\max}\to0$, $n|H|\to\infty$. Under these conditions, with the additional undersmoothing condition $\sqrt{n|H|}\,h_{\max}^{p+2}\to0$, the estimate centered by its displayed leading bias satisfies

\sqrt{n|H|}\{\hat m(x)-m(x)-B_h(x)\} \xrightarrow{d} N\!\left(0,\frac{\sigma^2(x)}{f(x)}\int L_p(u)^2\,du\right).

Expand the regression function.
For $u=H^{-1}(z-x)$, Taylor expansion gives $m(x+Hu)=\sum_{|\alpha|\leq p+1}D^\alpha m(x)h^\alpha u^\alpha/\alpha!+O(h_{\max}^{p+2})$ on the kernel support. The local polynomial reproduces all terms through degree $p$. Integrating the remaining degree-$p+1$ terms against $L_p$ gives $B_h(x)$.

Approximate the local moment matrix.
After dividing the empirical weighted moment matrix by $n|H|$, its entries converge in probability to the entries of $f(x)M$. This follows by changing variables $z=x+Hu$, using continuity of $f$, and applying the law of large numbers; $n|H|\to\infty$ ensures that the local sample size diverges. The limiting matrix is nonsingular.

Obtain the leading weights.
The inverse moment matrix therefore converges to $\{f(x)M\}^{-1}$. The intercept weights consequently have leading form $(n|H|f(x))^{-1}L_p(u_i)$. The regression-function part has the bias from Step 1; the Taylor remainder, on the stated undersmoothing scale, is negligible.

Compute the noise variance and limit.
The sum of squared leading weights times conditional variances converges, after multiplication by $n|H|$, to $\sigma^2(x)f(x)^{-1}\int L_p^2$, by the same change of variables. The largest normalized weight tends to zero as $n|H|\to\infty$; the conditional $2+\delta$ moment bound gives the Lindeberg condition. The triangular-array central limit theorem yields the stated normal limit. $\square$

Worked Example: Choosing Bandwidths at a Fixed Product

For a two-dimensional local linear fit with the same product Epanechnikov kernel, the leading bias for a diagonal Hessian with $m_{11}(x)=a$, $m_{22}(x)=b$, and zero mixed derivative is $(a h_1^2+b h_2^2)/10$. Suppose $a=1$, $b=16$, and the bandwidth product is fixed at $h_1h_2=0.01$. The variance constant is then unchanged as the bandwidths are reallocated, because it depends on their product.

To minimize the leading bias, minimize $a h_1^2+b h_2^2$ subject to $h_2=0.01/h_1$. Differentiating gives $2a h_1-2b(0.01)^2/h_1^3=0$, so $a h_1^4=b(0.01)^2$. Hence $h_1/h_2=\sqrt{b/a}=4$, giving $h_1=0.2$, $h_2=0.05$. The leading bias is $\{(0.2)^2+16(0.05)^2\}/10=0.008$. With isotropic bandwidths $h_1=h_2=0.1$, the bias is $\{0.01+16(0.01)\}/10=0.017$. The anisotropic choice reduces this leading bias while preserving the leading variance constant.

Worked Example: Effective Sample Size and Variance

Consider a two-dimensional local linear fit with the same kernel, $n=50{,}000$, $f(x)=0.2$, $\sigma^2(x)=9$, and bandwidths $h_1=0.05$, $h_2=0.2$. Their product is $0.01$, so the effective local sample size $n f(x)|H|$ is $50{,}000(0.2)(0.01)=100$.

For the product Epanechnikov kernel, $\int K^2=(\int k^2)^2=(3/5)^2=9/25$. Therefore the leading variance is $9(9/25)/100=0.0324$, with standard deviation approximately $0.18$. If $h_2$ is halved while $h_1$ stays fixed, the bandwidth product and effective local sample size are halved. The leading variance doubles to $0.0648$. This calculation isolates the variance cost of narrowing one coordinate.

Symmetry Can Remove a Nominal Leading Term

The bias expansion lists terms that can contribute; it does not guarantee that each coefficient is nonzero. For example, with a symmetric product kernel in two dimensions and a local linear fit, the equivalent kernel is symmetric in each coordinate. Thus $\int u_1^2u_2L_1(u)\,du=0$. If the degree-three Taylor term is proportional to $u_1^2u_2$, that term contributes no leading bias, even though its bandwidth factor is $h_1^2h_2$. Other degree-three terms may also vanish, depending on symmetry.

This is useful when interpreting an expansion: the order $p+1$ identifies the first possible contributions, while the kernel moments and derivatives decide which are actually present. It also explains why a bound based on $h_{\max}^{p+1}$ may be conservative for a particular function and kernel.

Common Mistake

A common mistake is to say that anisotropic bandwidths affect variance through each coordinate separately in the same way they affect bias. The coordinate-specific bias terms do involve $h^\alpha$, so that reasoning can seem plausible. But the leading variance is proportional to $1/(n\prod_j h_j)$, not to a sum of separate inverse bandwidths. The correction is to distinguish the two roles: directional bandwidths interact with directional derivatives in the bias, whereas the local volume—and hence the leading effective sample size—is governed by their product.

Key takeaway: A diagonal bandwidth matrix lets smoothing adapt to coordinate-specific scales. For a degree-$p$ fit, the first possible bias terms pair derivatives of total order $p+1$ with products $h^\alpha$; the leading variance scales as $1/(n|H|)$. Bandwidth allocation therefore changes the bias direction by direction, but changing the product changes the leading variance.

Check Your Understanding

Use the definitions and results above to answer these questions.

For a two-dimensional local linear fit, which bandwidth product determines the leading variance?
Why does a degree-$p$ local polynomial remove Taylor terms of total degree at most $p$ from the leading bias?
If $h_1$ is halved while all other bandwidths remain fixed, what happens to the leading variance?
Why can a degree-$p+1$ term in the Taylor expansion contribute zero bias?
At a fixed bandwidth product, what feature of the regression function can guide how bandwidth is allocated across coordinates?