Random-Design Density Effects on Bias

Why the Design Density Enters the Bias

In random-design regression, observations near a target point are not necessarily distributed evenly. If the design density rises across a kernel window, a local constant fit gives relatively more weight to observations on one side. When the regression function slopes, that imbalance can shift the fitted value even if the kernel itself is symmetric. Local linear fitting adjusts for the local slope in the data, and its leading bias does not contain this density-gradient contribution.

We make that distinction precise for an interior point. The calculations below describe the leading smoothing bias: the bias of the population weighted fit, or equivalently the leading deterministic term in the usual local-polynomial expansion. A finite sample also has random design and response variation, so this term alone is not the entire finite-sample bias or error.

Setup: Let \((X_i,Y_i)\), \(i=1,\ldots,n\), be independent and identically distributed, with \(X_i\) having density \(f\) and \(Y_i=m(X_i)+\varepsilon_i\), where \(E[\varepsilon_i\mid X_i]=0\). Fix an interior point \(x\). Assume \(f(x)>0\), and that \(f\) and \(m\) have continuous second derivatives in a neighborhood of \(x\). Use a symmetric, compactly supported kernel \(K\) with \(\int K(u)\,du=1\), \(\int uK(u)\,du=0\), and \(0<\mu_2=\int u^2K(u)\,du<\infty\). The bandwidth is \(h>0\) and tends to zero.

For the local constant fit, the empirical estimator is the kernel-weighted average of the responses. Its population counterpart replaces the empirical weighted sums by their expectations. Write \(u=(X-x)/h\). The population target is

m_{\mathrm{LC},h}(x) = \frac{\int K(u)m(x+hu)f(x+hu)\,du} {\int K(u)f(x+hu)\,du}.

The density appears in both integrals because the observations being averaged are drawn from \(f\). The key question is what remains after taking their ratio.

Local Constant Bias: A Density-Gradient Term

Subtract \(m(x)\) from the population target. The numerator of the resulting difference is an integral of \(K(u)\{m(x+hu)-m(x)\}f(x+hu)\). Taylor expansion gives

m(x+hu)-m(x) =hu\,m'(x)+\frac{h^2u^2}{2}m''(x)+o(h^2), \qquad f(x+hu)=f(x)+hu\,f'(x)+\frac{h^2u^2}{2}f''(x)+o(h^2).

Multiplying these expansions, integrating, and using symmetry eliminates the terms proportional to \(u\). The terms of order \(h^2\) are \(h^2u^2m'(x)f'(x)\) and \(h^2u^2m''(x)f(x)/2\). The denominator is \(f(x)+o(1)\). Thus:

Local constant bias expansion: Under the stated interior-point and smoothness conditions,

m_{\mathrm{LC},h}(x)-m(x) = \frac{h^2\mu_2}{2}m''(x) +h^2\mu_2m'(x)\frac{f'(x)}{f(x)} +o(h^2).

The first term is the curvature contribution. The second is caused by the slope of the regression function interacting with the slope of the design density. In particular, a symmetric kernel does not by itself remove this second term: symmetry removes the first kernel moment, but the product of the two first-order Taylor terms is proportional to \(u^2\), whose kernel moment is not zero.

Worked Example: A Linear Regression Function Under an Exponential Design

Let \(X\) have density \(f(t)=e^t/(e-1)\) on \(0<t<1\), let \(m(t)=t\), and estimate at \(x=1/2\) using the uniform kernel \(K(u)=1/2\) on \([-1,1]\). For sufficiently small \(h\), the whole kernel window is inside \((0,1)\). The kernel moments are \(\mu_2=\int_{-1}^1 u^2/2\,du=1/3\), \(m'(x)=1\), \(m''(x)=0\), and \(f'(x)/f(x)=1\).

The expansion therefore gives

m_{\mathrm{LC},h}(1/2)-m(1/2) = \frac{h^2}{3}+o(h^2).

The sign is positive: the design density is larger to the right, and the linear regression function also increases to the right. A symmetric kernel window consequently averages observations from the more densely sampled side more heavily.

Local Linear Fitting Cancels the Leading Density Effect

The population local linear fit minimizes the kernel-weighted squared discrepancy from a line in the rescaled coordinate \(u=(X-x)/h\). Write that line as \(m(x)+a+bu\); its fitted value at \(x\) is \(m(x)+a\). Define the moments

S_j=\int u^jK(u)f(x+hu)\,du,\qquad T_j=\int u^jK(u)\{m(x+hu)-m(x)\}f(x+hu)\,du.

The normal equations for the intercept adjustment \(a\) and slope coefficient \(b\) are \(S_0a+S_1b=T_0\) and \(S_1a+S_2b=T_1\). If \(S_0S_2-S_1^2>0\), solving these equations gives the following intercept:

a=\frac{S_2T_0-S_1T_1}{S_0S_2-S_1^2}.

The next result shows exactly where the cancellation occurs. The expansions use the same smoothness and kernel assumptions as above; the positive leading denominator follows from \(f(x)>0\) and \(\mu_2>0\).

Result and derivation: The local linear population fit has leading bias

a=\frac{h^2\mu_2}{2}m''(x)+o(h^2).

To verify it, Taylor expansion of the moment integrals, together with symmetry, yields

\begin{aligned} S_0&=f(x)+o(1),& S_1&=h\mu_2 f'(x)+o(h),& S_2&=\mu_2 f(x)+o(1),\\ T_0&=h^2\mu_2\left\{\frac{f(x)m''(x)}{2}+f'(x)m'(x)\right\}+o(h^2),& T_1&=h\mu_2 f(x)m'(x)+o(h). \end{aligned}

In the numerator \(S_2T_0-S_1T_1\), the order-\(h^2\) density-gradient terms are equal and cancel:

\begin{aligned} S_2T_0 &=h^2\mu_2^2 \left\{\frac{f(x)^2m''(x)}{2}+f(x)f'(x)m'(x)\right\} +o(h^2),\\ S_1T_1 &=h^2\mu_2^2 f(x)f'(x)m'(x)+o(h^2),\\ S_0S_2-S_1^2 &=\mu_2 f(x)^2+o(1). \end{aligned}

Subtracting the first two expressions leaves \(h^2\mu_2^2 f(x)^2m''(x)/2+o(h^2)\). Dividing by the denominator proves the stated expansion. Since the fitted value at \(x\) is \(m(x)+a\), this is also the local linear smoothing bias. \(\square\)

The calculation explains the mechanism, not just the final formula. The local linear fit uses a slope as well as an intercept. Its normal equations account for the imbalance represented by \(S_1\); in the intercept, that adjustment removes the leading contribution from \(m'(x)f'(x)\). Curvature remains.

Worked Example: Quadratic Regression Under the Same Design

Keep the exponential design, target \(x=1/2\), and uniform kernel from the previous calculation, but set \(m(t)=t^2\). Then \(m'(1/2)=1\), \(m''(1/2)=2\), and \(f'(1/2)/f(1/2)=1\).

The local constant expansion has two contributions:

\begin{aligned} m_{\mathrm{LC},h}(1/2)-m(1/2) &=\frac{h^2}{3}\left(\frac{2}{2}+1\cdot1\right)+o(h^2)\\ &=\frac{2h^2}{3}+o(h^2). \end{aligned}

For local linear fitting, only the curvature term remains:

m_{\mathrm{LL},h}(1/2)-m(1/2) =\frac{h^2}{3}\frac{2}{2}+o(h^2) =\frac{h^2}{3}+o(h^2).

Both fits retain curvature bias, but the local linear fit removes the additional \(h^2/3\) contribution from the changing design density.

Worked Example: A Flat Design Density

Let \(X\) be uniform on \((0,1)\), \(m(t)=t^2\), and \(x=1/2\), again using the uniform kernel with \(\mu_2=1/3\) and a sufficiently small bandwidth. Now \(f'(x)/f(x)=0\), while \(m''(x)=2\). The local constant expansion becomes

m_{\mathrm{LC},h}(1/2)-m(1/2) =\frac{h^2}{3}+o(h^2).

The local linear expansion gives the same leading term:

m_{\mathrm{LL},h}(1/2)-m(1/2) =\frac{h^2}{3}+o(h^2).

This does not mean that the two estimators are identical. It means that, in this interior example, their leading order-\(h^2\) smoothing biases agree because the density-gradient contribution to local constant bias is zero.

Interpreting the Expansions

These formulas concern the leading deterministic smoothing bias. With random observations, empirical weighted sums fluctuate around their population counterparts. Under standard additional conditions, including \(nh\to\infty\) and a well-behaved local design matrix, the empirical fits have the displayed \(h^2\) terms as their leading smoothing-bias terms, alongside random variation. The expansion does not say that a realized estimate equals its population target, nor does it by itself give the complete finite-sample expectation.

The density-gradient term has a direct implication: local constant smoothing can behave differently at two points with identical regression curvature if the design density has different local slopes. Local linear fitting is less sensitive to that particular feature. This cancellation is an interior result; near a boundary, the kernel window is one-sided and its moments differ, so the boundary calculations require their own expansions.

Common Pitfall

Pitfall: “The kernel is symmetric, so the first-derivative contribution to local constant bias must vanish.” Symmetry makes \(\int uK(u)\,du=0\), but the density and regression function are multiplied before integration. Their first-order terms multiply to \(h^2u^2m'(x)f'(x)\), whose integral is generally nonzero. The correction is to expand the product and then integrate; for local constant fitting this produces \(h^2\mu_2m'(x)f'(x)/f(x)\). For local linear fitting, the intercept normal equations cancel that leading term.

Key takeaway: At an interior point, local constant smoothing has a leading bias contribution from both regression curvature and the product of regression slope with the log-density slope \(f'(x)/f(x)\). Local linear fitting removes the latter at order \(h^2\), leaving the curvature term.

Check Your Understanding

Use the bias expansions and assumptions developed above.

For local constant fitting, what happens to the density-gradient contribution when \(m'(x)=0\)? When \(f'(x)=0\)?
Explain why symmetry of \(K\) does not eliminate the local constant term involving \(m'(x)f'(x)\).
Suppose \(f'(x)/f(x)<0\) and \(m'(x)>0\). What is the sign of the density-gradient contribution to local constant bias?
For \(m(t)=t\), why does the local linear population fit have zero smoothing bias, while local constant fitting can have nonzero bias?
Which term remains in the leading local linear bias when \(m''(x)\ne 0\)?