Local Linear Regression and First-Order Bias Removal

Why Fit a Local Line?

Local constant regression estimates the response near a target point by averaging nearby responses. That is a natural starting point, but the average can be pulled toward regions where observations are more densely concentrated. When the regression function is changing with the predictor, this creates a leading bias term involving the design density. Local linear regression allows a slope as well as an intercept in the neighborhood. Its moment equations adjust for the local tilt in the data and remove that leading density contribution at an interior point.

The calculation is local: observations farther from the target receive less weight, and the fitted value at the target is the fitted intercept. We first derive that intercept from the weighted least-squares problem, then examine its moment equations and bias.

Definition and Sample Estimator

Suppose the observations are $(X_i,Y_i)$, $i=1,\ldots,n$, and the target predictor value is $x$. Let $K$ be a kernel and $h>0$ a bandwidth. Define the rescaled predictor $u_i=(X_i-x)/h$. A degree-one local polynomial fit chooses an intercept $a$ and a coefficient $c$ for the rescaled predictor by minimizing the weighted sum of squared residuals.

Definition: The local linear estimator at $x$ is the intercept $\hat a$ in $$ (\hat a,\hat c)=\arg\min_{a,c}\sum_{i=1}^n K(u_i)\{Y_i-a-cu_i\}^2, \qquad \hat m_{\mathrm{LL}}(x)=\hat a. $$ Here $c$ is the slope with respect to $u_i$; the slope with respect to the original predictor $X_i$ is $c/h$.

Define the weighted sample moments $S_j=\sum_{i=1}^n K(u_i)u_i^j$ and $T_j=\sum_{i=1}^n K(u_i)u_i^jY_i$, for $j=0,1,2$ as needed. Differentiating the objective with respect to $a$ and $c$ gives the two normal equations:

\begin{pmatrix} S_0&S_1\\ S_1&S_2 \end{pmatrix} \begin{pmatrix} \hat a\\ \hat c \end{pmatrix} = \begin{pmatrix} T_0\\ T_1 \end{pmatrix}.

If $D=S_0S_2-S_1^2>0$, the weighted design matrix is invertible, and solving gives

\hat a=\frac{S_2T_0-S_1T_1}{D}, \qquad \hat c=\frac{S_0T_1-S_1T_0}{D}.

The condition $D>0$ requires positive total weight and at least two distinct rescaled predictor values with positive weight. If it fails, the weighted data do not identify both a local intercept and a local slope. When $S_1=0$, the intercept simplifies to $T_0/S_0$, the local weighted average. But in general $S_1$ is not zero, even when the kernel itself is symmetric: the observed predictor values in a particular sample need not be symmetric around $x$.

Worked Example: Solving the Sample Moment Equations

At a target point $x$, suppose two observations have rescaled predictors $u_1=-1$ and $u_2=0$, positive kernel weights $1$ and $2$, and responses $Y_1=1$ and $Y_2=2$. Calculate the local linear intercept and compare it with the local constant weighted average.

The predictor moments are $S_0=1+2=3$, $S_1=(-1)(1)+0(2)=-1$, and $S_2=(-1)^2(1)+0^2(2)=1$. The response moments are $T_0=1(1)+2(2)=5$ and $T_1=(-1)(1)(1)+0(2)(2)=-1$. Thus $D=3(1)-(-1)^2=2>0$, and

\hat a=\frac{1(5)-(-1)(-1)}{2}=2, \qquad \hat c=\frac{3(-1)-(-1)(5)}{2}=1.

The fitted line is $2+u$: it gives $1$ at $u=-1$ and $2$ at $u=0$, so both weighted residuals are zero. The local constant estimate is instead $T_0/S_0=5/3$. The line's intercept is the fitted value at the target $u=0$, not the weighted average of the observed responses.

The Moment Equations and Affine Reproduction

The normal equations say that the weighted residuals have zero weighted sum and zero weighted first moment. In other words, the fitted residuals are orthogonal, under the kernel weights, to both the constant function and the rescaled predictor. This is the key distinction from a local constant fit, which enforces only the first condition.

Theorem: If $D>0$ and the responses satisfy $Y_i=\alpha+\beta u_i$ for all observations with positive weight, the local linear estimator satisfies $\hat a=\alpha$ and $\hat c=\beta$. Thus it reproduces every affine function exactly at the target.

Proof. Substitute $Y_i=\alpha+\beta u_i$ into the two response moments. Then $T_0=\alpha S_0+\beta S_1$ and $T_1=\alpha S_1+\beta S_2$. Consequently, the normal equations hold for $(a,c)=(\alpha,\beta)$. Since $D>0$, their solution is unique, so $(\hat a,\hat c)=(\alpha,\beta)$. $\square$

Worked Example: Exact Recovery of a Linear Regression Function

Let the target be $x=1$, the bandwidth be $h=0.2$, and the regression function be $m(X)=3+2X$. Suppose the rescaled predictors are $-1,0,1$, with kernel weights $1,2,1$. The corresponding predictor values are $0.8,1,1.2$, and the responses are $4.6,5,5.4$.

The moments are $S_0=4$, $S_1=0$, and $S_2=2$. The response moments are $T_0=4.6+2(5)+5.4=20$ and $T_1=-4.6+5.4=0.8$. Therefore $D=4(2)=8$, and

\hat a=\frac{2(20)}{8}=5, \qquad \hat c=\frac{4(0.8)}{8}=0.4.

At the target $X=1$, the true response is $m(1)=5$, which is exactly the fitted intercept. The coefficient $0.4$ is the slope in the rescaled coordinate $u$; dividing by $h=0.2$ gives the slope $2$ with respect to $X$. This exact reproduction holds for affine response values whenever the weighted design matrix is nonsingular, not just for this symmetric set of predictors.

Interior Bias: Why the Density Term Disappears

To isolate the bias mechanism, consider the population version of the fit at an interior point $x$. Assume the predictor has density $f$, with $f(x)>0$, and the conditional mean is $m(t)=E[Y\mid X=t]$. Suppose $K$ is symmetric, nonnegative, supported on a bounded interval, and has moments $\mu_j=\int u^jK(u)\,du$, with $\mu_0>0$ and $\mu_2>0$. In particular, symmetry gives $\mu_1=\mu_3=0$. Assume $f$ is continuously differentiable near $x$, and $m$ is twice continuously differentiable there. These conditions allow the local Taylor expansions below, with remainders uniform over the bounded kernel support as $h$ tends to zero.

For comparison, the population local constant estimate is the kernel-weighted conditional mean, equivalently the ratio of the integrals of $K(u)m(x+hu)f(x+hu)$ and $K(u)f(x+hu)$. Expanding both around $x$, using symmetry to remove the odd terms, and subtracting $m(x)$ gives

m_{\mathrm{LC},h}(x)-m(x) = h^2\frac{\mu_2}{\mu_0} \left\{\frac{m''(x)}{2} +m'(x)\frac{f'(x)}{f(x)}\right\} +o(h^2).

The term $m'(x)f'(x)/f(x)$ is the leading design-density contribution: it appears when the regression function is locally sloped and the density changes across the neighborhood. For the local linear fit, the population normal equations adjust for that tilt. The following derivation makes the cancellation explicit.

Theorem: Under the conditions just stated, let $(a_h,c_h)$ minimize $$ \int K(u)\{m(x+hu)-a-cu\}^2 f(x+hu)\,du, $$ and assume the weighted moment matrix is nonsingular for sufficiently small $h$. Then $$ a_h-m(x)=h^2\frac{\mu_2}{2\mu_0}m''(x)+o(h^2). $$ In particular, the coefficient of $h^2m'(x)f'(x)/f(x)$ is zero.

Proof. Write the Taylor expansion about $x$ as $m(x+hu)=m(x)+hu\,m'(x)+r_h(u)$, where $r_h(u)=\tfrac12h^2u^2m''(x)+o(h^2)$ uniformly on the support of $K$. Fitting $m(x)+hu\,m'(x)$ already accounts for the constant and linear terms exactly. Thus write $a_h=m(x)+\delta_a$ and $c_h=hm'(x)+\delta_c$; the corrections $(\delta_a,\delta_c)$ are the weighted least-squares coefficients for fitting $r_h(u)$ by a constant and a multiple of $u$.

Let $M_h$ be the $2$-by-$2$ moment matrix with entries $\int u^{j+k}K(u)f(x+hu)\,du$, for $j,k\in\{0,1\}$. Let $b_h$ have entries $\int u^jK(u)f(x+hu)r_h(u)\,du$, for $j\in\{0,1\}$. The normal equations give $(\delta_a,\delta_c)^\mathsf{T}=M_h^{-1}b_h$. Continuity of $f$ and the uniform expansion for $r_h$ imply

M_h\longrightarrow f(x)\begin{pmatrix}\mu_0&\mu_1\\\mu_1&\mu_2\end{pmatrix}, \qquad \frac{b_h}{h^2}\longrightarrow \frac{f(x)m''(x)}{2} \begin{pmatrix}\mu_2\\\mu_3\end{pmatrix}.

The limiting matrix is $f(x)\operatorname{diag}(\mu_0,\mu_2)$, which is invertible because $f(x)>0$, $\mu_0>0$, and $\mu_2>0$. Matrix inversion is continuous at an invertible matrix. Since $\mu_3=0$, the first component of $M_h^{-1}b_h/h^2$ tends to $m''(x)\mu_2/(2\mu_0)$. That component is $\delta_a/h^2$, proving the stated expansion. The density $f(x)$ cancels, and no $f'(x)$ term occurs at this order. $\square$

Worked Example: Comparing the Two Interior Biases

Let $X$ have density $f(t)=2(1+t)/3$ on $[0,1]$, let $m(t)=t^2$, and take $x=0.5$. Use a uniform symmetric kernel on $[-1,1]$, so $\mu_2/\mu_0=1/3$, and set $h=0.1$. At the target, $f(x)=1$, $f'(x)=2/3$, $m'(x)=1$, and $m''(x)=2$. In particular, $f'(x)/f(x)=2/3$.

The local constant leading bias is

0.1^2\left\{\frac{2}{2} +1\left(\frac{2}{3}\right)\right\}\frac{1}{3} = 0.01\left(\frac{5}{3}\right)\frac{1}{3} = \frac{1}{180} \approx 0.005556.

The local linear leading bias is instead

0.1^2\frac{2}{2}\frac{1}{3} = \frac{1}{300} \approx 0.003333.

The difference, about $0.002222$, is exactly the leading density contribution $h^2m'(x)\{f'(x)/f(x)\}(\mu_2/\mu_0)$. This comparison concerns the population leading terms; it does not say that every finite-sample local linear estimate has smaller error.

What Is Removed—and What Is Not

Local linear regression removes the leading interior bias contribution caused by a changing design density and a nonzero local slope. It does not remove the curvature bias: the term involving $m''(x)$ remains. Nor does the argument claim that the estimator is unbiased for finite samples. The population expansion describes the leading behavior as the bandwidth shrinks under the stated smoothness and interior-point conditions.

A common mistake is to say that local linear regression “eliminates bias.” The moment equations show a more precise fact: fitting a line reproduces an affine regression function exactly, and the population intercept has no leading $h^2m'(x)f'(x)/f(x)$ term. If the regression function has curvature, its local departure from a line still contributes bias. The interior expansion also relies on symmetry and on staying away from the boundary; with one-sided neighborhoods, the relevant kernel moments need not have the same cancellations.

Key takeaway: The local linear intercept is determined by two weighted moment equations, not simply by a weighted average. Its local slope lets the fit accommodate an uneven distribution of predictor values, removing the leading design-density term from interior bias while leaving curvature bias.

Check Your Understanding

Use the normal equations and the bias expansions to answer these questions.

What condition on $S_0,S_1,S_2$ ensures that the sample local linear intercept and slope are uniquely determined?
Why can a symmetric kernel still produce $S_1\ne 0$ in a particular sample?
For an affine regression function, why does the local linear intercept equal the function value at the target whenever the weighted design matrix is nonsingular?
Which term in the local constant interior bias is removed from the leading local linear bias?
Does the local linear bias expansion imply that the estimator has zero bias when $m''(x)\ne 0$? Explain.