Kernel Regression as Local Weighted Least Squares

From a Local Polynomial to a Weighted Least-Squares Problem

To estimate a regression function near a point, local polynomial regression fits a polynomial using observations near that point, giving more influence to observations with larger kernel weights. This description becomes precise as a weighted least-squares problem: the polynomial coefficients minimize a weighted sum of squared residuals. Once that problem is written in matrix form, the fitted value at the target follows directly from the coefficient vector.

Suppose the observed data are $(X_i,Y_i)$, $i=1,\ldots,n$, where each $X_i$ is a scalar predictor and each $Y_i$ is a response. Fix a target point $x_0$, a bandwidth $h>0$, a polynomial degree $p\geq 0$, and a nonnegative kernel function $K$. Define the standardized predictor coordinate $u_i=(X_i-x_0)/h$. The local polynomial uses powers of this coordinate, so its intercept represents the fitted response at $x_0$.

Definition: A degree-$p$ local polynomial fit at $x_0$ minimizes $$ \sum_{i=1}^n K(u_i)\left\{Y_i-\sum_{j=0}^p\beta_j u_i^j\right\}^2, \qquad u_i=\frac{X_i-x_0}{h}. $$ Here $\beta_0,\ldots,\beta_p$ are the coefficients being fitted. If the minimizer is unique, the fitted value at the target is $\hat m_p(x_0)=\hat\beta_0$, since $u=0$ at $X=x_0$.

The weight is often written $K((X_i-x_0)/h)$. Some conventions include a factor $1/h$, defining $K_h(X_i-x_0)=K((X_i-x_0)/h)/h$. For a fixed positive bandwidth, multiplying every weight by the same positive constant multiplies the objective by that constant and does not change its minimizer. Thus the factor $1/h$ does not affect the fitted coefficients.

Design Matrix, Weights, and Response Vector

For each observation, place the polynomial basis values $1,u_i,\ldots,u_i^p$ in one row. Put the responses in a vector and the kernel weights on the diagonal of a matrix. This yields the complete weighted least-squares representation.

R= \begin{pmatrix} 1&u_1&u_1^2&\cdots&u_1^p\\ 1&u_2&u_2^2&\cdots&u_2^p\\ \vdots&\vdots&\vdots&&\vdots\\ 1&u_n&u_n^2&\cdots&u_n^p \end{pmatrix}, \qquad Y= \begin{pmatrix}Y_1\\Y_2\\\vdots\\Y_n\end{pmatrix}, \qquad W=\operatorname{diag}\{K(u_1),\ldots,K(u_n)\}.

Writing $\beta=(\beta_0,\ldots,\beta_p)^\mathsf{T}$, the objective is $(Y-R\beta)^\mathsf{T}W(Y-R\beta)$. The matrix $R$ is the design matrix for this local fit, $Y$ is the response vector, and $W$ records how strongly each observation contributes. Observations with zero kernel weight contribute nothing to the objective.

Theorem: If $W$ has nonnegative diagonal entries and $R^\mathsf{T}WR$ is invertible, the weighted least-squares minimizer is $$ \hat\beta=(R^\mathsf{T}WR)^{-1}R^\mathsf{T}WY. $$ The fitted value at $x_0$ is $$ \hat m_p(x_0)=e_1^\mathsf{T}\hat\beta =e_1^\mathsf{T}(R^\mathsf{T}WR)^{-1}R^\mathsf{T}WY, $$ where $e_1=(1,0,\ldots,0)^\mathsf{T}$ selects the intercept.

Invertibility is the condition that makes the coefficient vector uniquely determined by the weighted data. In particular, there must be enough linearly independent polynomial basis rows among observations with positive weights. The result does not require a probability model for the responses: it is an algebraic statement about the stated objective.

Why the Normal-Equation Formula Holds

The formula can be derived by expanding the objective and identifying its minimizer. Let $Q(\beta)=(Y-R\beta)^\mathsf{T}W(Y-R\beta)$. Because $W$ is symmetric,

Q(\beta) =Y^\mathsf{T}WY-2\beta^\mathsf{T}R^\mathsf{T}WY +\beta^\mathsf{T}R^\mathsf{T}WR\beta.

Set $A=R^\mathsf{T}WR$ and $b=R^\mathsf{T}WY$. The matrix $A$ is positive definite when it is invertible: for any nonzero vector $v$, $v^\mathsf{T}Av=(Rv)^\mathsf{T}W(Rv)\geq 0$, so $A$ is positive semidefinite; an invertible positive semidefinite matrix is positive definite. Now let $\hat\beta=A^{-1}b$. Completing the square gives

Q(\beta) =Y^\mathsf{T}WY-b^\mathsf{T}A^{-1}b +(\beta-A^{-1}b)^\mathsf{T}A(\beta-A^{-1}b).

The final term is nonnegative and equals zero exactly when $\beta=A^{-1}b$, since $A$ is positive definite. Hence this choice uniquely minimizes $Q$, proving the coefficient formula. At $X=x_0$, the standardized coordinate is zero, and the fitted polynomial is $\hat\beta_0+\hat\beta_1(0)+\cdots+\hat\beta_p(0)^p=\hat\beta_0$. This proves the target-value formula as well. $\square$

Worked Examples

Worked Example: Degree Zero Gives a Weighted Average

Take $x_0=0$, $h=1$, and the triangular kernel $K(u)=1-|u|$ for $|u|\leq 1$, with weight zero outside this interval. Observe $(X_i,Y_i)=(-0.5,1),(0,3),(0.5,5)$. Find the degree-zero fit at zero.

For degree zero, each row of $R$ contains just a 1. The standardized coordinates are $-0.5,0,0.5$, so the weights are $0.5,1,0.5$. The coefficient minimizes the weighted sum of squared differences from a single constant. The normal equation is

(0.5+1+0.5)\hat\beta_0 =0.5(1)+1(3)+0.5(5)=6.

The total weight is 2, so $\hat\beta_0=6/2=3$. The degree-zero fitted value at the target is therefore $\hat m_0(0)=3$.

Worked Example: A Degree-One Fit from Its Matrix

Use $x_0=0$, $h=1$, and the same triangular kernel. The observations are $(-0.5,2),(0,1),(0.5,4)$, with standardized coordinates $-0.5,0,0.5$ and weights $0.5,1,0.5$. Multiplying all weights by 2 has no effect on the minimizer, so use $W=\operatorname{diag}(1,2,1)$. Find the degree-one fitted value at zero.

The design matrix has rows $(1,u_i)$. The weighted cross-product and response product are

R^\mathsf{T}WR= \begin{pmatrix}4&0\\0&0.5\end{pmatrix}, \qquad R^\mathsf{T}WY= \begin{pmatrix}8\\1\end{pmatrix}.

For example, the upper-right entry is $1(-0.5)+2(0)+1(0.5)=0$, and the second component of $R^\mathsf{T}WY$ is $1(-0.5)(2)+2(0)(1)+1(0.5)(4)=1$. The normal equations give

\hat\beta_0=\frac{8}{4}=2, \qquad \hat\beta_1=\frac{1}{0.5}=2.

Thus the fitted local line is $2+2u$, and its value at the target $u=0$ is $\hat m_1(0)=2$.

Worked Example: A Degree-Two Fit with a Uniform Kernel

Take $x_0=0$, $h=1$, and $K(u)=1/4$ for $|u|\leq2$, zero otherwise. This is a nonnegative kernel with integral one. The four observations are $(-1,1),(0,2),(1,2),(2,5)$. All four receive the same positive weight, so the common factor $1/4$ can be omitted. Find the degree-two fitted value at zero.

The rows of $R$ are $(1,u_i,u_i^2)$. With $W$ taken as the identity matrix after omitting the common factor, summing the powers of the four coordinates gives

R^\mathsf{T}R= \begin{pmatrix} 4&2&6\\ 2&6&8\\ 6&8&18 \end{pmatrix}, \qquad R^\mathsf{T}Y= \begin{pmatrix}10\\11\\23\end{pmatrix}.

The normal equations are $4\beta_0+2\beta_1+6\beta_2=10$, $2\beta_0+6\beta_1+8\beta_2=11$, and $6\beta_0+8\beta_1+18\beta_2=23$. Subtracting half the first equation from the second gives $5\beta_1+5\beta_2=6$. Subtracting $1.5$ times the first equation from the third gives $5\beta_1+9\beta_2=8$. Their difference yields $4\beta_2=2$, so $\hat\beta_2=1/2$, then $\hat\beta_1=7/10$. Substitution into the first equation gives $4\hat\beta_0+1.4+3=10$, hence $\hat\beta_0=7/5$. Therefore $\hat m_2(0)=7/5$.

The Fitted Value as a Linear Combination of Responses

When the weighted design matrix has full column rank, the target fitted value is also a linear combination of the observed responses. Define the row vector of coefficients

\ell^\mathsf{T} =e_1^\mathsf{T}(R^\mathsf{T}WR)^{-1}R^\mathsf{T}W.

Then $\hat m_p(x_0)=\ell^\mathsf{T}Y=\sum_{i=1}^n\ell_iY_i$. These coefficients depend on the predictor values, target, bandwidth, kernel, and polynomial degree, but not on the observed response values. They are generally not the same as the diagonal kernel weights: kernel weights define the least-squares objective, while $\ell_i$ describe the resulting fitted value.

Result: If the fit includes an intercept and $R^\mathsf{T}WR$ is invertible, the response-combination coefficients sum to one: $\sum_{i=1}^n\ell_i=1$.

To verify this, let $\mathbf{1}$ denote the $n$-vector of ones. The first column of $R$ is $\mathbf{1}$, so $Re_1=\mathbf{1}$. Therefore $R^\mathsf{T}W\mathbf{1}=R^\mathsf{T}WRe_1$. Substituting into the coefficient sum gives

\ell^\mathsf{T}\mathbf{1} =e_1^\mathsf{T}(R^\mathsf{T}WR)^{-1}R^\mathsf{T}W\mathbf{1} =e_1^\mathsf{T}(R^\mathsf{T}WR)^{-1}(R^\mathsf{T}WR)e_1 =e_1^\mathsf{T}e_1=1.

This identity does not say that each $\ell_i$ is nonnegative. The kernel weights in $W$ are nonnegative by assumption, but the coefficients obtained after fitting a polynomial need not be. The identity is instead a consequence of including the constant basis function.

Common Mistakes

Using the wrong target coordinate. The polynomial is written in $u_i=(X_i-x_0)/h$, so at the target $x_0$, $u=0$. Its fitted value is the intercept, not the sum of all fitted coefficients.
Confusing $W$ with the response-combination coefficients. $W$ contains the kernel weights used in the objective. The final coefficients $\ell_i$ also depend on the design matrix and need not equal those weights.
Forgetting the rank condition. If $R^\mathsf{T}WR$ is singular, the inverse formula is unavailable and the coefficients may not be uniquely determined. Check the positively weighted design before using it.
Thinking a common weight factor changes the fit. Multiplying every weight by a positive constant changes the objective's scale, not its minimizer.

Key takeaway: Build a local polynomial fit by forming the basis matrix from standardized predictor coordinates, placing kernel weights in a diagonal matrix, and solving the weighted normal equations. The fitted value at the target is the estimated intercept.

Check Your Understanding

Use the matrix formulation to answer the following questions.

For a degree-two fit, what are the entries in the design-matrix row for an observation with standardized coordinate $u_i$?
Why does the fitted value at $x_0$ equal the intercept when the polynomial is expressed in standardized coordinates centered at $x_0$?
What condition allows the inverse in the weighted least-squares formula to exist?
For a degree-zero fit, explain why the fitted coefficient is a kernel-weighted average of the responses with positive weights.
Why does multiplying all kernel weights by the same positive constant leave the fitted coefficients unchanged?