From the Local Fit to a Number
Local linear regression estimates the regression function near a target point by fitting a weighted line to nearby observations. Its formulas are compact, but hand calculations can go wrong when a determinant, a weight, or a residual degrees-of-freedom correction is miscomputed. The reliable approach is to write down the weighted sums first, solve the two normal equations, and then check the result against both the fit and its equivalent weights.
Suppose the observed data are \((x_i,Y_i)\), \(i=1,\ldots,n\), and the target is \(x_0\). For bandwidth \(h>0\), define \(u_i=(x_i-x_0)/h\) and let \(q_i=K(u_i)\geq 0\) be the weight assigned to observation \(i\). A local linear fit minimizes the weighted sum of squared residuals over an intercept \(a\) and a slope \(c\) in the rescaled coordinate \(u\). The fitted value at \(x_0\) is the fitted intercept \(a\), since \(u=0\) there.
The weighted normal equations use the additional sums \(T_k=\sum_i q_i u_i^kY_i\), for \(k=0,1\):
When \(D>0\), solving this two-by-two system gives
The entries \(S_k\) depend on the design and kernel weights, while \(T_k\) also depend on the observed responses. Keep those sums distinct: they make the arithmetic auditable and reveal whether the final coefficients satisfy the normal equations.
Worked Fits and Equivalent Weights
A fitted value can also be written as a weighted sum of the observed responses. These are the equivalent weights: they show exactly how each response contributes to the estimate at the target. They can be positive or negative, particularly near a boundary.
To verify the formula, write the weighted least-squares solution as \((Z'WZ)^{-1}Z'WY\), where row \(i\) of \(Z\) is \((1,u_i)\) and \(W\) is diagonal with entries \(q_i\). The first row of \((Z'WZ)^{-1}\) is \((S_2,-S_1)/D\). Multiplying this row by \(Z'W\) gives the coefficient of \(Y_i\) as \(q_i(S_2-S_1u_i)/D\), as claimed. Summing these coefficients gives \((S_2S_0-S_1S_1)/D=1\). Their weighted first moment is \((S_2S_1-S_1S_2)/D=0\). Thus the estimate reproduces a constant response exactly and has zero response to a linear trend in \(u\).
Worked Example: A Symmetric Interior Fit
Take \(x_0=0\), \(h=2\), a box kernel with \(q_i=1\) for \(|u_i|\leq 1\), and the five observations shown below. The resulting rescaled locations are symmetric around zero.
| \(u_i\) | \(Y_i\) |
|---|---|
| -1 | 1 |
| -0.5 | 2 |
| 0 | 2 |
| 0.5 | 4 |
| 1 | 5 |
Calculate the design sums: \(S_0=5\), \(S_1=-1-0.5+0+0.5+1=0\), and \(S_2=1+0.25+0+0.25+1=2.5\). Therefore \(D=5(2.5)-0^2=12.5\). The response sums are \(T_0=1+2+2+4+5=14\) and \(T_1=(-1)(1)+(-0.5)(2)+0+(0.5)(4)+(1)(5)=5\).
The fitted intercept and slope in the rescaled coordinate are
As a check, the fitted values are \(2.8+2u_i\), namely \(0.8,1.8,2.8,3.8,4.8\). Their residuals are \(0.2,0.2,-0.8,0.2,0.2\). The equivalent weights are \(l_i=q_iS_2/D=2.5/12.5=0.2\) for every observation, so \(\sum_i l_iY_i=0.2(14)=2.8\). Symmetry is why the interior estimate here is the ordinary average.
Worked Example: A Fit Near a Boundary
Now use \(x_0=0\), \(h=3\), and a box kernel on the observations \((x_i,Y_i)=(0,1),(1,3),(2,2),(3,5)\). The target is at the left edge of the observed locations. The rescaled locations are \(u_i=0,1/3,2/3,1\), with all four weights equal to one.
The sums are \(S_0=4\), \(S_1=0+1/3+2/3+1=2\), and \(S_2=0+1/9+4/9+1=14/9\). Thus
For the responses, \(T_0=1+3+2+5=11\) and \(T_1=0+(1/3)(3)+(2/3)(2)+5=22/3\). Consequently,
and
The fit is \(1.1+3.3u\), giving fitted responses \(1.1,2.2,3.3,4.4\). Substituting into the normal equations checks the arithmetic: \(4(1.1)+2(3.3)=11=T_0\), and \(2(1.1)+(14/9)(3.3)=2.2+5.133\ldots=22/3=T_1\).
The equivalent weights are \(l_i=(S_2-S_1u_i)/D\). At \(u=0,1/3,2/3,1\), these are respectively \(0.7,0.4,0.1,-0.2\). They sum to one, and their weighted location sum is \(0(0.7)+(1/3)(0.4)+(2/3)(0.1)+1(-0.2)=0\). The fitted value is \(0.7(1)+0.4(3)+0.1(2)-0.2(5)=1.1\). The negative weight is not an arithmetic error: the one-sided fit uses a negative contribution from the farthest point to correct the slope and estimate at the boundary.
Worked Example: Nonuniform Kernel Weights at an Interior Point
Let \(x_0=0\), \(h=2\), and use the triangular kernel \(K(u)=1-|u|\) for \(|u|\leq1\), and zero otherwise. For observations at \(x=-1,0,1\) with responses \(1,2,4\), the rescaled locations are \(-0.5,0,0.5\) and the weights are \(0.5,1,0.5\).
Here \(S_0=2\), \(S_1=(-0.5)(0.5)+0+(0.5)(0.5)=0\), and \(S_2=0.5(0.25)+0+0.5(0.25)=0.25\). Thus \(D=2(0.25)=0.5\). Also, \(T_0=0.5(1)+1(2)+0.5(4)=4.5\) and \(T_1=0.5(-0.5)(1)+0+0.5(0.5)(4)=0.75\). The fit is
The equivalent weights are \(l_i=q_iS_2/D\), because \(S_1=0\). They are \(0.25,0.5,0.25\), and \(0.25(1)+0.5(2)+0.25(4)=2.25\). The fitted values at the three locations are \(0.75,2.25,3.75\). This example illustrates that symmetry makes the equivalent weights proportional to the kernel weights, but the fitted intercept need not equal the unweighted average.
Estimating the Standard Error
The standard error calculation requires a noise model. Treat the design points and weights as fixed, assume \(Y=Z\beta+\varepsilon\), and suppose the errors have conditional mean zero and common conditional variance \(\sigma^2\), with no conditional covariances between distinct errors. Under this linear mean model, weighted residuals estimate the noise variance. The assumption that the mean is exactly linear over the included locations is important for unbiasedness; if the true regression function is curved, residual variation can also include approximation error.
For the variance-estimator claim, \(H^2=H\) and \(H'W=WH\), as follows directly from \(H=Z(Z'WZ)^{-1}Z'W\). Since the mean \(Z\beta\) is in the column space of \(Z\), \(e=(I-H)\varepsilon\). Therefore
The last equality uses \(H'W=WH\) and \(H^2=H\). Dividing by the stated trace gives an unbiased estimator. Separately, the equivalent-weight representation gives \(\hat a=\sum_i l_iY_i\). Independence of the errors and their common variance imply \(\operatorname{Var}(\hat a)=\operatorname{Var}(\sum_i l_i\varepsilon_i)=\sigma^2\sum_i l_i^2\). These two calculations justify the plug-in standard error.
With equal unit weights and \(m\) positive-weight observations, \(H\) is the ordinary rank-two projection, so \(\operatorname{tr}\{W(I-H)\}=m-2\). With equal weights \(q_i=c>0\), the trace is instead \(c(m-2)\), while \(e'We=c\,e'e\); the ratio is still the usual residual sum of squares divided by \(m-2\).
Worked Example: Boundary Standard Error
Continue the boundary example, with four unit weights and fit \(1.1+3.3u\). Its residuals are \(-0.1,0.8,-1.3,0.6\), so the residual sum of squares is
The residual variance estimate is \(\hat\sigma^2=2.70/(4-2)=1.35\). The squared equivalent weights sum to \(0.7^2+0.4^2+0.1^2+(-0.2)^2=0.49+0.16+0.01+0.04=0.70\). Thus the estimated variance of the fitted value is \(1.35(0.70)=0.945\), and its estimated standard error is \(\sqrt{0.945}\approx0.972\). This plug-in calculation quantifies noise under the stated linear-mean and homoskedastic-error assumptions; it does not account for bias from curvature.
Common Mistakes and Practical Checks
A determinant error propagates into both coefficients, so calculate it separately before dividing. For instance, if \(S_0=5\), \(S_1=2.5\), and \(S_2=1.875\), then
not \(0.625\). If \(T_0=16\) and \(T_1=9.5\), the coefficient numerators are \(1.875(16)-2.5(9.5)=6.25\) and \(5(9.5)-2.5(16)=7.5\). Dividing by the correct determinant gives \(\hat a=6.25/3.125=2\) and \(\hat c=7.5/3.125=2.4\). The normal equations confirm the result: \(5(2)+2.5(2.4)=16\) and \(2.5(2)+1.875(2.4)=9.5\). Dividing by an incorrect determinant produces coefficients that fail these checks.
Another mistake is to use \(m-2\) as the denominator for every weighted residual sum of squares. That is correct for unit weights; with common weights \(c\), the trace denominator is \(c(m-2)\). For unequal weights, use \(\operatorname{tr}\{W(I-H)\}\), not an assumed count-based correction. Finally, do not assume all equivalent weights are positive: their sum must be one, and their first moment must be zero, but boundary correction can require negative weights.
Compute \(S_0,S_1,S_2,T_0,T_1\) directly from the stated locations, responses, and weights.
Calculate \(D=S_0S_2-S_1^2\), find both coefficients, and substitute them into both normal equations.
Confirm that equivalent weights sum to one and have zero first moment, then use a variance estimator appropriate to the weights and error assumptions.
The calculations matter because the local fit is more than a pair of coefficients: its equivalent weights expose leverage, boundary behavior, and the sensitivity of the fitted value to individual responses. Checking sums, determinants, normal equations, and variance denominators turns a fragile hand calculation into a reproducible one.
Check Your Understanding
Use the formulas and checks above to work through these questions.
- For a symmetric design with equal weights, why is \(S_1=0\), and how does that simplify the fitted intercept?
- For the boundary example, verify directly that the four equivalent weights sum to one and their weighted rescaled locations sum to zero.
- Explain why a negative equivalent weight can occur near a boundary without violating the defining properties of the weights.
- If four observations have equal weights \(q_i=2\), what is \(\operatorname{tr}\{W(I-H)\}\) for a rank-two local linear fit? What happens to the ratio of weighted residual sum of squares to this denominator?
- Under what error and mean assumptions is the residual-based variance estimate in this tutorial unbiased?