Deriving the Local Polynomial Variance

Why Derive the Variance?

The variance of a local polynomial fitted value depends not only on the noise level but also on how the observations are weighted and how many predictors fall near the target point. Under random design, the local number of observations is approximately proportional to $nhf(x)$: it increases with sample size $n$, bandwidth $h$, and design density $f(x)$. The kernel and polynomial degree determine an additional constant.

We derive that constant from the weighted least-squares representation. The result is conditional on the observed predictors, then approximated in probability over random designs. Keeping those two steps distinct is important: the exact conditional variance uses the noise variances at the observed predictor values, while the leading approximation uses their values near the target point.

Setup and Exact Conditional Variance

Let $X_1,\ldots,X_n$ be independent and identically distributed predictors with density $f$, and let $Y_i=m(X_i)+\varepsilon_i$. Fix an interior target point $x$, a polynomial degree $p$, and a bandwidth $h$. Write $U_i=(X_i-x)/h$ and $r_p(u)=(1,u,\ldots,u^p)^T$. The local polynomial coefficient vector minimizes the weighted sum of squared residuals using weights $K(U_i)$.

Definition: Let $e_0=(1,0,\ldots,0)^T$, the vector selecting the intercept. When the weighted design matrix is invertible, define $$ \widehat\beta_p(x)=\left\{\sum_{i=1}^n K(U_i)r_p(U_i)r_p(U_i)^T\right\}^{-1} \sum_{i=1}^n K(U_i)r_p(U_i)Y_i, \qquad \widehat m_p(x)=e_0^T\widehat\beta_p(x). $$

The coefficient vector uses powers of the rescaled distance $U_i$, so its first entry is the fitted value at $x$. For the conditional variance calculation, assume that, conditional on the full predictor vector $(X_1,\ldots,X_n)$, the errors are independent, have conditional means zero, and satisfy $\operatorname{Var}(\varepsilon_i\mid X_1,\ldots,X_n)=\sigma^2(X_i)$. These are assumptions on the full-design conditional moments.

Result: Conditional on all predictors, the fitted value has variance $$ \operatorname{Var}\{\widehat m_p(x)\mid X_1,\ldots,X_n\} = e_0^T H_n^{-1}Q_nH_n^{-1}e_0, $$ where $$ H_n=\sum_{i=1}^n K(U_i)r_p(U_i)r_p(U_i)^T, \qquad Q_n=\sum_{i=1}^n K(U_i)^2\sigma^2(X_i)r_p(U_i)r_p(U_i)^T. $$

To verify this formula, conditional on the predictors the fitted coefficient is a fixed linear transformation of the response vector. Its error-dependent part is $H_n^{-1}\sum_i K(U_i)r_p(U_i)\varepsilon_i$. Conditional independence makes the covariance matrix of this sum equal to $Q_n$; multiplying on the left and right by $H_n^{-1}$ gives the coefficient covariance. Selecting its first component gives the displayed variance.

Worked Example: The Exact Variance for a Local Constant Fit

For degree $p=0$, $r_0(U_i)=1$, so the fitted value is $\widehat m_0(x)=\sum_i K(U_i)Y_i/\sum_i K(U_i)$, provided the denominator is nonzero. The exact conditional variance is therefore $$ \operatorname{Var}\{\widehat m_0(x)\mid X_1,\ldots,X_n\} = \frac{\sum_i K(U_i)^2\sigma^2(X_i)} {\{\sum_i K(U_i)\}^2}. $$

The squared kernel weights in the numerator arise because the variance of a weighted sum involves squared coefficients. Replacing $K(U_i)^2$ by $K(U_i)$ would generally give a different, incorrect variance.

Leading Variance Under Random Design

Assume $K$ is bounded, nonnegative, and supported on $[-1,1]$. Define the kernel moment matrices $$ S_p=\int_{-1}^1 K(u)r_p(u)r_p(u)^T\,du, \qquad T_p=\int_{-1}^1 K(u)^2r_p(u)r_p(u)^T\,du. $$ Assume $S_p$ is positive definite. Suppose $f$ is continuous and positive at $x$, and $\sigma^2$ is bounded and continuous near $x$. Take $h\to0$ and $nh\to\infty$, with $p$ fixed.

Theorem: Under these conditions and the full-design conditional error assumptions above, $$ \operatorname{Var}\{\widehat m_p(x)\mid X_1,\ldots,X_n\} = \frac{\sigma^2(x)}{nhf(x)} \,e_0^TS_p^{-1}T_pS_p^{-1}e_0 + o_p\!\left(\frac{1}{nh}\right). $$ Equivalently, the conditional variance multiplied by $nhf(x)/\sigma^2(x)$ converges in probability to $e_0^TS_p^{-1}T_pS_p^{-1}e_0$, when $\sigma^2(x)>0$.

Here $o_p(1/(nh))$ means that after division by $1/(nh)$, the remainder converges to zero in probability over the random predictors. In particular, the leading order is $1/(nhf(x))$, multiplied by the noise variance and a constant determined by $K$ and $p$.

Proof. Normalize the two matrices in the exact variance by defining $A_n=H_n/(nh)$ and $B_n=Q_n/(nh)$. Then the exact variance is $$ \frac{1}{nh}e_0^TA_n^{-1}B_nA_n^{-1}e_0. $$ For an entry of $A_n$, its expectation is $$ E[A_n]_{jk} = \int_{-1}^1 K(u)u^{j+k}f(x+hu)\,du, \qquad 0\leq j,k\leq p. $$ Continuity of $f$ at $x$ and boundedness of the integrand imply that this converges to $f(x)(S_p)_{jk}$. Each summand before averaging is bounded and is zero unless $X_i$ lies within distance $h$ of $x$. Since $f$ is bounded near $x$, the probability of that event is at most a constant times $h$. Thus the variance of each entry of $A_n$ is at most $C/(nh)$ for a fixed constant $C$. Chebyshev's inequality and $nh\to\infty$ give $A_n\xrightarrow{p}f(x)S_p$.

The same argument applies to $B_n$. Its entry expectations are $$ E[B_n]_{jk} = \int_{-1}^1 K(u)^2u^{j+k}\sigma^2(x+hu)f(x+hu)\,du, $$ which converge to $f(x)\sigma^2(x)(T_p)_{jk}$. Boundedness of $\sigma^2$ near $x$ gives the same $C/(nh)$ variance bound, so $B_n\xrightarrow{p}f(x)\sigma^2(x)T_p$.

Because $f(x)>0$ and $S_p$ is positive definite, $f(x)S_p$ is invertible. Matrix inversion and multiplication are continuous at these limiting matrices. Therefore, with probability tending to one, $A_n$ is invertible and $$ e_0^TA_n^{-1}B_nA_n^{-1}e_0 \xrightarrow{p} \frac{\sigma^2(x)}{f(x)}e_0^TS_p^{-1}T_pS_p^{-1}e_0. $$ Substitution into the normalized exact variance proves the result. $\square$

Worked Variance Constants

For a symmetric kernel, odd moments vanish. That can simplify the matrix calculation, but the variance constant still depends on the polynomial degree: fitting higher-degree terms changes the inverse moment matrix, even though only the intercept is reported.

Worked Example: Local Linear Fit With a Uniform Kernel

Take $K(u)=1/2$ for $|u|\leq1$, zero otherwise, and $p=1$. Direct integration gives $$ S_1= \begin{pmatrix} 1&0\\ 0&1/3 \end{pmatrix}, \qquad T_1= \begin{pmatrix} 1/2&0\\ 0&1/6 \end{pmatrix}. $$ Thus $S_1^{-1}=\operatorname{diag}(1,3)$, and $e_0^TS_1^{-1}=(1,0)$. It follows that $$ e_0^TS_1^{-1}T_1S_1^{-1}e_0=\frac12. $$ The leading conditional variance is consequently $\sigma^2(x)/(2nhf(x))$. The slope coefficient does not enter this final scalar, but including it in the fit is reflected in the moment-matrix calculation.

Worked Example: Local Quadratic Fit With a Uniform Kernel

Use the same uniform kernel and set $p=2$. The even moments give $$ S_2= \begin{pmatrix} 1&0&1/3\\ 0&1/3&0\\ 1/3&0&1/5 \end{pmatrix}, \qquad T_2= \begin{pmatrix} 1/2&0&1/6\\ 0&1/6&0\\ 1/6&0&1/10 \end{pmatrix}. $$ To compute the constant, first solve $S_2v=e_0$. The middle equation gives $v_1=0$. The other two equations are $v_0+v_2/3=1$ and $v_0/3+v_2/5=0$, whose solution is $v=(9/4,0,-15/4)^T$. Therefore $$ e_0^TS_2^{-1}T_2S_2^{-1}e_0 =v^TT_2v =\frac{81}{32}-\frac{45}{16}+\frac{45}{32} =\frac98. $$ The leading variance is $9\sigma^2(x)/(8nhf(x))$. The calculation illustrates why the variance constant must be computed from the correct inverse moment matrix; it is not obtained merely by using the local constant or local linear constant.

Worked Example: Local Constant Fit With a Gaussian Kernel

Let $K(u)=(2\pi)^{-1/2}\exp(-u^2/2)$ on the real line and take $p=0$. The same derivation applies with integrals over the real line; the Gaussian kernel and its square have finite integrals. Here $S_0=\int K(u)\,du=1$, while $$ T_0=\int_{-\infty}^{\infty}K(u)^2\,du =\frac{1}{2\pi}\int_{-\infty}^{\infty}e^{-u^2}\,du =\frac{1}{2\sqrt{\pi}}. $$ Since $S_0=1$, the variance constant is $1/(2\sqrt{\pi})$, and the leading variance is $\sigma^2(x)/(2\sqrt{\pi}\,nhf(x))$. The constant involves the integral of the squared kernel, not the integral of the kernel, which is one.

Interpretation and a Common Pitfall

The factor $1/f(x)$ expresses a local information effect. At a point with lower design density, fewer observations typically fall in a bandwidth-sized neighborhood, so the variance is larger. Increasing $n$ or $h$ reduces the leading variance, although increasing $h$ changes the amount of smoothing and is not a variance-only decision. The conditional noise variance $\sigma^2(x)$ describes the local response noise; the moment-matrix constant describes how the chosen weighted fit combines observations.

Pitfall: conditioning on too little. From $E[\varepsilon_i\mid X_i]=0$ and $\operatorname{Var}(\varepsilon_i\mid X_i)=\sigma^2(X_i)$ alone, one cannot assert the stated variance conditional on all predictors. Other predictors could change the conditional variance of $\varepsilon_i$. The correction is to assume the conditional means, variances, and independence given the full predictor vector, as done here, or to give assumptions that imply those full-design conditions.
Pitfall: using unsquared weights in the variance. The variance of a weighted sum uses squared weights, which is why $T_p$ contains $K^2$, whereas $S_p$ contains $K$.
Pitfall: treating the approximation as an exact finite-sample identity. The exact conditional variance depends on the realized predictors through $H_n$ and $Q_n$. The simpler expression with $f(x)$, $\sigma^2(x)$, and moment integrals is a leading random-design approximation under the stated conditions.

Key takeaway: Under random design, the leading conditional variance of a local polynomial fitted value is $\sigma^2(x)/(nhf(x))$ times $e_0^TS_p^{-1}T_pS_p^{-1}e_0$. The squared-kernel matrix $T_p$, the design matrix $S_p$, and full-design conditional error assumptions are all essential to the derivation.

Check Your Understanding

Use the definitions and derivation above to answer these questions.

In the exact conditional variance, why does $Q_n$ use $K(U_i)^2$ rather than $K(U_i)$?
What assumptions ensure that the conditional variance of each error given all predictors equals $\sigma^2(X_i)$?
For a local constant fit, identify $S_0$ and $T_0$ and explain which one contains the squared kernel.
How does the leading variance change when $f(x)$ is halved while the other terms are unchanged?
For the uniform-kernel local quadratic example, what equation does $v=S_2^{-1}e_0$ solve, and why does it enter the variance constant?