Leave-One-Out Cross-Validation for Local Polynomial Fits

Why Leave One Observation Out?

A bandwidth controls how much nearby data influence a local polynomial fit. A narrow bandwidth can follow local features closely but may produce unstable predictions; a wide bandwidth is steadier but can miss variation. Plug-in methods choose a bandwidth using estimated bias and variance components. Leave-one-out cross-validation offers a different route: assess how well a fit predicts observations that were not used to construct that fit.

For each observed pair, temporarily remove that entire pair, fit the local polynomial using the remaining observations, and predict the response at the omitted covariate. The squared difference between that prediction and the omitted response is its prediction error. Averaging these errors gives a criterion that can be compared across bandwidths.

Definition of the Criterion

Suppose the observed data are $(X_i,Y_i)$, $i=1,\ldots,n$. Fix a polynomial degree $p$ and a bandwidth $h$. At a target $x$, a local polynomial fit minimizes a kernel-weighted sum of squared residuals over polynomial coefficients. Write its fitted value at $x$ as $\hat m_h(x)$. The notation $\hat m_{-i,h}(X_i)$ means the same prediction at $X_i$, but with observation $i$ excluded from the fit.

Definition: The leave-one-out cross-validation criterion for bandwidth $h$, with polynomial degree $p$ held fixed, is $$ \operatorname{CV}(h)=\frac{1}{n}\sum_{i=1}^{n}\left\{Y_i-\hat m_{-i,h}(X_i)\right\}^{2}. $$ The leave-one-out prediction for observation $i$ must be computed from the other $n-1$ pairs, not from a fit that used $Y_i$.

Bandwidth selection over a specified candidate set $\mathcal H$ is then $$ \hat h_{\mathrm{CV}}\in\arg\min_{h\in\mathcal H}\operatorname{CV}(h). $$ If there is more than one minimizer, a tie-breaking rule is needed to return a single value. This criterion compares bandwidths for the chosen degree, kernel, and candidate set; it does not by itself choose the polynomial degree.

Worked Example: Computing a Leave-One-Out Score

Take four observations with covariates $X=(0,1,2,3)$ and responses $Y=(0,1,2,3)$. Use a local constant fit and a uniform kernel with bandwidth $h=3.1$. Thus every observation lies in the kernel window for every target, and all included observations receive equal weight.

For the first observation, omit $Y_1=0$. The prediction is the mean of the other three responses, $(1+2+3)/3=2$, so the squared error is $(0-2)^2=4$.

For the second observation, the prediction is $(0+2+3)/3=5/3$. Its squared error is $(1-5/3)^2=4/9$.

For the third observation, the prediction is $(0+1+3)/3=4/3$. Its squared error is $(2-4/3)^2=4/9$.

For the fourth observation, the prediction is $(0+1+2)/3=1$, so the squared error is $(3-1)^2=4$. Therefore, $$ \operatorname{CV}(3.1)=\frac{1}{4}\left(4+\frac{4}{9}+\frac{4}{9}+4\right) =\frac{1}{4}\cdot\frac{80}{9} =\frac{20}{9}. $$

Each squared error compares the omitted response with a prediction based only on the other three responses. The full-data fitted values are not used in this calculation.

The Deleted-Residual Formula

For fixed covariates and bandwidth, a local polynomial estimate is linear in the responses whenever the weighted least-squares problem has a unique solution. Consequently, the fitted values at the observed covariates can be written as $\hat{\boldsymbol Y}=S_h\boldsymbol Y$, where $S_h$ is the smoother matrix and $\boldsymbol Y=(Y_1,\ldots,Y_n)^\mathsf{T}$. Its $i$th diagonal entry, $S_{h,ii}$, is the leverage for the fitted value at $X_i$. The deletion identity below lets us obtain leave-one-out residuals from a full-data fit.

Theorem: Fix $h$ and $i$. Suppose the local weighted least-squares design matrix at $X_i$ has full column rank both with and without observation $i$, and the weight of observation $i$ is positive. Then $S_{h,ii}\ne 1$ and $$ Y_i-\hat m_{-i,h}(X_i) =\frac{Y_i-\hat m_h(X_i)}{1-S_{h,ii}}. $$

To prove the identity, consider the local fit specifically at target $X_i$. Let $r_j$ be the polynomial design vector for observation $j$ in this fit, $w_j$ its nonnegative kernel weight, and let $R$ be the matrix with rows $r_j^\mathsf{T}$. The full-data coefficient vector is $$ \hat\beta=(R^\mathsf{T}WR)^{-1}R^\mathsf{T}W\boldsymbol Y, $$ where $W$ is the diagonal matrix of weights. Write $A=R^\mathsf{T}WR$, $r_i$ for the target observation’s design vector, and $w_i$ for its weight. The fit with observation $i$ deleted has coefficient vector $$ \hat\beta_{-i}=(A-w_i r_i r_i^\mathsf{T})^{-1} \left(R^\mathsf{T}W\boldsymbol Y-w_i r_iY_i\right). $$ The rank-one inverse identity gives $$ (A-w_i r_i r_i^\mathsf{T})^{-1} =A^{-1}+\frac{w_i A^{-1}r_i r_i^\mathsf{T}A^{-1}} {1-w_i r_i^\mathsf{T}A^{-1}r_i}. $$ The denominator is nonzero under the stated rank conditions. Define $q=r_i^\mathsf{T}A^{-1}r_i$ and $\hat y_i=r_i^\mathsf{T}\hat\beta=\hat m_h(X_i)$. Substituting the deleted right-hand side into the inverse identity and multiplying by $r_i^\mathsf{T}$ yields $$ \hat m_{-i,h}(X_i)=\frac{\hat y_i-w_iqY_i}{1-w_iq}. $$ The full-data smoother’s diagonal entry at $i$ is $S_{h,ii}=w_iq$: the coefficient of $Y_i$ in $r_i^\mathsf{T}\hat\beta$. Subtracting the deleted prediction from $Y_i$ therefore gives $$ Y_i-\hat m_{-i,h}(X_i) =Y_i-\frac{\hat y_i-S_{h,ii}Y_i}{1-S_{h,ii}} =\frac{Y_i-\hat y_i}{1-S_{h,ii}}. $$ The deletion fit exists by assumption, so the denominator is nonzero. This proves the identity. $\square$

The result provides a computational shortcut: when the full-data fit and its smoother matrix are available, each deleted residual can be calculated without fitting the model $n$ additional times. The identity does not remove the need to check that each deletion fit exists. If $S_{h,ii}$ is very close to one, division by $1-S_{h,ii}$ can also make the deleted residual large and numerically sensitive.

Worked Example: Checking the Deletion Formula for a Local Linear Fit

Use the three observations $(X,Y)=(0,0),(1,1),(2,4)$. Fit a local linear polynomial with a uniform kernel and bandwidth $h=3$, so all observations are included with equal weight at each target. At each observed target, this is the ordinary least-squares line fitted to all three points.

The covariate and response means are $\bar X=1$ and $\bar Y=5/3$. The slope is $$ \frac{\sum_i(X_i-\bar X)(Y_i-\bar Y)} {\sum_i(X_i-\bar X)^2} =\frac{5/3+0+7/3}{1+0+1} =\frac{4}{2}=2, $$ and the intercept is $5/3-2(1)=-1/3$. Thus the fitted values are $-1/3,5/3,11/3$, and the full-data residuals are $1/3,-2/3,1/3$.

For a straight-line regression with an intercept and equal weights, the diagonal entries of the hat matrix are $1/n+(X_i-\bar X)^2/\sum_j(X_j-\bar X)^2$. Here they are $5/6,1/3,5/6$. The deletion formula gives residuals $$ \frac{1/3}{1-5/6}=2,\qquad \frac{-2/3}{1-1/3}=-1,\qquad \frac{1/3}{1-5/6}=2. $$

Direct deletion confirms the calculation. Omitting the first point, the line through $(1,1)$ and $(2,4)$ predicts $-2$ at $X=0$, giving residual $0-(-2)=2$. Omitting the second, the line through $(0,0)$ and $(2,4)$ predicts $2$ at $X=1$, giving residual $1-2=-1$. Omitting the third, the line through $(0,0)$ and $(1,1)$ predicts $2$ at $X=2$, giving residual $4-2=2$. Hence the cross-validation score is $(2^2+(-1)^2+2^2)/3=3$, as obtained by either calculation.

What Prediction Risk Does the Criterion Measure?

Cross-validation is useful because its expectation has a direct prediction interpretation. Suppose the observations are independent and identically distributed, $E[Y\mid X=x]=m(x)$, and $E[Y^2]<\infty$. For a fixed bandwidth $h$, the fit used to predict $Y_i$ depends on the other observations but not on $Y_i$. Thus the omitted response serves as an independent test response for a fit trained on $n-1$ observations.

Theorem: Under the conditions just stated, and whenever the leave-one-out fits are defined, $$ E[\operatorname{CV}(h)] =E\!\left[\left\{Y^\ast-\hat m_{-n,h}(X^\ast)\right\}^2\right], $$ where $(X^\ast,Y^\ast)$ is an independent draw from the data distribution and $\hat m_{-n,h}$ is fitted using $n-1$ independent training observations.

For each index $i$, the collection of the other $n-1$ observations and the omitted pair $(X_i,Y_i)$ has the same joint distribution as an independent training sample of size $n-1$ and a new test pair $(X^\ast,Y^\ast)$. Therefore, $$ E\!\left[\{Y_i-\hat m_{-i,h}(X_i)\}^2\right] =E\!\left[\{Y^\ast-\hat m_{-n,h}(X^\ast)\}^2\right]. $$ Taking the average over $i=1,\ldots,n$ and using linearity of expectation proves the result. $\square$

This identity concerns expected prediction error for a procedure trained on $n-1$ observations, with expectation over both the training data and the test pair. It does not say that the observed criterion equals the risk, nor that its minimizing bandwidth must minimize the risk for the particular full sample. If the conditional variance is $\sigma^2(x)=\operatorname{Var}(Y\mid X=x)$, conditioning on the training data and test covariate separates test noise: $$ E\!\left[\{Y^\ast-\hat m_{-n,h}(X^\ast)\}^2\mid X^\ast,\text{training data}\right] =\sigma^2(X^\ast)+\{m(X^\ast)-\hat m_{-n,h}(X^\ast)\}^2. $$ The noise term does not depend on the prediction, but it remains part of the squared-error score.

Worked Example: Comparing Two Bandwidths

Return to $X=(0,1,2,3)$ and $Y=(0,1,2,3)$, using a local constant fit and a uniform kernel. Compare $h=1.1$ with $h=3.1$. At $h=1.1$, only observations at distance at most one are included; at $h=3.1$, all observations are included.

For $h=1.1$, omitting the first point leaves only the adjacent response $1$ in its window. The prediction is $1$, so the squared error is $(0-1)^2=1$. Omitting the second point leaves responses $0$ and $2$, whose mean is $1$; the squared error is $(1-1)^2=0$. Omitting the third point leaves responses $1$ and $3$, whose mean is $2$; the squared error is $(2-2)^2=0$. Omitting the last point leaves only the adjacent response $2$, so the squared error is $(3-2)^2=1$. Consequently, $$ \operatorname{CV}(1.1)=\frac{1+0+0+1}{4}=\frac{1}{2}. $$

For $h=3.1$, the score computed earlier is $\operatorname{CV}(3.1)=20/9$. Since $1/2<20/9$, this comparison selects $h=1.1$ from the two candidates. The choice is based on omitted-observation prediction errors, not on which bandwidth gives the smallest residuals when each observation is included in its own fit.

Interpreting and Using the Selected Bandwidth

For a fixed degree and kernel, a practical procedure is to specify a candidate set of positive bandwidths, calculate the leave-one-out score at each candidate, and choose a minimizer. The candidate set must be chosen so that the required local fits are estimable, including after deletion. In particular, a local polynomial fit may be undefined when too few suitably placed observations receive positive weight. Such a bandwidth cannot be scored unless a separate, explicitly defined fallback rule is used.

The score is an empirical estimate of prediction performance, not a theorem that the selected bandwidth is optimal in every sample. It can be noisy: each term uses the same overlapping data sets, so the $n$ squared errors are not independent. Also, the exact expectation identity applies to a fixed bandwidth. Once the same data are used to search many bandwidths, the selected score is not an unbiased estimate of the selected procedure’s risk. The criterion still supplies a principled comparison, but a very small difference between candidate scores should not be treated as conclusive evidence that one bandwidth is superior.

A common mistake is to omit only the response while retaining that observation in the local fit’s weighted least-squares calculation. That does not produce a held-out prediction: the observation still influences the estimated coefficients through its design row and weight. The correction is to remove the full pair $(X_i,Y_i)$ from the fit for the $i$th term, then evaluate the resulting polynomial at $X_i$. Another frequent error is to use the full-data residual $Y_i-\hat m_h(X_i)$ directly as the cross-validation residual. The deletion formula shows exactly why that is generally wrong: the correct residual divides the full-data residual by $1-S_{h,ii}$.

Key takeaway: Leave-one-out cross-validation chooses a bandwidth by averaging squared prediction errors for observations excluded from their respective fits. For a linear local polynomial smoother, each deleted residual equals the full-data residual divided by one minus its leverage, provided the required fits exist.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

In the definition of $\operatorname{CV}(h)$, which data must be excluded to compute the prediction for observation $i$, and where is that prediction evaluated?
For a linear smoother with full-data residual $Y_i-\hat m_h(X_i)=0.2$ and $S_{h,ii}=0.6$, calculate the leave-one-out residual using the deletion formula.
Why does the expected cross-validation criterion for fixed $h$ correspond to test risk for a fit trained on $n-1$ observations rather than $n$?
Using the four-point data in the bandwidth comparison, explain why the local constant score at $h=1.1$ is $1/2$.
State one reason the bandwidth minimizing the observed cross-validation score need not minimize prediction risk in a particular sample.