KNN Regression by Local Averaging

Why Regression Uses a Local Average

In regression, the response is quantitative, and the goal is to predict its value from the predictors. A simple K-nearest-neighbor (KNN) regression prediction uses the observed responses of training cases that are close to the query point. The distance rule decides which cases contribute; the unweighted regression rule then gives each selected response the same contribution.

This is a local method: a query point in one part of the predictor space can have a different set of neighbors, and therefore a different prediction, from a query point elsewhere. The resulting prediction is naturally interpreted as a local estimate of the mean response at the query’s predictor values. It is an estimate, not a guarantee that the conditional mean is constant throughout the neighborhood.

Definition: Let the training data be $(\mathbf{x}_1,y_1),\ldots,(\mathbf{x}_n,y_n)$, where $\mathbf{x}_i$ is the predictor vector for observation $i$ and $y_i$ is its numerical response. For a query point $\mathbf{x}$, let $N_k(\mathbf{x})$ denote the indices of its $k$ nearest training observations under the specified distance rule. The unweighted KNN regression prediction is

\hat{m}_k(\mathbf{x})=\frac{1}{k}\sum_{i\in N_k(\mathbf{x})}y_i, \qquad 1\leq k\leq n.

The selected responses are averaged equally. Distance determines membership in $N_k(\mathbf{x})$, but does not give a closer selected observation extra weight.

The distance measure and any predictor scaling must be specified as part of the modeling setup. Once the neighbor set has been determined, however, the unweighted prediction requires only the responses in that set. If observations are tied at the boundary of the $k$-neighbor set, a stated tie-handling rule is needed; the examples below have no boundary ties.

Why the Neighborhood Mean Appears

The arithmetic mean is not an arbitrary summary. Within a fixed neighborhood, it is the constant prediction that minimizes the sum of squared residuals for those observations. This gives the averaging rule a direct connection to squared-error prediction.

Derivation: Suppose the selected neighborhood contains $k$ responses $y_i$. If we predict one common value $a$ for all of them and use squared error, the total error is $Q(a)=\sum_{i\in N_k(\mathbf{x})}(y_i-a)^2$. Differentiating with respect to $a$ gives $Q'(a)=-2\sum_{i\in N_k(\mathbf{x})}(y_i-a)$. Setting this equal to zero yields $ka=\sum_{i\in N_k(\mathbf{x})}y_i$, so $a=\frac{1}{k}\sum_{i\in N_k(\mathbf{x})}y_i$. Since $Q''(a)=2k>0$, this value is the unique minimizer. Thus the unweighted neighborhood mean is the best constant prediction for the selected observations under squared error.

For regression, the target quantity at predictor value $\mathbf{x}$ is often the conditional mean $m(\mathbf{x})=E[Y\mid\mathbf{X}=\mathbf{x}]$, when that expectation exists. The KNN average estimates this target by borrowing responses from nearby observed predictor values. If the conditional mean changes little across the selected neighborhood and the observations represent the relevant population, their average can be informative about $m(\mathbf{x})$. If the conditional mean changes sharply across that region, the average can smooth together responses from meaningfully different predictor values.

Worked Example: Predicting a Local Claim Duration

A hypothetical claims team uses one numerical predictor, a file-complexity score, to predict the number of days needed to review a claim. The training data are $(1,8),(4,10),(6,12),(9,16),(11,21)$, where each pair gives score and review days. For a new file with score $8$, use $k=3$ and predict its review duration.

Method. Find the three smallest absolute score differences from 8, then take the arithmetic mean of their observed review durations.

The absolute distances from the query score are: score 1, $|1-8|=7$; score 4, $|4-8|=4$; score 6, $|6-8|=2$; score 9, $|9-8|=1$; and score 11, $|11-8|=3$. In increasing order, the three nearest scores are 9, 6, and 11. Their responses are 16, 12, and 21 days.

The prediction is

\hat{m}_3(8)=\frac{16+12+21}{3}=\frac{49}{3}=16.333\ldots\text{ days}.

The prediction is therefore $16.333\ldots$ days, or 16.33 days when rounded to two decimal places. The check is that exactly three responses were included and their sum is 49. A common mistake would be to include the response at score 4 because it is the next most familiar-looking score; its distance is 4, which is greater than the distance 3 for score 11, so it is not among the three nearest observations.

Interpreting the Prediction as a Conditional Mean Estimate

The notation $m(\mathbf{x})=E[Y\mid\mathbf{X}=\mathbf{x}]$ describes an average response for cases with predictor values $\mathbf{x}$, not a response that every such case must have. KNN regression approximates this quantity by using cases whose predictor values lie near $\mathbf{x}$. The prediction can be viewed as a local estimate because the averaging set is chosen using predictor proximity rather than by averaging every response in the training data.

This interpretation relies on more than the arithmetic. The response must have a finite conditional mean, and the training observations must be suitable representatives of the population or process being predicted. The distance measure must also make proximity meaningful for the predictors. If an irrelevant or poorly scaled predictor dominates distance, the chosen neighborhood may not consist of cases with similar conditional means. The earlier material on feature scaling and distance choices explains why that modeling setup matters.

The local estimate is not the same thing as the unknown conditional mean itself. Individual responses can vary around that mean, and a finite neighborhood contains only a sample of nearby cases. A small neighborhood can be sensitive to individual responses; a broad neighborhood can combine cases whose predictor values have different expected responses. This tutorial’s rule specifies the prediction for a chosen $k$; it does not establish that a particular $k$ is optimal.

Worked Example: Converting a Local Prediction to Dollars

A hypothetical risk team models claim payments in hundreds of dollars. For a query profile, the three nearest training observations have distances $0.20$, $0.50$, and $0.70$, with observed payments of 28, 31, and 35 hundreds of dollars. The next-nearest observation has distance $0.90$ and payment 60 hundreds of dollars. Use unweighted KNN regression with $k=3$, then calculate the residual if the query’s eventual observed payment is 33 hundreds of dollars. Use residual $=$ observed value minus predicted value.

Method. The three smallest distances identify the responses 28, 31, and 35. The observation at distance $0.90$ is not included. The prediction, in hundreds of dollars, is

\hat{m}_3(\mathbf{x})=\frac{28+31+35}{3}=\frac{94}{3} =31.333\ldots\text{ hundreds of dollars}.

Because one hundred dollars equals $\$100$, convert the unrounded prediction to dollars by multiplying by 100:

\frac{94}{3}\times \$100 =\$3{,}133.333\ldots \approx \$3{,}133.33.

The residual in hundreds of dollars is

33-\frac{94}{3} =\frac{99-94}{3} =\frac{5}{3}\text{ hundreds of dollars} =\$166.666\ldots \approx \$166.67.

The positive residual means the observed payment exceeds the prediction. The check is that the prediction uses only the three nearest observations and their response sum is 94. A common unit-conversion trap is to round $94/3$ to 31.33 hundreds first and then report that amount as $\$3{,}133.33$. In fact, 31.33 hundreds is $\$3{,}133.00$. The stated $\$3{,}133.33$ comes from converting the unrounded value $94/3$ hundreds to dollars and then rounding to cents.

What the Local Average Does—and Does Not—Say

A neighborhood average summarizes observed responses near the query, but it does not imply that the query has the average response, or that every nearby case has the same expected response. For example, if one selected observation has an unusually large response, it contributes just as much to the unweighted mean as any other selected observation. That contribution follows from the equal-weight rule; it is not reduced merely because the response looks unusual.

The local-versus-global distinction is also important. Averaging all training responses would generally produce one overall mean, regardless of the query point. Unweighted KNN regression instead uses the query to choose a local subset before averaging. When a query lies near cases with systematically different responses from the overall data, the local prediction can differ substantially from the global mean.

Worked Example: Comparing a Local Mean with the Overall Mean

A hypothetical equipment-monitoring study records a vibration score and a numerical repair-time response. For a query score, the four nearest observations have repair times of 8, 10, 14, and 20 hours. Two other, more distant observations have repair times of 40 and 50 hours. Use the four nearest cases for an unweighted KNN prediction, calculate the mean response across all six observations, and explain which quantity is local.

Local prediction. The selected response sum is $8+10+14+20=52$ hours. Therefore,

\hat{m}_4(\mathbf{x})=\frac{52}{4}=13\text{ hours}.

Overall mean. The sum across all six responses is $52+40+50=142$ hours, so the overall mean is

\bar{y}=\frac{142}{6}=\frac{71}{3} =23.666\ldots\text{ hours}.

The KNN prediction is 13 hours because it averages only the four responses selected by proximity to the query. The overall mean, approximately 23.67 hours, includes all six observations and does not depend on the query’s location. The check is that the local calculation divides 52 by 4, while the global calculation divides 142 by 6. Substituting the overall mean for the KNN prediction would discard the neighborhood information.

Common Mistakes and Checks

Averaging distances instead of responses. Distances rank observations and determine neighborhood membership. The prediction is the mean of the selected response values.
Giving closer selected cases extra influence. In unweighted KNN regression, all selected responses have weight $1/k$. A different weighting rule is not the unweighted rule.
Including the wrong number of observations. For a query with no boundary tie, sort the distances and include exactly the $k$ smallest. Do not include the next case just because its distance is close.
Calling the prediction the true conditional mean. It is a data-based estimate of $E[Y\mid\mathbf{X}=\mathbf{x}]$, and its quality depends on the neighborhood and the data.
Mixing units after rounding. Convert the unrounded prediction to the requested units before rounding the final amount. If rounding in an intermediate unit, convert that rounded amount consistently.
Using a training response for a new query by accident. A genuine new query is not one of the training observations. When assessing prediction on held-out data, the held-out response must not be used to form its neighborhood average.

Key takeaway: Unweighted KNN regression first selects the $k$ nearest training observations and then averages their responses. The mean minimizes squared error among constant predictions for that selected set, and serves as a local estimate of the conditional mean when nearby cases provide relevant information about the query.

Check Your Understanding

For each problem, identify the selected responses before calculating the prediction.

A one-predictor training set has predictor-response pairs $(0,4),(3,10),(6,16),(8,22),(11,30)$. For query $x=5$, use $k=3$. Which responses are selected, and what is the prediction? (Answer: responses 10, 16, and 22; prediction $16$.)
For a query, four nearest observations have distances $0.1,0.4,0.6,0.8$ and responses $5,9,13,19$. A fifth observation has distance $1.0$ and response 100. With $k=4$, calculate the prediction and state whether the response 100 is included. (Answer: prediction $11.5$; the response 100 is not included.)
The three nearest responses to a query are 7, 11, and 14. Calculate the unweighted prediction. If the eventual observed response at the query is 12, calculate the residual using observed minus predicted. (Answer: prediction $32/3=10.666\ldots$; residual $4/3=1.333\ldots$.)
A fixed neighborhood contains responses 2, 5, 8, and 11. Find the constant prediction that minimizes their sum of squared errors, and calculate that minimum sum. (Answer: prediction $6.5$; minimum sum of squared errors $45$.)
A hypothetical payment model uses thousands of dollars. The three nearest responses are 18, 24, and 33, and the observed response for the query is 28. Calculate the prediction and the residual in thousands of dollars, then convert both to dollars. (Answer: prediction $25$ thousand dollars or $\$25{,}000$; residual $3$ thousand dollars or $\$3{,}000$.)

What You'll Learn