From Distances to a Regression Prediction
In K-nearest-neighbor (KNN) regression, a prediction is built from the observed responses of training observations that are close to a query point. The calculation has two distinct parts: determine which observations are nearest using predictor values, then average their response values. Keeping those parts separate is essential. A large response is not automatically a close neighbor, and closeness is determined by the specified distance measure—not by the response.
This tutorial uses the ordinary, unweighted local-average rule. It is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. The predictors, distance measure, and value of \(k\) must be specified before calculating a prediction.
The neighbors are selected using the predictors and the chosen distance metric. The responses are used only after that selection. Unless stated otherwise, examples below use absolute distance for one predictor and Euclidean distance for two predictors. In a real analysis, any preprocessing or scaling used to define distances must be determined without using held-out observations.
The rule requires a quantitative response, a specified predictor distance, and an integer \(k\) no larger than the training sample size. It makes no particular distributional assumption about the responses. If two observations tie at the \(k\)th distance, the neighbor set may not be unique unless a tie rule is supplied. A calculation must state how that tie is handled; the examples below have no cutoff ties.
Why the Local Average Appears
Once the neighbors have been selected, KNN regression assigns one constant prediction to that local group. The arithmetic mean is the constant that minimizes the sum of squared response errors within the group. This gives a direct justification for the averaging step.
Differentiate with respect to the proposed constant \(a\):
Setting this derivative equal to zero gives \(a=(1/k)\sum_{i\in N_k(\mathbf{x}_0)}y_i\). Also, \(d^2Q(a)/da^2=2k>0\), so this value is the unique minimum. This derivation explains why the ordinary KNN regression rule averages responses; it does not say that the selected neighborhood itself is the best possible one.
Worked Examples
Worked Example: An Interior Query and Several Values of k
A training sample has one predictor \(x\) and response \(y\):
| Observation | \(x\) | \(y\) |
|---|---|---|
| A | 0 | 2 |
| B | 2 | 6 |
| C | 4 | 5 |
| D | 6 | 13 |
| E | 8 | 10 |
Predict the response at the interior query \(x_0=3.2\) using absolute distance. Calculate the predictions for \(k=1,2,3,4,\) and \(5\).
Method: rank by absolute predictor distance, then average the first \(k\) responses. The distances are:
In increasing order, the neighbors are C (distance \(0.8\), response \(5\)), B (\(1.2\), response \(6\)), D (\(2.8\), response \(13\)), A (\(3.2\), response \(2\)), and E (\(4.8\), response \(10\)). Each distance is distinct, so there is no ambiguity at any cutoff.
For \(k=1\), only C is used, so the prediction is \(5\). For \(k=2\), the prediction is \((5+6)/2=11/2=5.5\). For \(k=3\), it is \((5+6+13)/3=24/3=8\). For \(k=4\), it is \((5+6+13+2)/4=26/4=6.5\). For \(k=5\), it is \((5+6+13+2+10)/5=36/5=7.2\).
Check. At \(k=5\), every observation is included, and the response total is \(36\); \(36/5=7.2\), as reported. The predictions need not increase or decrease as \(k\) grows: adding a response of \(2\) at \(k=4\) lowers the average from \(8\) to \(6.5\). The mistake to avoid is averaging observations in predictor order or response order instead of selecting them by distance.
Worked Example: A Query Near the Boundary of the Observed Range
A hypothetical service study records response \(y\) at predictor values \(x\) as follows: \((1,12),(2,10),(4,14),(7,20),(10,18)\). Use absolute distance to predict at \(x_0=1.1\), which lies near the lower boundary of the observed predictor range. Calculate predictions for \(k=1,2,3,4,\) and \(5\).
Method: calculate each distance from \(1.1\), order the observations, and successively add the next response. The distances in the order the data are listed are \(0.1,0.9,2.9,5.9,\) and \(8.9\). Thus, the neighbor order and responses are \((1,12),(2,10),(4,14),(7,20),(10,18)\). No distances tie.
For \(k=1\), the prediction is \(12\). For \(k=2\), it is \((12+10)/2=22/2=11\). For \(k=3\), it is \((12+10+14)/3=36/3=12\). For \(k=4\), it is \((12+10+14+20)/4=56/4=14\). For \(k=5\), it is \((12+10+14+20+18)/5=74/5=14.8\).
Check. The five responses sum to \(74\), so the all-observation average is \(14.8\). Because the query is near the lower edge, the nearby observations all lie to its right in predictor space. Unlike an interior query, there are no observed predictors on the left to provide a local comparison. This one-sided neighborhood is a boundary feature, not a reason to reorder neighbors or invent unobserved data. The mistake to avoid is treating “near the boundary” as meaning “use only the single edge observation”; the stated \(k\) still determines how many nearest responses are averaged.
Worked Example: Euclidean Neighbors in Two Predictors
A hypothetical portfolio study has two predictors, \(x_1\) and \(x_2\), and a quantitative response \(y\). The training observations are \((0,0)\mapsto4\), \((1,0)\mapsto8\), \((0,2)\mapsto10\), \((3,1)\mapsto14\), and \((2,4)\mapsto20\). Predict at the interior query \(\mathbf{x}_0=(1.2,1.1)\) using Euclidean distance. Find the prediction for \(k=1,2,3,4,\) and \(5\).
Method: calculate Euclidean distances from the query in both predictors, rank them, and average the corresponding responses. For example, the distance to \((1,0)\) is:
The full distance calculations are: to \((0,0)\), \(\sqrt{1.44+1.21}=\sqrt{2.65}\approx1.628\); to \((0,2)\), \(\sqrt{1.44+0.81}=1.500\); to \((3,1)\), \(\sqrt{3.24+0.01}=\sqrt{3.25}\approx1.803\); and to \((2,4)\), \(\sqrt{0.64+8.41}=\sqrt{9.05}\approx3.008\).
The order is \((1,0)\mapsto8\), \((0,2)\mapsto10\), \((0,0)\mapsto4\), \((3,1)\mapsto14\), and \((2,4)\mapsto20\). For \(k=1\), the prediction is \(8\). For \(k=2\), it is \((8+10)/2=9\). For \(k=3\), it is \((8+10+4)/3=22/3\approx7.333\), rounded to three decimal places. For \(k=4\), it is \((8+10+4+14)/4=36/4=9\). For \(k=5\), it is \((8+10+4+14+20)/5=56/5=11.2\).
Check. The distance ranking uses both coordinate differences; for example, the squared distance to \((2,4)\) is \(0.64+8.41=9.05\), not just the squared difference in \(x_1\). The five responses sum to \(56\), confirming the \(k=5\) prediction of \(11.2\). The mistake to avoid is ranking by one predictor alone when the specified metric is Euclidean distance in two predictors.
Reading the Calculation Correctly
At an interior query, observations may occur on different sides of the query in predictor space. At a boundary query, the available nearby observations may be concentrated on one side. In either case the calculation is the same: use the specified distance to rank the training observations, then average exactly the responses belonging to the selected neighbor set. Boundary location changes which observations are available as neighbors; it does not change the prediction formula.
The prediction is a local average, not a fitted line through the selected points. For the unweighted rule, the prediction must lie between the smallest and largest responses among the selected neighbors. It also cannot be outside the range of all training responses, since every prediction is an average of some training responses. These checks can identify an arithmetic or neighbor-selection error, but they do not establish that the prediction is accurate.
A common ranking error is to sort the training observations by \(x\) and then take the first \(k\), regardless of the query. In the first example, if \(k=3\), an incorrect selection might take \(x=0,2,4\), yielding \((2+6+5)/3=13/3\approx4.333\). The query is \(3.2\), however, and the three closest predictors are \(4,2,6\), not \(0,2,4\). The correct prediction is \((5+6+13)/3=8\). Always rank distances from the query—not predictor values from the smallest upward.
Check Your Understanding
For each problem, identify the neighbor order before calculating the requested prediction. Assume the stated distance measure and use the ordinary, unweighted local average.
- A one-predictor training set is \((0,3),(2,7),(5,11),(9,15)\), where each pair is \((x,y)\). At \(x_0=4.1\), find the predictions for \(k=2\) and \(k=3\). (Answer: \(k=2: 9;\ k=3: 7\))
- A one-predictor data set is \((2,20),(4,16),(7,13),(11,9)\). For the boundary-near query \(x_0=2.5\), calculate the predictions for \(k=1\) and \(k=3\). (Answer: \(k=1: 20;\ k=3: 49/3\approx16.333\))
- Using Euclidean distance, the training points are \((3,1)\mapsto10\), \((1,1)\mapsto6\), \((0,0)\mapsto2\), and \((2,4)\mapsto14\). At query \((2.2,1.1)\), find the \(k=2\) and \(k=3\) predictions. (Answer: \(k=2: 8;\ k=3: 6\))
- A one-predictor training set contains \((0,2)\) and \((2,10)\). At query \(x_0=1\), find the \(k=1\) prediction if exactly one neighbor must be selected and no tie rule is specified. Then find the prediction if the rule is to include every observation tied at the \(k\)th distance and average all included responses. (Answer: The exact-\(k\) prediction is not uniquely determined; it can be \(2\) or \(10\). Including all tied observations gives \(6\).)
- A one-predictor data set is \((1,4),(3,8),(6,17),(10,9)\). At \(x_0=5.2\), calculate the predictions for \(k=2\) and \(k=3\), showing the neighbor order. (Answer: Neighbor order is \(x=6,3,1,10\); \(k=2: 12.5;\ k=3: 29/3\approx9.667\))