The K-Nearest-Neighbor Prediction Rule

From a Query to a Prediction

A prediction method must specify which information from the training data it uses for a new case. K-nearest neighbors (KNN) uses the training observations whose predictor values are closest to those of the case being predicted. The method is local: it bases a prediction on a selected neighborhood rather than fitting one global equation for every possible predictor value.

For a training set of $n$ observations, write the predictors of observation $i$ as $\mathbf{x}_i$ and its response as $y_i$, for $i=1,\ldots,n$. A query point $\mathbf{x}_0$ is a new predictor value for which a response prediction is required. A distance or dissimilarity rule measures how near each training predictor vector is to the query. The exact choice of distance is a separate modeling decision; here, assume it has already been specified and can be used to rank the training observations.

Definition: For a chosen positive integer $k\le n$, the K-nearest-neighbor set, denoted $N_k(\mathbf{x}_0)$, consists of the $k$ training observations with the smallest distances from their predictor vectors $\mathbf{x}_i$ to the query $\mathbf{x}_0$. The KNN prediction uses the responses $y_i$ for $i\in N_k(\mathbf{x}_0)$. Unless a problem specifies otherwise, assume there is no tie at the boundary between the $k$th and $(k+1)$th distances.

The same neighborhood idea serves two different prediction tasks. For a numerical response, the basic rule averages the responses in the neighborhood. For a categorical response, it selects the class that occurs most often among those neighbors. These are unweighted rules: each selected neighbor contributes equally, while observations outside the selected neighborhood do not contribute to the prediction.

The Basic Prediction Rules

Let $N_k(\mathbf{x}_0)$ be the selected neighborhood. If $y_i$ is numerical, the basic KNN regression prediction is the arithmetic mean of the $k$ responses:

\widehat{y}(\mathbf{x}_0)=\frac{1}{k}\sum_{i\in N_k(\mathbf{x}_0)} y_i.

If $y_i$ is categorical and $\mathcal{C}$ is the set of possible classes, the basic KNN classification prediction is the class with the greatest count in the neighborhood:

\widehat{c}(\mathbf{x}_0) = \underset{c\in\mathcal{C}}{\operatorname{arg\,max}} \sum_{i\in N_k(\mathbf{x}_0)} I(y_i=c),

where $I(y_i=c)=1$ when observation $i$ belongs to class $c$, and $I(y_i=c)=0$ otherwise. If classes tie for the greatest count, this expression does not select a unique class; a tie rule must be specified. The examples below avoid such a tie.

Worked Example: Averaging the Three Closest Claim Amounts

Method. Use the basic KNN regression rule with $k=3$. A hypothetical insurer has six training observations for a claim-severity prediction. The predictor vector describes a claim’s recorded features, and the response is claim amount in thousands of dollars. The distance calculations have already been made using a chosen distance rule.

Observation	Distance to query	Claim amount (thousands of dollars)
A	0.5	10
B	1.0	14
C	1.5	18
D	2.0	22
E	3.0	26
F	9.0	70

Step 1: Select the neighborhood. The three smallest distances are $0.5$, $1.0$, and $1.5$, so $N_3(\mathbf{x}_0)=\{A,B,C\}$. Observations D, E, and F are not included.

Step 2: Average the selected responses. The total of the three selected amounts is $10+14+18=42$ thousand dollars. Divide by $k=3$:

\widehat{y}(\mathbf{x}_0)=\frac{42}{3}=14 \quad\text{thousand dollars}.

The prediction is therefore $14,000. Check and common mistake: the selected responses have mean 14, as required. Averaging all six responses would give $(10+14+18+22+26+70)/6=160/6=26.6667$ thousand dollars, which is not the $k=3$ prediction. That calculation uses the entire training set rather than the three nearest observations.

Worked Example: Classifying a New Service Request

Method. Use the basic KNN classification rule with $k=5$. A hypothetical service team labels each past request either “route to specialist” or “handle routinely.” A new request has four recorded predictors. A specified distance rule gives the following five nearest training observations:

Neighbor, in distance order	Distance	Observed class
1	0.4	Route to specialist
2	0.7	Handle routinely
3	1.1	Handle routinely
4	1.6	Route to specialist
5	2.0	Handle routinely

Step 1: Count each class. “Route to specialist” occurs in neighbors 1 and 4, so its count is 2. “Handle routinely” occurs in neighbors 2, 3, and 5, so its count is 3.

Step 2: Choose the class with the larger count. Since $3>2$, the KNN classification decision is “handle routinely.”

Check and common mistake: all five listed neighbors contribute one vote. The closest observation alone has the specialist label, but $k=5$ calls for the majority among five neighbors, not the class of the single closest observation. The votes are $3$ to $2$, so there is no tie.

Why Averaging and Majority Voting Fit Their Losses

The regression mean and classification majority rule are not arbitrary summaries. Each minimizes a natural loss over the selected neighborhood. For regression, suppose a single constant $a$ is used as the prediction for all $k$ selected responses, and assess it by the sum of squared errors. Define

Q(a)=\sum_{i\in N_k(\mathbf{x}_0)}(y_i-a)^2.

Differentiate with respect to $a$ and set the derivative to zero:

\frac{dQ(a)}{da} =-2\sum_{i\in N_k(\mathbf{x}_0)}(y_i-a)=0.

Rearranging gives $ka=\sum_{i\in N_k(\mathbf{x}_0)}y_i$, and hence $a$ equals the neighborhood mean. Since the second derivative is $d^2Q(a)/da^2=2k>0$, this value minimizes the sum of squared errors. Thus the KNN regression mean is the constant prediction that best fits the selected responses under squared-error loss.

For classification, suppose a single class $c$ is assigned to the query and errors are counted with zero-one loss: an observation receives loss 0 if its class is $c$, and loss 1 otherwise. If $n_c$ of the $k$ neighbors belong to $c$, the total loss from assigning $c$ is $k-n_c$. Minimizing $k-n_c$ is equivalent to maximizing $n_c$. Therefore the class with the most votes minimizes the number of classification errors within the neighborhood. This argument explains the rule; it does not guarantee that the local neighborhood will represent future cases perfectly.

How to Apply the Rule Reliably

A KNN calculation has two distinct stages: determine who is near the query, then combine the corresponding responses. Keeping them separate makes the method easier to apply and helps prevent errors.

State the prediction task.
Identify the query, the response, and whether that response is numerical (regression) or categorical (classification).

Use the specified distance rule.
Compare the query’s predictor vector with each training predictor vector. Do not rank observations by their responses.

Select exactly $k$ neighbors.
Order the training observations by distance and include the $k$ closest, subject to any stated boundary-tie rule.

Combine their responses appropriately.
For regression, calculate their arithmetic mean. For classification, count their class labels and choose the most frequent class.

The value of $k$ is part of the prediction rule. With $k=1$, the prediction uses one observed response or class. With $k=n$, every training observation is included. For intermediate values, changing $k$ can change the neighborhood and therefore the prediction. A small $k$ is especially dependent on a few observations; a large $k$ incorporates observations that may be farther from the query. The appropriate value depends on the prediction problem and is generally assessed using data reserved for model development, rather than chosen by inspecting final test results.

Common Misreadings and Scope

Using the wrong observations. The neighborhood is determined by predictor-space distances from the query. It is not the $k$ smallest or largest response values.
Using the wrong number of neighbors. If $k=5$, include five observations, not the closest observation plus five more.
Mixing up the two prediction tasks. Average numerical responses for the basic regression rule; count categorical labels for the basic classification rule.
Assuming distance alone determines the prediction. In the basic rule, distance determines which observations enter the neighborhood. Once selected, each neighbor contributes equally; this rule does not give closer selected observations extra weight.
Ignoring an unresolved tie. A tie at the neighborhood boundary can make membership ambiguous, and a tie in class counts can leave the predicted class ambiguous. The problem must provide or assume a convention to resolve the relevant tie.

KNN is a prediction procedure, not a fitted global formula that assigns a single coefficient to each predictor. Its result depends on the training observations available, the distance rule, the chosen $k$, and the responses within the resulting neighborhood. In applications, those choices must be made consistently with the prediction goal and evaluated without using held-out test outcomes to tune the method. These distinctions are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Key takeaway: First identify the $k$ closest training observations to the query. Then average their responses for regression or select their most frequent class for classification. The responses outside that neighborhood do not enter the basic KNN prediction.

Check Your Understanding

For each question, identify the neighbors before combining their responses. The answers are shown at the end of each item.

A numerical-response training set has distances $0.3, 0.8, 1.4, 2.2$ and corresponding responses $6, 12, 18, 30$. For $k=2$, what is the basic KNN regression prediction? (Answer: 9)
For $k=3$, the three nearest observations have classes A, B, and A. What is the basic classification decision, and how many votes does it receive? (Answer: Class A, with 2 votes)
A query has five nearest numerical responses $4, 7, 10, 13, 16$, in order of increasing distance. With $k=3$, calculate the prediction. (Answer: 7)
For $k=5$, the nearest class labels are red, blue, blue, red, blue. State the predicted class and verify that the vote is not tied. (Answer: Blue, 3 votes to red’s 2)
A hypothetical risk team predicts a numerical review duration in minutes. The four nearest training observations have distances $0.2, 0.5, 0.9, 1.3$ and durations $8, 14, 17, 25$. Using $k=3$, find the prediction and explain why the fourth duration is excluded. (Answer: 13 minutes; $8+14+17=39$, and $39/3=13$; the fourth observation is not among the three nearest.)

What You'll Learn