From Neighbors to a Class Prediction
A K-nearest-neighbor (KNN) classifier predicts a label for a new observation by examining the labels of nearby observations whose classes are known. Once the \(k\) neighbors have been selected, the classification step is a count: each neighbor contributes one vote for its class. This makes the rule easy to apply, but it is important to count all \(k\) labels rather than relying on the closest observation alone.
The distance measure and any feature scaling determine which observations are neighbors. They must be specified before making the prediction. The focus here is what to do with the selected neighbors’ class labels. The rule is unweighted: every selected neighbor contributes exactly one vote, regardless of its distance from the query point. This tutorial is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
The counts satisfy \(\sum_{g=1}^{G} n_g=k\), because each neighbor has exactly one class label and casts exactly one vote. If one class has a uniquely largest count, the rule gives a unique prediction. If two or more classes share the largest count, the vote alone does not specify a unique prediction. No additional tie rule is assumed here.
In everyday language, “majority vote” can suggest that a class must receive more than half of all votes. The classification rule is more precisely based on the largest vote count, also called the plurality winner. For example, with seven neighbors, a class receiving three votes can win if all other classes receive fewer than three, even though three is not more than half of seven. The distinction matters when there are three or more possible classes.
Why the Largest Count Is the Rule
The voting rule can be justified by looking at the number of classification errors within the selected neighborhood. Suppose we assign every neighbor’s location to one candidate class \(C_g\). Among the \(k\) neighbors, the \(n_g\) observations already labeled \(C_g\) would be classified correctly, while the other \(k-n_g\) would be classified incorrectly. Thus, the number of errors associated with assigning class \(C_g\) is \(k-n_g\).
Since \(k\) is the same for every candidate class, minimizing \(k-n_g\) is equivalent to maximizing \(n_g\). Therefore, the class with the most votes is the class that would make the fewest errors among the selected neighbors. This is a local, empirical justification: it describes the labels in the chosen neighborhood and does not guarantee that the prediction will be correct for a new observation.
For a tie, the same argument gives the same number of local errors for each tied class. For example, if two classes each receive two of five votes, assigning either class would produce three errors among those five labels. The counts do not distinguish between them, so the unweighted vote alone cannot select one uniquely.
Applying the Vote
A reliable calculation separates neighbor selection from label counting. First use the specified distance and value of \(k\) to identify the neighborhood. Then record every selected label, tally the counts by class, and compare the counts. The distances establish which observations enter the vote; in the unweighted rule, distances do not change the number of votes each selected neighbor receives.
Use the stated distance measure and select exactly \(k\) labeled observations. If the neighbors are supplied in rank order, do not include the next observation beyond the first \(k\).
Count how many of the selected observations belong to each possible class. Check that all the counts add to \(k\).
Name the class with a uniquely largest count, or state that the largest counts are tied and the vote gives no unique prediction.
Worked Example: A Binary Classification Vote
A hypothetical insurer uses a KNN classifier to label a new small commercial property as either “routine review” or “special review.” After applying its fixed scaling and distance measure, the five nearest labeled properties have the following distances and labels. The distances are given in increasing order; the sixth property is not selected.
| Rank | Distance | Observed label |
|---|---|---|
| 1 | 0.12 | Routine review |
| 2 | 0.20 | Special review |
| 3 | 0.28 | Routine review |
| 4 | 0.35 | Special review |
| 5 | 0.47 | Special review |
| 6 | 0.61 | Routine review |
Method. Use the unweighted majority-vote rule on the \(k=5\) selected observations. The fifth selected distance is 0.47, so the observation at distance 0.61 is excluded.
Count the labels. “Routine review” appears at ranks 1 and 3, giving two votes. “Special review” appears at ranks 2, 4, and 5, giving three votes. The check is \(2+3=5=k\).
Prediction and check. Special review has the uniquely largest count, three to two, so the predicted class is “special review.” The closest neighbor alone is labeled “routine review,” but that is not the result of the five-neighbor vote. Using only the closest label would discard four selected observations and would apply a different rule.
Worked Example: A Multiclass Vote Without a Strict Majority
A hypothetical conservation team classifies a newly measured tree as Birch, Maple, or Cedar. For \(k=7\), the selected neighbors, already ordered by increasing distance, have labels Birch, Maple, Cedar, Birch, Maple, Birch, Cedar.
Method. Count the seven labels by class. Birch occurs in positions 1, 4, and 6, so \(n_{\text{Birch}}=3\). Maple occurs in positions 2 and 5, so \(n_{\text{Maple}}=2\). Cedar occurs in positions 3 and 7, so \(n_{\text{Cedar}}=2\).
Prediction and check. Birch is the unique class with the largest count, so the predicted class is Birch. Three votes are not more than half of seven, but a strict majority is not required: the rule selects the uniquely largest count. The two other classes each have fewer votes, so there is no tie for the largest count.
Common mistake avoided. Requiring more than half of all votes would incorrectly leave this example unclassified. The relevant comparison is whether any other class has as many votes as Birch, not whether Birch has more than \(7/2\) votes.
Worked Example: A Tied Multiclass Vote
A hypothetical field survey assigns a newly observed plant to Birch, Maple, or Cedar. The five selected neighbors, ordered by distance, have labels Birch, Maple, Cedar, Birch, Maple.
Method. Count each class among the \(k=5\) neighbors. Birch appears twice, Maple appears twice, and Cedar appears once. The check is \(2+2+1=5\).
Prediction and check. Birch and Maple tie with two votes each, while Cedar has one vote. Birch and Maple therefore share the largest count. There is no unique prediction under majority vote alone; the data and rule given do not justify selecting Birch over Maple or Maple over Birch.
Common mistake avoided. It would be incorrect to name Birch as the prediction merely because Birch appears first in the list, or to present Birch as a unique winner after noting the tie. A tied largest count must be reported as a tie unless an additional rule is specified.
Interpretation and Common Traps
A KNN classification is conditional on the neighborhood used. Changing \(k\), the distance measure, or the feature scaling can change which observations are selected and therefore change the vote. These choices are part of defining the procedure; they are not changed during the count. For the calculation itself, assume that \(k\) is a positive integer no greater than the number of labeled training observations and that the set of \(k\) neighbors has been determined. If observations tie in distance at the selection boundary, the neighborhood must be specified before the vote can be computed.
- Counting only the nearest neighbor. KNN classification uses the labels of all \(k\) selected observations. The closest observation contributes one vote, not the entire prediction.
- Letting distance change an unweighted vote. In an unweighted classifier, a selected neighbor contributes one vote whether it is very close or farther away within the selected neighborhood. Do not award extra votes because one distance is smaller.
- Including \(k+1\) observations. If \(k=5\), only the five nearest observations vote. A sixth observation does not enter just because its label would strengthen a particular class.
- Confusing a plurality with a strict majority. A unique largest count determines the class even if that count is less than half of \(k\). A tie for the largest count is different: it leaves no unique prediction under the rule.
- Failing to check the total. The class counts must sum to \(k\). If they do not, at least one neighbor has been omitted, counted twice, or assigned incorrectly.
Check Your Understanding
For each problem, apply the unweighted vote to the stated neighborhood and check that the class counts sum to \(k\).
- A binary classifier uses \(k=5\). The selected labels are A, B, A, B, A. What class is predicted? (Answer: A, with 3 votes to B’s 2.)
- A three-class classifier uses \(k=6\). Its selected labels are X, Y, Z, Y, X, Y. What class is predicted, and is it a strict majority? (Answer: Y, with 3 votes; it is not a strict majority of 6, but is the unique largest count.)
- For \(k=4\), the neighbor labels are Cedar, Maple, Cedar, Maple. What does the vote predict? (Answer: No unique prediction; Cedar and Maple tie with 2 votes each.)
- A classifier uses \(k=7\), and the class counts are A: 1, B: 2, C: 4. State the predicted class and verify the total. (Answer: C; the counts sum to 7 and C has the unique largest count.)
- For \(k=5\), the ranked neighbor labels are P, Q, R, Q, P, R. Which labels enter the vote, and what is the result? (Answer: The first five labels enter; P and Q each have 2 votes and R has 1, so there is no unique prediction.)