Introduction
A K-nearest-neighbor classifier can return a class label, such as “renewal” or “no renewal,” but that label does not show how strongly the nearby observations favor the prediction. A query with 51% of its neighbors in one class and a query with 95% in that class receive the same hard label, even though their local class compositions differ considerably.
Near a classification boundary, moving the query can change which observations are among its \(k\) nearest neighbors. Tracking the resulting class fractions makes the local change visible. It also explains why a hard-label map can hide useful variation: the label changes only when the winning class changes, while the fractions can shift before or without that happening.
For two classes, call one class 1 and the other class 0. The estimated fraction for class 1 is \(\widehat{p}_1(x)\), and the class 0 fraction is \(1-\widehat{p}_1(x)\). With an odd \(k\) and two classes, majority vote predicts class 1 when \(\widehat{p}_1(x)>1/2\), and class 0 when \(\widehat{p}_1(x)<1/2\). With an even \(k\), or a tie in a multiclass vote, the classifier needs a tie-breaking rule.
These fractions are empirical proportions among the selected neighbors. They can be interpreted as local probability estimates, but they are not a guarantee that the true conditional probability equals the observed fraction. Their values depend on the training observations, the distance measure, the value of \(k\), and the query location.
How Neighbor Changes Move the Fractions
Hold the training data, distance measure, and \(k\) fixed while moving a query. As long as the set of \(k\) nearest observations stays the same, the unweighted class fractions stay the same too. A fraction changes when the neighborhood composition changes. In particular, if one neighbor leaves the set and a different observation enters, the change in the class-\(c\) fraction is:
The formula follows directly by writing the original class count as a sum of indicators. Removing the outgoing observation subtracts its indicator; adding the incoming observation adds its indicator. Dividing the new count by \(k\) gives the new fraction. Thus, replacing a non-\(c\) neighbor with a class-\(c\) neighbor raises \(\widehat{p}_c\) by \(1/k\); the reverse replacement lowers it by \(1/k\); replacing one class-\(c\) neighbor with another leaves that class’s fraction unchanged.
For a binary problem with odd \(k\), a one-neighbor replacement can change the majority when the class-1 count crosses from \((k-1)/2\) to \((k+1)/2\). For example, when \(k=5\), changing from two class-1 neighbors to three changes the estimate from \(2/5\) to \(3/5\), and changes the majority label. Other replacements can change the fractions without changing the winning class.
Worked Example: A Query Moves Across a Binary Classification Boundary
A hypothetical renewal study has five training observations on a one-dimensional predictor scale. Their locations and labels are \(0\) (Class A), \(2\) (Class A), \(4\) (Class B), \(6\) (Class B), and \(8\) (Class B). Use ordinary distance and unweighted \(k=3\). Compare queries at \(x=2.8\) and \(x=3.2\). Find the class fractions and hard predictions at both locations.
Method. Calculate each training observation’s distance from the query, select the three smallest distances, and count the labels. This avoids assuming that a query’s class fraction changes smoothly as it moves.
At \(x=2.8\), the distances from locations \(0,2,4,6,8\), in that order, are \(2.8,0.8,1.2,3.2,5.2\). The three nearest locations are \(2\) (A), \(4\) (B), and \(0\) (A). Thus, the class-A fraction is \(2/3\), and the class-B fraction is \(1/3\). The hard prediction is Class A.
At \(x=3.2\), the distances in the same order are \(3.2,1.2,0.8,2.8,4.8\). The three nearest locations are \(4\) (B), \(2\) (A), and \(6\) (B). The class-A fraction is now \(1/3\), and the class-B fraction is \(2/3\). The hard prediction is Class B.
The fractions changed by \(1/3\) in opposite directions as the query crossed the point at which the third neighbor changes from location 0 (A) to location 6 (B). The majority label changed as well. The exact midpoint \(x=3\) creates a tie in the distance of these competing observations for the third-neighbor position, so a specified tie rule would be needed to define the neighborhood there.
Check. Each query has exactly three neighbors, and the two class fractions at each query sum to 1. A common mistake is to use the neighbor set from \(x=2.8\) again at \(x=3.2\); recalculating the distances shows that location 6 enters and location 0 leaves.
Fraction Changes Can Precede or Avoid a Label Change
A hard classifier reports the class with the largest fraction, not the full vector of fractions. In a binary problem, this compresses the estimate to one of two labels, separated by the threshold \(1/2\). A query can therefore move within the same predicted region while its class fractions change. Conversely, a relatively small change in fractions can change the label when it takes the estimate across the threshold.
With several classes, the same principle applies: the hard prediction is the class with the largest fraction, while the vector of fractions records the full local composition. Two queries can have the same winning class but different fractions for every class. A fraction of zero means that no neighbor in the selected set has that label; it does not establish that the class is impossible at that query.
Worked Example: A One-Neighbor Swap Changes the Vote
A hypothetical classifier uses five neighbors and a one-dimensional predictor. The training observations are at locations \(0\) (A), \(2\) (A), \(4\) (A), \(6\) (B), \(8\) (B), \(10\) (B), and \(12\) (A). Compare queries at \(x=4.8\) and \(x=5.2\). Calculate the class-B fraction and predicted label at each query.
Method. Rank the distances at each query and identify all five neighbors. In particular, check the fifth neighbor rather than assuming that the same five observations remain nearest on both sides.
At \(x=4.8\), the distances from locations \(0,2,4,6,8,10,12\) are \(4.8,2.8,0.8,1.2,3.2,5.2,7.2\). The five nearest observations are \(4\) (A), \(6\) (B), \(2\) (A), \(8\) (B), and \(0\) (A). There are two B labels out of five, so \(\widehat{p}_B(4.8)=2/5=0.4\). The predicted label is A.
At \(x=5.2\), those distances are \(5.2,3.2,1.2,0.8,2.8,4.8,6.8\). The five nearest observations are \(6\) (B), \(4\) (A), \(8\) (B), \(2\) (A), and \(10\) (B). There are three B labels out of five, so \(\widehat{p}_B(5.2)=3/5=0.6\). The predicted label is B.
The fifth neighbor changes from location 0 (A) to location 10 (B), increasing the B fraction by \(1/5\). This is enough to move the estimate from below \(1/2\) to above \(1/2\), so the hard label changes.
Check. At \(x=5\), locations 0 and 10 are tied in distance for the fifth-neighbor position. Since their labels differ, including one rather than the other can give different class fractions. A common mistake is to report a unique estimate at that exact query without stating how the tie is resolved.
Reading Multiclass Fractions
For \(C\) possible classes, the local estimate is the vector \((\widehat{p}_1(x),\ldots,\widehat{p}_C(x))\), with each entry between 0 and 1 and the entries summing to 1. The hard prediction is an argmax: it selects a class with the largest entry, subject to a tie rule if multiple entries are largest. Retaining the vector gives more information than retaining only that selected class.
Worked Example: The Winning Class Stays the Same
A hypothetical three-class classifier uses five neighbors. Training observations lie on a line at locations \(0\) (C), \(2\) (A), \(4\) (B), \(6\) (B), \(8\) (B), \(10\) (A), \(12\) (C), and \(14\) (B). In the order A, B, C, calculate the class-fraction vector and hard prediction for queries at \(x=4.8\) and \(x=5.2\).
Method. At each location, identify the five nearest observations and count each of the three labels. This comparison shows how the full vector can change even when its largest entry does not.
At \(x=4.8\), the distances to locations \(0,2,4,6,8,10,12,14\) are \(4.8,2.8,0.8,1.2,3.2,5.2,7.2,9.2\). The five nearest observations are \(4\) (B), \(6\) (B), \(2\) (A), \(8\) (B), and \(0\) (C). The counts in the requested order are A: 1, B: 3, C: 1. The fraction vector is \((1/5,3/5,1/5)=(0.2,0.6,0.2)\), and the hard prediction is B.
At \(x=5.2\), the distances in that order are \(5.2,3.2,1.2,0.8,2.8,4.8,6.8,8.8\). The five nearest observations are \(6\) (B), \(4\) (B), \(8\) (B), \(2\) (A), and \(10\) (A). The counts are A: 2, B: 3, C: 0. The fraction vector is \((2/5,3/5,0)=(0.4,0.6,0)\), and the hard prediction remains B.
The predicted class is unchanged, but the estimated A fraction rises from 0.2 to 0.4 and the C fraction falls from 0.2 to 0. The hard label conceals both changes. In particular, the zero C fraction describes this five-neighbor set; it is not proof that class C cannot occur near the query.
Check. Each vector sums to 1, and B remains the unique largest fraction at both query locations. A common mistake is to treat an unchanged hard prediction as evidence that the local estimates have not changed.
Common Mistakes and Interpretation
- Reading a label as a probability. A hard prediction such as Class A does not reveal whether its fraction is 0.51 or 0.95. Report the fractions when local composition matters.
- Assuming fractions vary continuously. With a fixed training set, fixed \(k\), and unweighted voting, the fractions remain constant while the neighbor set stays fixed. They can jump when membership changes.
- Forgetting the denominator. Each class count is divided by \(k\), not by the number of neighbors belonging to a particular class. The class fractions must sum to 1.
- Ignoring ties at the cutoff. If observations tie in distance for the last available neighbor place, the selected set, and therefore the fractions, may depend on the tie rule.
- Overinterpreting a local fraction. A fraction is an estimate based on the selected training observations, not a guarantee about the true probability or about observations outside the neighborhood.
Check Your Understanding
For each problem, use the stated distance rule and count the labels in the nearest-neighbor set. Answers are included so you can check both the fractions and the hard predictions.
- A one-dimensional training set has locations \(0\) (A), \(2\) (A), \(4\) (B), \(6\) (B), and \(8\) (A). With unweighted \(k=3\), find the class-B fraction and prediction at \(x=2.6\). (Answer: The neighbors are \(2\) (A), \(4\) (B), and \(0\) (A); the B fraction is \(1/3\), and the prediction is A.)
- Using the same training set and \(k=3\), find the class-B fraction and prediction at \(x=3.4\). Compare the fraction with your result at \(x=2.6\). (Answer: The neighbors are \(4\) (B), \(2\) (A), and \(6\) (B); the B fraction is \(2/3\), and the prediction is B. The B fraction increased by \(1/3\).)
- A three-class training set has locations \(0\) (A), \(2\) (B), \(4\) (C), \(6\) (B), \(8\) (A), and \(10\) (C). With \(k=3\), find the class-fraction vector in the order A, B, C at \(x=3.2\). (Answer: The neighbors are \(4\) (C), \(2\) (B), and \(6\) (B); the vector is \((0,2/3,1/3)\), and the prediction is B.)
- For a binary classifier with \(k=7\), a query has four class-1 neighbors and three class-0 neighbors. One class-1 neighbor is replaced by a class-0 neighbor as the query moves. Find the new class-1 fraction and state whether the majority label changes. (Answer: The new fraction is \(3/7\); the majority changes from class 1 to class 0.)
- A five-neighbor binary classifier has three class-A neighbors at one query and four class-A neighbors at a nearby query. State the class-A fractions and explain whether the hard label can differ. (Answer: The fractions are \(3/5=0.6\) and \(4/5=0.8\); both predict class A, so the hard label is unchanged despite a fraction increase of \(1/5\).)