Distance-Weighted KNN Classification

Why Weight Neighbors?

In K-nearest-neighbor (KNN) classification, the $k$ closest training observations vote on the class of a new observation. Equal voting gives every selected neighbor the same influence, whether it is extremely close to the query point or only just inside the neighborhood. Distance-weighted voting changes that rule: a closer observation can contribute more to the classification than a more distant one.

This distinction matters when the neighbors disagree. A small number of very close observations may provide a different local signal from the majority of neighbors counted equally. The weighted rule makes that distinction explicit, but it does not remove the need to choose a distance measure, a neighborhood size $k$, and a weighting rule.

Definition: Let $x$ be a query observation, and let $N_k(x)$ be the indices of its $k$ nearest labeled training observations under a specified distance measure. Write $d_i=d(x,x_i)$ for the distance from $x$ to training observation $x_i$, and let $y_i$ be its class label. Assign each selected neighbor a nonnegative weight $w_i$. For class $c$, its weighted vote is $$ V_c(x)=\sum_{i\in N_k(x)}w_i\,\mathbf{1}(y_i=c), $$ where $\mathbf{1}(y_i=c)$ equals 1 when $y_i=c$ and 0 otherwise. If the total selected weight is positive, the weighted local class-probability estimate is $$ \widehat{P}_w(Y=c\mid x)= \frac{V_c(x)}{\displaystyle\sum_{i\in N_k(x)}w_i}, \qquad \sum_{i\in N_k(x)}w_i>0. $$ The weighted classification predicts a class with the largest $V_c(x)$, equivalently the largest normalized estimate.

The notation $\widehat{P}_w(Y=c\mid x)$ describes a local estimate based on the training labels near $x$. It is not a guarantee that the estimated value is a calibrated probability for future observations. KNN classification is nonparametric: this rule does not specify a parametric response distribution. Its output depends on the observed neighbors, the chosen distance, $k$, and the weights.

Choosing and Applying Weights

A common distance-weighting rule is the inverse-distance power rule $$ w_i=\frac{1}{d_i^p},\qquad p>0. $$ When all selected distances are positive, this rule gives larger weights to closer observations. A larger value of $p$ makes the weights decline more quickly as distance increases. The value of $p$ is a modeling choice; it is not determined by the class labels alone.

The inverse-distance formula is not defined when a selected distance is zero. Therefore, do not apply it directly unless all selected distances are positive. Other rules can assign finite weights, but the rule for zero-distance observations must be stated before calculating. For the examples below, all distances are positive and the specified rule is $w_i=1/d_i$, except where stated otherwise.

Weights need not add to one in advance. The probability estimate divides each class's weighted vote by the total weight. Multiplying every weight by the same positive constant changes neither the estimates nor the predicted class. In contrast, the weights cannot all be zero: in that case the denominator is zero and the normalized estimates are undefined.

Find the neighborhood.
Use the specified distance measure to identify the $k$ nearest training observations to $x$.

Assign weights.
Apply the stated rule to each selected neighbor's distance, checking that the rule is defined for those distances.

Add weights by class.
For each class, sum the weights of selected observations with that label to obtain $V_c(x)$.

Normalize and decide.
Check that total weight is positive, divide each class vote by the total, and predict a class with the largest estimate.

Why the Estimates Sum to One

Suppose the possible classes form a finite set $\mathcal{C}$, and every selected training observation has exactly one of those labels. For each selected observation $i$, the indicators over all classes sum to one: $$ \sum_{c\in\mathcal{C}}\mathbf{1}(y_i=c)=1. $$ Consequently, adding the weighted votes over all classes counts each selected neighbor's weight exactly once: $$ \sum_{c\in\mathcal{C}}V_c(x) =\sum_{i\in N_k(x)}w_i\sum_{c\in\mathcal{C}}\mathbf{1}(y_i=c) =\sum_{i\in N_k(x)}w_i. $$ When this total is positive, normalizing gives $$ \sum_{c\in\mathcal{C}}\widehat{P}_w(Y=c\mid x) =\frac{\sum_{c\in\mathcal{C}}V_c(x)}{\sum_{i\in N_k(x)}w_i} =1. $$ This explains both why the denominator is the total selected weight and why a positive denominator is necessary.

Worked Calculations

Worked Example: Close Neighbors Reverse the Equal-Vote Result

Hypothetical scenario. A classifier estimates whether a newly reported claim belongs to class A or class B. Five selected neighbors have distances $1,2,3,4,6$, with labels A, A, B, B, B in that order. Use inverse-distance weights $w_i=1/d_i$. Find both local estimates and compare the weighted prediction with equal voting.

Method: calculate the weights, then sum within each class. The weights in neighbor order are $$ 1,\quad \frac12,\quad \frac13,\quad \frac14,\quad \frac16. $$ Class A receives the first two weights: $$ V_A=1+\frac12=\frac32. $$ Class B receives the last three: $$ V_B=\frac13+\frac14+\frac16 =\frac{4}{12}+\frac{3}{12}+\frac{2}{12} =\frac{9}{12}=\frac34. $$ The total weight is $$ W=\frac32+\frac34=\frac64+\frac34=\frac94>0. $$

The normalized estimates are $$ \widehat{P}_w(Y=A\mid x)=\frac{3/2}{9/4}=\frac23\approx0.6667, \qquad \widehat{P}_w(Y=B\mid x)=\frac{3/4}{9/4}=\frac13\approx0.3333. $$ Thus weighted voting predicts A. Equal voting counts two A labels and three B labels, so it predicts B. The estimates sum to $0.6667+0.3333=1.0000$ after rounding. The common mistake to avoid is using the unweighted label counts as the weighted votes: the A vote is $3/2$, not 2, and the B vote is $3/4$, not 3.

Worked Example: Inverse-Square Weights Change the Winner

Hypothetical scenario. A model classifies a policy review as A or B. Five neighbors have distances $0.5,1,2,2,4$ and labels A, B, A, B, B. Use $w_i=1/d_i^2$, and calculate the weighted estimates, the weighted prediction, and the equal-vote prediction.

Method: use the stated power for every distance. The weights are $$ \frac{1}{0.5^2}=4,\quad \frac{1}{1^2}=1,\quad \frac{1}{2^2}=\frac14,\quad \frac{1}{2^2}=\frac14,\quad \frac{1}{4^2}=\frac1{16}. $$ Class A occurs at distances $0.5$ and $2$, so $$ V_A=4+\frac14=\frac{64}{16}+\frac4{16}=\frac{68}{16}. $$ Class B occurs at distances $1,2,4$, so $$ V_B=1+\frac14+\frac1{16} =\frac{16}{16}+\frac4{16}+\frac1{16} =\frac{21}{16}. $$ The total is $W=(68+21)/16=89/16>0$. Therefore, $$ \widehat{P}_w(Y=A\mid x)=\frac{68}{89}\approx0.7640, \qquad \widehat{P}_w(Y=B\mid x)=\frac{21}{89}\approx0.2360. $$ The weighted prediction is A. Equal voting gives two A labels and three B labels, so equal voting predicts B. The estimates sum to $89/89=1$. The common mistake is to use inverse distance rather than inverse squared distance; the stated power must be applied to every distance before summing votes.

Worked Example: Comparing Local Estimates Across Three Classes

Hypothetical scenario. A classifier assigns one of three service categories—Bronze, Silver, or Gold—to a new account. Four neighbors have distances $1,2,4,8$ and labels Bronze, Bronze, Silver, Gold. Use inverse-distance weights. Find each local estimate and identify the predicted class.

Method: calculate one vote total for each class. The weights are $1,1/2,1/4,1/8$, so $$ V_{\mathrm{Bronze}}=1+\frac12=\frac32,\qquad V_{\mathrm{Silver}}=\frac14,\qquad V_{\mathrm{Gold}}=\frac18. $$ The total is $$ W=\frac32+\frac14+\frac18=\frac{12}{8}+\frac{2}{8}+\frac18=\frac{15}{8}>0. $$ The local estimates are $$ \widehat{P}_w(Y=\mathrm{Bronze}\mid x) =\frac{3/2}{15/8}=\frac{12}{15}=\frac45=0.8000, $$ $$ \widehat{P}_w(Y=\mathrm{Silver}\mid x) =\frac{1/4}{15/8}=\frac{2}{15}\approx0.1333, \qquad \widehat{P}_w(Y=\mathrm{Gold}\mid x) =\frac{1/8}{15/8}=\frac{1}{15}\approx0.0667. $$ The estimates sum to $12/15+2/15+1/15=1$, and the weighted prediction is Bronze. Equal voting gives Bronze $2/4$, Silver $1/4$, and Gold $1/4$, so it also predicts Bronze, but its class fractions differ from the weighted estimates. The common mistake is to normalize each class vote by $k=4$; weighted estimates use the total weight $15/8$, not the number of neighbors.

Interpreting the Result and Avoiding Traps

For equal voting, set every selected weight to one. Then $V_c(x)$ is the number of neighbors in class $c$, the total weight is $k$, and the estimate is the corresponding class fraction. Distance weighting generalizes that calculation by replacing counts with sums of weights. The predicted class still has the largest vote, but the vote size now reflects both labels and distances.

A decreasing weight rule gives closer neighbors at least as much influence as farther neighbors. It can change a classification when the close neighbors favor a class that has fewer observations in the selected neighborhood. As the first two examples show, equal-vote and weighted predictions can differ. A change is not inevitable: in the three-class example, the winner stayed the same even though the estimates changed.

Do not divide by the number of neighbors. Divide each weighted vote by the sum of the weights. Dividing by $k$ generally fails to produce estimates that sum to one.
Check the denominator. Nonnegative weights alone do not ensure a defined estimate. Require $\sum_{i\in N_k(x)}w_i>0$.
Use the specified weight rule exactly. Inverse distance and inverse squared distance can lead to different votes.
Do not overstate the probability interpretation. These are normalized local class estimates, not automatic guarantees of calibration.
Do not apply inverse distance at zero distance. The expression $1/d_i^p$ is undefined for $d_i=0$, so a separate rule is needed in that case.

Key takeaway: Distance-weighted KNN classification sums neighbor weights within each class and divides by the positive total weight to estimate local class probabilities. The class with the largest weighted vote is predicted; compared with equal voting, closer observations can change both the estimates and the winning class.

Check Your Understanding

For each problem, use only the stated neighbors and weighting rule. Round probability estimates to four decimals where needed.

A query has three neighbors at distances $1,2,4$, labeled X, Y, Y. Using $w_i=1/d_i$, calculate both weighted estimates and compare the weighted prediction with equal voting. (Answer: X $4/7\approx0.5714$, Y $3/7\approx0.4286$; weighted prediction X, equal-vote prediction Y.)
Four neighbors have distances $1,2,4,4$, labeled Renewal, Lapse, Lapse, Renewal. Using $w_i=1/d_i$, find both weighted estimates and state how the weighted result compares with equal voting. (Answer: Renewal $0.6250$, Lapse $0.3750$; weighted prediction Renewal, while equal voting is tied 2–2.)
Four neighbors have distances $0.5,1,2,4$, labeled A, B, B, C. Use $w_i=1/d_i$. Calculate all three estimates and identify the predicted class. (Answer: A $8/15\approx0.5333$, B $2/5=0.4000$, C $1/15\approx0.0667$; predict A.)
Four neighbors have distances $1,2,3,6$, labeled X, Y, Y, Y. With $w_i=1/d_i$, calculate the weighted estimates and compare the weighted outcome with equal voting. (Answer: X $0.5000$, Y $0.5000$, so weighted voting is tied; equal voting predicts Y.)
Four neighbors have distances $0.5,1,2,3$, labeled A, B, B, A. Use the supplied weights $4,2,1,0.5$, respectively. Find the estimates and compare with equal voting. (Answer: A $4.5/7.5=0.6000$, B $3/7.5=0.4000$; weighted prediction A, while equal voting is tied 2–2.)

What You'll Learn