Feature Weights in KNN Distance

Why Weight Features in KNN?

K-nearest-neighbor methods select observations by comparing their predictor values with those of a query. If two predictors are not equally useful for measuring similarity, an unweighted distance may give them equal influence anyway. Feature weights let the analyst emphasize a predictor by increasing its contribution to distance, or reduce its contribution by decreasing its weight.

A feature weight changes the geometry used to find neighbors. It is distinct from a distance-based weight assigned to an already selected neighbor when forming a prediction: feature weights can change which observations enter the neighborhood in the first place. The calculations below focus on numeric predictors and weighted Euclidean distance, while also stating the corresponding Minkowski form.

Definition: For observations $x=(x_1,\ldots,x_p)$ and $z=(z_1,\ldots,z_p)$, a weighted Minkowski distance of order $r\geq 1$ is $$ d_w(x,z)=\left(\sum_{j=1}^{p}w_j|x_j-z_j|^r\right)^{1/r}, $$ where each feature weight $w_j\geq 0$. The weighted Euclidean distance is the special case $r=2$: $$ d_w(x,z)=\left(\sum_{j=1}^{p}w_j(x_j-z_j)^2\right)^{1/2}. $$

The weight $w_j$ multiplies the $r$th power of the difference for feature $j$; it is not itself a distance and is not multiplied directly by the absolute difference in the displayed formula. A larger weight makes a fixed difference on that feature contribute more. A zero weight removes that feature from the distance calculation.

If every weight is strictly positive, the weighted Minkowski formula is a metric on numeric feature vectors: distances are nonnegative, symmetric, zero only when the vectors agree, and satisfy the triangle inequality. If some weights are zero, two different vectors that differ only on zero-weight features can have distance zero. The result is then a pseudometric on the full feature space, though it is a metric after ignoring those features.

How the Weights Change Distance

For weighted Euclidean distance, a feature’s contribution to the squared distance is $w_j(x_j-z_j)^2$. Thus multiplying one weight by four multiplies that feature’s squared contribution by four; it multiplies its contribution to the distance scale by a factor of two when that feature is considered alone. The resulting neighbor ranking depends on the total contributions from all features, not on any one contribution in isolation.

The formula also has a useful interpretation. Define a transformed coordinate $x_j^*=\sqrt{w_j}x_j$ for each feature with a nonnegative weight. Then $$ d_w(x,z)=\left(\sum_{j=1}^{p}\left(\sqrt{w_j}x_j-\sqrt{w_j}z_j\right)^2\right)^{1/2}. $$ So weighted Euclidean distance is ordinary Euclidean distance after scaling each coordinate by the square root of its weight. This explains why larger weights stretch a coordinate and make differences on it matter more.

Worked Example: Changing the Nearest Observation

A hypothetical insurer uses two numeric predictors in a KNN regression model. For a query with predictor vector $(0,0)$, two historical observations have vectors $A=(1,2)$ and $B=(2,0)$, with responses 8 and 20, respectively. Find the $k=1$ prediction first with weights $(1,1)$, then with weights $(4,1)$. Use weighted Euclidean distance.

Method. Calculate each weighted distance from the query, rank the candidates, and use the response of the closest observation. This avoids assuming that a larger value on a feature automatically means a larger contribution: both the feature difference and its weight enter the calculation.

With weights $(1,1)$, the squared distance to $A$ is $1(1-0)^2+1(2-0)^2=1+4=5$, so $d_w((0,0),A)=\sqrt{5}\approx2.236$.

The squared distance to $B$ is $1(2-0)^2+1(0-0)^2=4+0=4$, so $d_w((0,0),B)=\sqrt{4}=2$. Since $2<\sqrt{5}$, $B$ is the nearest observation. The $k=1$ regression prediction is therefore 20.

Now use weights $(4,1)$. The squared distance to $A$ is $4(1-0)^2+1(2-0)^2=4+4=8$, so the distance is $\sqrt{8}\approx2.828$.

The squared distance to $B$ is $4(2-0)^2+1(0-0)^2=16+0=16$, so the distance is 4. Now $A$ is closer because $\sqrt{8}<4$, and the $k=1$ prediction changes to 8.

Check. Under equal weights, $B$ is closer; after increasing the first feature’s weight, $A$ is closer. The corresponding predictions are 20 and 8. The ranking change is consistent with the fact that $A$ differs less than $B$ on the first feature.

Why a Common Rescaling Does Not Change the Neighbors

The relative sizes of the weights determine the neighbor ranking. Multiplying all weights by the same positive constant changes the numerical distances but not their order. This matters because weights may be normalized, for example to sum to one, without changing which observations are selected.

Result: If $c>0$ and every feature weight $w_j$ is replaced by $cw_j$, then every weighted Minkowski distance is multiplied by $c^{1/r}$. Therefore, the neighbor ranking is unchanged, apart from any ties that were already present.

To verify the result, let $d_w(x,z)$ denote the distance before rescaling. After replacing $w_j$ with $cw_j$, factor $c$ out of the sum and then take the $r$th root:

\begin{aligned} d_{cw}(x,z) &=\left(\sum_{j=1}^{p}cw_j|x_j-z_j|^r\right)^{1/r}\\ &=\left(c\sum_{j=1}^{p}w_j|x_j-z_j|^r\right)^{1/r}\\ &=c^{1/r}d_w(x,z). \end{aligned}

The multiplier $c^{1/r}$ is the same positive number for every candidate observation. If one candidate had a smaller distance than another before rescaling, it remains smaller afterward. Equal distances also remain equal.

Worked Example: Two Features, Two Weight Choices

A hypothetical claims team considers three historical cases with predictor vectors $A=(1,2)$, $B=(2,0)$, and $C=(0,3)$. The query is $(0,0)$, and their recorded responses are 8, 20, and 50, respectively. Find the $k=2$ local-average prediction under weights $(4,1)$ and under weights $(1,4)$.

Method. Compute the weighted squared distances, rank the three observations, and average the two responses belonging to the nearest cases. Since taking a square root preserves order, squared distances are sufficient for ranking, but the distances will also be reported for verification.

Under weights $(4,1)$, the squared distances are:

For $A$: $4(1^2)+1(2^2)=4+4=8$, giving distance $\sqrt{8}\approx2.828$.
For $B$: $4(2^2)+1(0^2)=16+0=16$, giving distance $4$.
For $C$: $4(0^2)+1(3^2)=0+9=9$, giving distance $3$.

The two smallest squared distances are 8 for $A$ and 9 for $C$, so the nearest cases are $A$ and $C$. Their local-average prediction is $(8+50)/2=58/2=29$.

Under weights $(1,4)$, the squared distances are:

For $A$: $1(1^2)+4(2^2)=1+16=17$, giving distance $\sqrt{17}\approx4.123$.
For $B$: $1(2^2)+4(0^2)=4+0=4$, giving distance $2$.
For $C$: $1(0^2)+4(3^2)=0+36=36$, giving distance $6$.

The two smallest squared distances are 4 for $B$ and 17 for $A$, so the nearest cases are $B$ and $A$. Their local-average prediction is $(20+8)/2=28/2=14$.

Check. Increasing the first feature’s weight favors cases with a small first-coordinate difference, resulting in neighbors $A$ and $C$. Increasing the second feature’s weight favors cases with a small second-coordinate difference, resulting in neighbors $B$ and $A$. The predictions, 29 and 14, are the averages of the correct response pairs.

Feature Weights and Classification

The same geometry change applies to KNN classification. Once the weights are set, calculate distances and select the $k$ nearest observations; then apply the specified classification rule, such as majority vote for an unweighted vote. Changing feature weights can change the selected cases and therefore change the predicted class. It does not, by itself, alter the class labels of the training observations.

Worked Example: A Weight Change Reverses a Classification

A hypothetical model classifies a new policy as either “retain” or “exit” using two numeric predictors. The query is $(0,0)$. Historical case $P=(2,0)$ has class “retain,” and case $Q=(0,1)$ has class “exit.” Use $k=1$ and weighted Euclidean distance. Compare equal weights $(1,1)$ with weights $(0.1,1)$.

Method. For $k=1$, the predicted class is the label of the nearest case. The first weight change downweights differences in the first predictor, so the distances must be recalculated rather than assuming the original nearest case remains selected.

With weights $(1,1)$, the squared distance to $P$ is $1(2^2)+1(0^2)=4$, so its distance is 2. The squared distance to $Q$ is $1(0^2)+1(1^2)=1$, so its distance is 1. Case $Q$ is nearer, and the predicted class is “exit.”

With weights $(0.1,1)$, the squared distance to $P$ is $0.1(2^2)+1(0^2)=0.4$, so its distance is $\sqrt{0.4}\approx0.632$. The squared distance to $Q$ remains $0.1(0^2)+1(1^2)=1$, so its distance remains 1. Case $P$ is now nearer, and the predicted class is “retain.”

Check. Downweighting the first predictor reduces the penalty for $P$'s difference on that feature from a squared-distance contribution of 4 to 0.4. The distances confirm that the nearest case, and hence the $k=1$ classification, changes.

Choosing and Interpreting Weights

Feature weights should be interpreted relative to the scale of the predictors. If one predictor is measured in large numerical units and another in small units, raw differences can dominate distance for reasons unrelated to predictive relevance. Earlier material on feature scaling addresses that issue: weights do not automatically correct incompatible units. A weighting scheme should be applied to predictors whose measurement scales have been considered.

Weights can encode a prespecified importance judgment, or they can be selected using data. If they are selected empirically, the choice should be assessed using training and validation data in a way that does not use the final test set to tune the model. This is a generalization concern: a weighting scheme that creates attractive neighbors in the training sample may not improve predictions on new cases.

It is often helpful to normalize nonnegative weights so that they sum to one. This makes their relative shares easier to communicate, but normalization alone does not change neighbor order. For example, weights $(4,1)$ and $(0.8,0.2)$ have the same ratio; the latter multiplies all distances by $1/\sqrt{5}$ in the Euclidean case. The neighbor ranking is identical.

Feature weights also differ from kernel weights and distance weights used after neighbors are identified. Feature weights change the distance calculation itself. A change can therefore alter the identity of the $k$ nearest cases before any prediction averaging or voting takes place.

Common Mistakes

Multiplying differences by weights outside the formula. For weights $(4,1)$ and difference vector $(1,2)$, the weighted Euclidean distance is $\sqrt{4(1^2)+1(2^2)}=\sqrt{8}$, not $\sqrt{(4\cdot1)^2+(1\cdot2)^2}=\sqrt{20}$. The weights multiply squared differences.
Assuming that every larger weight guarantees a particular candidate is closer. A candidate’s distance depends on all weighted feature contributions. Recompute the full sum for every candidate.
Thinking that a common rescaling changes the ranking. Multiplying all weights by the same positive constant multiplies every distance by the same factor, so the order and any existing ties are preserved.
Confusing feature weights with neighbor weights. Feature weights determine which observations are nearest. Neighbor weights affect how selected observations contribute to a prediction.

Key takeaway: In weighted KNN, calculate the contribution $w_j|x_j-z_j|^r$ for every feature and candidate before ranking distances. Relative feature weights can change the selected neighborhood and prediction; multiplying all weights by a common positive constant cannot change the ranking.

Check Your Understanding

For each problem, use the stated weighted Euclidean distance and show the feature contributions before deciding which cases are nearest.

A query is $(0,0)$. Candidate $U=(1,2)$ has response 5, and candidate $V=(2,0)$ has response 11. With weights $(1,1)$, which candidate is nearest and what is the $k=1$ regression prediction? (Answer: $V$; prediction 11.)
For a query $(0,0)$, candidates $A=(1,2)$, $B=(2,0)$, and $C=(0,3)$ have responses 8, 20, and 50. With weights $(4,1)$, identify the two nearest cases and their local-average prediction. (Answer: $A$ and $C$; prediction 29.)
For a query $(0,0)$, candidate $A=(1,1)$ and candidate $B=(2,0)$ are compared using weights $(0.25,0.75)$. Which is closer? If both weights are multiplied by 4, what happens to the distances and ranking? (Answer: They are equally close; both distances double, and the tie remains.)
A query is $(0,0)$. Case $P=(2,0)$ is “retain,” and case $Q=(0,1)$ is “exit.” For $k=1$, compare weights $(1,1)$ with weights $(0.1,1)$. What is the predicted class under each choice? (Answer: “Exit” with $(1,1)$; “retain” with $(0.1,1)$.)
A query is $(0,0)$. Candidate $M=(1,1)$ has response 10, $N=(0,2)$ has response 20, and $O=(2,0)$ has response 40. With weights $(1,4)$ and $k=2$, which candidates are selected and what is their local-average prediction? (Answer: $M$ and $O$; prediction 25.)

What You'll Learn