Distance-Weighted KNN Regression

Why Weight Neighbors by Distance?

K-nearest-neighbor regression predicts a numeric response at a query point using the responses of its \(K\) nearest training observations. An ordinary local average gives each selected neighbor the same influence, whether it is just inside the neighborhood or near its edge. Distance-weighted KNN regression changes that rule: a neighbor closer to the query point receives more weight than a farther neighbor.

The method still uses a specified distance measure and a selected set of \(K\) neighbors. It changes how their responses are combined, not how the distances or neighborhood are obtained. The distances must be calculated consistently with the chosen predictors and their scaling; a distance is meaningful only relative to those choices.

Definition: Let \(x_0\) be a query point, and let \(N_K(x_0)\) denote its \(K\) selected nearest training observations. For each selected observation \(i\), let \(d_i\) be its distance from \(x_0\), and let \(y_i\) be its numeric response. Assign a nonnegative raw weight \(w_i=g(d_i)\), where \(g\) is a decreasing function of distance. If \(\sum_{i\in N_K(x_0)}w_i>0\), define the normalized weight \(\alpha_i=w_i/\sum_{j\in N_K(x_0)}w_j\). The distance-weighted KNN regression prediction is \(\hat f(x_0)=\sum_{i\in N_K(x_0)}\alpha_i y_i\).

A common choice is inverse-distance weighting, \(w_i=1/d_i\), or inverse-distance-to-a-power weighting, \(w_i=1/d_i^p\) for a specified \(p>0\). These choices require \(d_i>0\). Another decreasing choice is \(w_i=1/(1+d_i)\), which is defined at distance zero. In every case, the chosen weights must have a positive sum so that normalization is possible. The rule for a zero-distance observation must be specified if an inverse-power formula is used; that formula itself does not apply at \(d_i=0\).

The normalized weights are nonnegative and sum to one. They can therefore be read as the relative contributions to the prediction, but they are not probabilities of outcomes. The prediction is a weighted average of the observed responses in the selected neighborhood.

From Raw Weights to a Prediction

The calculation has three parts: choose the weight function, calculate the raw weights from the distances, then normalize and combine the responses. Normalization is essential because the raw weights generally do not sum to one.

Calculate raw weights.
Apply the specified decreasing function to each selected neighbor’s distance, using the same formula for every neighbor.

Normalize.
Divide each raw weight by the sum of all raw weights. Check that the normalized weights sum to one.

Combine responses.
Multiply each response by its normalized weight and add the results. Keep exact fractions or unrounded values until the final reported prediction.

Worked Example: Inverse-Squared Weights for Three Neighbors

A hypothetical claim-severity analysis has selected three neighboring records for a query case. Their distances are \(1\), \(2\), and \(3\), and their recorded responses, in thousands of dollars, are \(5\), \(11\), and \(23\). Use \(w_i=1/d_i^2\) to predict the response. Report the prediction to four decimal places.

Method. Apply inverse-squared distance weights, normalize them, and calculate the weighted response average. All distances are positive, so the specified formula applies.

The raw weights are \(w_1=1/1^2=1\), \(w_2=1/2^2=1/4\), and \(w_3=1/3^2=1/9\). Their sum is \(1+1/4+1/9=36/36+9/36+4/36=49/36\).

The normalized weights are \(\alpha_1=1/(49/36)=36/49\), \(\alpha_2=(1/4)/(49/36)=9/49\), and \(\alpha_3=(1/9)/(49/36)=4/49\). As a check, \(36/49+9/49+4/49=49/49=1\).

The prediction is \((36/49)(5)+(9/49)(11)+(4/49)(23)\). The weighted numerator is \(180/49+99/49+92/49=371/49\), so the prediction is \(371/49=7.571428\ldots\). Rounded to four decimal places, it is 7.5714 thousand dollars.

Check and common mistake. The largest weight belongs to the nearest neighbor, and the prediction lies between the smallest and largest responses, as a weighted average should. A frequent arithmetic error is to miscalculate the weighted numerator; here its exact value before division by the raw-weight sum is \(371/36\), and dividing by \(49/36\) gives \(371/49\), not \(7.3636\).

Why Normalization Works

Normalization converts raw weights, whose scale depends on the weight function, into relative weights. Multiplying every raw weight by the same positive constant leaves the normalized weights and prediction unchanged. For example, the weights \(1\) and \(1/2\) have the same normalized values as \(2\) and \(1\).

There is also an optimization interpretation. Given fixed nonnegative weights with a positive sum, consider choosing a single value \(a\) to represent the neighborhood by minimizing the weighted sum of squared deviations from its responses.

Q(a)=\sum_{i\in N_K(x_0)}w_i(y_i-a)^2.

Differentiating with respect to \(a\) gives

\frac{dQ(a)}{da} =-2\sum_{i\in N_K(x_0)}w_i(y_i-a).

At a minimum, this derivative is zero. Rearranging the resulting equation gives the weighted average:

\sum_i w_i y_i-a\sum_i w_i=0 \quad\Longrightarrow\quad a=\frac{\sum_i w_i y_i}{\sum_i w_i} =\sum_i\left(\frac{w_i}{\sum_jw_j}\right)y_i.

This stationary point is the unique minimum when the weights are nonnegative and have positive sum: the second derivative is \(2\sum_iw_i>0\). Thus, the normalized distance-weighted prediction is exactly the value minimizing the weighted squared error within the selected neighborhood. This justification concerns the local prediction rule; it does not establish that distance weighting will always predict better than ordinary averaging.

Worked Calculations and Comparison

Worked Example: A Decreasing Weight That Is Not Inverse Distance

For a hypothetical portfolio-severity estimate, two selected neighbors have distances \(1\) and \(2\), with responses \(30\) and \(18\), respectively. Use \(w_i=1/(1+d_i)\). Find the normalized weights and prediction.

Method. Calculate the specified raw weights, divide by their sum, and apply the resulting relative weights to the responses. This weight function is defined for both distances.

The raw weights are \(w_1=1/(1+1)=1/2\) and \(w_2=1/(1+2)=1/3\). Their sum is \(1/2+1/3=5/6\). Therefore, \(\alpha_1=(1/2)/(5/6)=3/5\) and \(\alpha_2=(1/3)/(5/6)=2/5\). They sum to \(3/5+2/5=1\).

The prediction is \((3/5)(30)+(2/5)(18)=18+36/5=18+7.2=\) 25.2.

Check and common mistake. The closer neighbor receives the larger normalized weight, and \(25.2\) lies between \(18\) and \(30\). Using \(2/3\) and \(1/3\) as the normalized weights would be incorrect: those are not the normalization of raw weights \(1/2\) and \(1/3\). The correct normalized weights are \(3/5\) and \(2/5\).

Worked Example: Comparing Weighted and Ordinary Local Averages

A hypothetical reserve estimate uses three selected records at distances \(2\), \(4\), and \(8\), with responses \(15\), \(21\), and \(33\), in thousands of dollars. Compare the inverse-distance prediction with the ordinary local average. Which prediction is closer to the response of the nearest neighbor?

Method. First calculate the inverse-distance weighted prediction. Then calculate the ordinary mean of the same three responses so the comparison changes only the weighting rule.

The raw inverse-distance weights are \(1/2\), \(1/4\), and \(1/8\). Their sum is \(4/8+2/8+1/8=7/8\). The normalized weights are \((1/2)/(7/8)=4/7\), \((1/4)/(7/8)=2/7\), and \((1/8)/(7/8)=1/7\). They sum to one.

The weighted prediction is \((4/7)(15)+(2/7)(21)+(1/7)(33) =60/7+42/7+33/7 =135/7 =19.285714\ldots\). To four decimal places, the weighted prediction is 19.2857 thousand dollars.

The ordinary local average is \((15+21+33)/3=69/3=\) 23 thousand dollars. The nearest response is \(15\). The weighted prediction is \(135/7-15=30/7\approx4.2857\) away from \(15\); the ordinary average is \(23-15=8\) away. Therefore, the weighted prediction is closer to the nearest response.

Check and common mistake. The inverse-distance numerator using raw weights is \(15/2+21/4+33/8=60/8+42/8+33/8=135/8\). Dividing by the raw-weight sum \(7/8\) gives \(135/7\), not \(18.4286\). The ordinary average uses equal weights and is not the inverse-distance result.

How the Weighting Choice Changes the Prediction

When the same set of neighbors and distances is used, a strictly decreasing weight function assigns a larger weight to a closer observation. Increasing the power \(p\) in \(1/d_i^p\) generally makes the relative influence of distance differences more pronounced. It does not change which observations are in the selected neighborhood; it changes their contributions within it.

If all selected neighbors are assigned equal weights, the normalized weights are each \(1/K\), and the rule reduces to the ordinary local average. If distances differ but the weight function is decreasing, the weighted result generally differs from that average. Equal responses, or particular combinations of distances and responses, can nevertheless make the two predictions coincide.

Because normalized weights are nonnegative and sum to one, the prediction cannot be smaller than the smallest selected response or larger than the largest selected response. This is a useful arithmetic check. It also means that this local weighted average does not extrapolate beyond the response range of its selected neighbors.

Common Mistakes

Using the raw weighted sum as the prediction. The prediction is \(\sum_i w_i y_i/\sum_i w_i\), not just \(\sum_iw_iy_i\). Equivalently, use normalized weights that sum to one.
Normalizing by the wrong total. Add the raw weights before dividing each one. Do not assume weights such as \(1/2\) and \(1/3\) already sum to one.
Reversing the influence of distance. Under a decreasing weight function, a smaller distance must produce a larger weight, not a smaller one.
Changing the neighborhood while comparing averages. To isolate the effect of weighting, compare weighted and ordinary predictions using the same selected neighbors and responses.
Applying inverse distance at zero distance. The expression \(1/d_i^p\) is undefined at \(d_i=0\). A calculation using that formula requires positive distances or an explicitly specified alternative rule.
Rounding too early. Keep weights as fractions where convenient, normalize using their full values, and round only the final prediction to the requested precision.

Key takeaway: Distance-weighted KNN regression is a normalized local average. Calculate the raw weights from distance, divide each by their total, and then average the selected responses using those normalized weights.

Check Your Understanding

For each problem, use the stated neighbors and weight rule. Check that normalized weights sum to one before calculating the prediction.

A query has two selected neighbors at distances \(1\) and \(3\), with responses \(12\) and \(24\). Using inverse-distance weights, find the normalized weights and prediction. (Answer: normalized weights \(3/4\) and \(1/4\); prediction \(15\).)
Three neighbors have distances \(2\), \(5\), and \(10\), and responses \(10\), \(16\), and \(28\). Use inverse-distance weights. Find the normalized weights and prediction. (Answer: normalized weights \(5/8\), \(1/4\), and \(1/8\); prediction \(13.75\).)
Two neighbors have distances \(1\) and \(2\), and responses \(40\) and \(16\). Use \(w_i=1/(1+d_i)\). Find the normalized weights and prediction. (Answer: normalized weights \(3/5\) and \(2/5\); prediction \(30.4\).)
Three neighbors have distances \(1\), \(2\), and \(4\), and responses \(4\), \(10\), and \(22\). Use inverse-squared weights and compare the prediction with the ordinary local average. (Answer: normalized weights \(16/21\), \(4/21\), and \(1/21\); weighted prediction \(6\); ordinary average \(12\).)
Three neighbors have distances \(3\), \(3\), and \(6\), and responses \(8\), \(20\), and \(32\). Use inverse-distance weights. Find the normalized weights, prediction, and ordinary local average. (Answer: normalized weights \(2/5\), \(2/5\), and \(1/5\); weighted prediction \(17.6\); ordinary average \(20\).)

What You'll Learn