Why Weight Neighbors by Distance?
A nearest-neighbor prediction uses nearby observations to estimate an outcome at a query point. With uniform weights, every selected neighbor contributes equally, whether it is extremely close or barely included. Distance weighting changes this: closer observations receive more influence. A compact kernel can also give observations outside a specified distance range exactly zero weight.
These choices affect the prediction even when the neighbors themselves are unchanged. They also express different assumptions about how local information should be used: equally across the neighborhood, more strongly near the query, or only within a finite radius with a particular decline toward its edge. All examples below use hypothetical data. No interest-rate or mortality assumption is required for these KNN calculations.
Three Ways to Assign Weights
The methods differ in how weight changes as distance increases. For uniform KNN, first select the \(k\) nearest observations and give each selected observation raw weight 1; observations outside those \(k\) receive weight 0. This is a sudden cutoff: distance makes no difference among the selected neighbors.
A common inverse-distance scheme gives a selected observation at distance \(d_i>0\) raw weight \(1/d_i\). More generally, \(1/d_i^p\), for a chosen \(p>0\), makes the decline steeper as \(p\) increases. In this tutorial, inverse-distance weighting means \(p=1\). It assumes distances are positive; the formula is undefined at zero distance. Weighting and normalization must use the same set of selected neighbors.
A kernel scheme converts distance to a relative distance \(u_i=d_i/h\), where \(h>0\) is a bandwidth measured in the same units as the distances. A compact kernel is zero beyond its finite support, so only points within a specified range can contribute. Two standard choices are the triangular kernel and the Epanechnikov kernel.
With either kernel, use raw weight \(w_i=K(d_i/h)\) and normalize the weights before predicting. Both kernels give zero weight at and beyond \(d_i=h\). Within the support, the triangular weight falls linearly with relative distance; the Epanechnikov weight falls as a quadratic function. A kernel prediction is undefined if all candidate observations receive zero weight, because the normalization denominator is then zero.
| Scheme | Raw weight | How influence declines | Range with nonzero weight |
|---|---|---|---|
| Uniform over \(k\) neighbors | 1 for selected neighbors | Does not decline within the selected set | The selected \(k\); all others have weight 0 |
| Inverse distance, \(p=1\) | \(1/d_i\) | Proportional to the reciprocal of distance | All selected neighbors with positive distance |
| Triangular kernel | \(\max(0,1-d_i/h)\) | Linear decline to zero at \(h\) | \(d_i<h\) |
| Epanechnikov kernel | \(\frac{3}{4}(1-(d_i/h)^2)\) within support | Quadratic decline to zero at \(h\) | \(d_i<h\) |
The choice of method is distinct from the choice of neighborhood size. For example, a model can first select the \(k\) nearest observations and then apply inverse-distance weights to those \(k\). A bandwidth-based kernel instead defines a distance scale \(h\); depending on the data, the number of observations inside that radius can vary from one query to another.
Why the Weighted Regression Formula Works
The weighted average is not merely a convenient arithmetic rule. It is the constant prediction that best fits the observed outcomes locally under a weighted squared-error criterion. This provides a short justification for the regression formula.
For nonnegative raw weights \(w_i\) with \(\sum_i w_i>0\), consider choosing a single prediction \(a\) to minimize \(Q(a)=\sum_i w_i(y_i-a)^2\).
\(Q'(a)=-2\sum_iw_i(y_i-a)\). At a minimum, \(Q'(a)=0\), so \(\sum_i w_i y_i-a\sum_iw_i=0\).
Thus \(a=\sum_iw_i y_i/\sum_iw_i=\sum_i\tilde w_i y_i\). Also \(Q''(a)=2\sum_iw_i>0\), so this stationary point is the unique minimum.
For classification, the weighted class fraction uses the same normalized weights but averages indicator values instead of numerical outcomes. It is a local estimate of class probability; assigning a predicted class then requires choosing the class with the largest estimated fraction, subject to any stated tie rule.
Worked Example: Three Regression Weighting Schemes
A hypothetical insurer uses the distance from a new policy to three historical policies to predict a continuous annual service cost. The distances are \(1\), \(2\), and \(4\) distance units, and the observed costs are \(\$100\), \(\$80\), and \(\$40\), respectively. Compare uniform weights over these three observations, inverse-distance weights, and triangular-kernel weights with \(h=4\). There is no interest, mortality, or probability-distribution assumption in this prediction problem.
Uniform method. Each selected observation has raw weight 1, so the total weight is \(1+1+1=3\). The prediction is \((100+80+40)/3=220/3=\$73.333\ldots\).
Inverse-distance method. The raw weights are \(1/1=1\), \(1/2=0.5\), and \(1/4=0.25\). Their sum is \(1.75=7/4\), so the normalized weights are \(4/7\), \(2/7\), and \(1/7\). The prediction is
Triangular-kernel method. Here \(u_i=d_i/h\), giving relative distances \(1/4\), \(1/2\), and \(1\). The raw weights are \(3/4\), \(1/2\), and 0. Their sum is \(5/4\), so the normalized weights are \(3/5\), \(2/5\), and 0. The prediction is \(100(3/5)+80(2/5)=\$92.00\).
Rounded to cents, the three predictions are \(\$73.33\), \(\$85.71\), and \(\$92.00\). The kernel gives the observation exactly at the bandwidth zero weight; it does not include that observation in the weighted average. A common mistake is to assign it weight 1 because it is a selected neighbor. The correction is to calculate \(K(d_i/h)\): at \(d_i=h\), the triangular kernel is zero.
Worked Example: Weighted Class Probabilities
A hypothetical underwriting review classifies nearby policies as either “review required” (R) or “no review” (N). Four selected neighbors have distances \(0.5,1,2,3\), with labels R, R, N, N, respectively. Compare the local R fraction under uniform weights, inverse-distance weights, and a triangular kernel with \(h=3\). The labels are categorical, and no interest, mortality, or distributional assumption is used.
Uniform weights. Two of the four equally weighted observations are R. Thus \(\hat p_R=2/4=0.5000\); the local N fraction is also \(0.5000\).
Inverse-distance weights. The raw weights are \(2,1,1/2,1/3\). Their sum is \(23/6\). The R total is \(2+1=3\), so
Triangular kernel. The relative distances are \(1/6,1/3,2/3,1\), giving raw weights \(5/6,2/3,1/3,0\). The total is \(11/6\). The R weight is \(5/6+2/3=9/6\), hence \(\hat p_R=(9/6)/(11/6)=9/11=0.8182\). The N fraction is \(2/11=0.1818\). The predicted class is R under all three schemes, but the estimated local probability differs. The common trap is to treat weighted class totals as probabilities before dividing by the sum of all weights.
Worked Example: Comparing Compact Kernels
For a hypothetical regression problem, four historical observations lie at distances \(0.5,1,2,3\) from a query, with outcomes \(12,20,40,60\) dollars. Use bandwidth \(h=4\) and compare the triangular and Epanechnikov predictions. The outcomes are hypothetical; no interest, mortality, or distributional assumption is needed.
Triangular kernel. The relative distances are \(1/8,1/4,1/2,3/4\), so the raw weights are \(7/8,3/4,1/2,1/4\). In units of eighths, these are \(7,6,4,2\), with sum 19. The weighted outcome total is \(12(7)+20(6)+40(4)+60(2)=84+120+160+120=484\). Therefore the prediction is \(484/19=\$25.4737\ldots\).
Epanechnikov kernel. The common factor \(3/4\) appears in every raw weight and cancels when normalizing, so use \(1-u_i^2\) for the calculation. In units of \(1/64\), the weights are \(63,60,48,28\), since \(1-(1/8)^2=63/64\), \(1-(1/4)^2=60/64\), \(1-(1/2)^2=48/64\), and \(1-(3/4)^2=28/64\). Their sum is 199, and the weighted outcome total in those same units is \(12(63)+20(60)+40(48)+60(28)=756+1200+1920+1680=5556\). The prediction is \(5556/199=\$27.9196\ldots\).
Rounded to cents, the predictions are \(\$25.47\) and \(\$27.92\). The Epanechnikov kernel retains more relative weight at the middle distances here than the triangular kernel does, changing the weighted average. A common mistake is to compare raw weights without normalizing; the prediction must divide the weighted outcome total by the total weight.
Interpreting Neighborhood Decay
A weighting scheme describes how quickly the influence of a point fades as it moves away from the query. Uniform weighting has no gradual decay inside its selected set: an observation just inside the selection boundary counts as much as the closest one, and an observation just outside counts not at all. Inverse-distance weighting gives progressively less influence with no finite distance cutoff, as long as the observation remains in the selected set. With \(1/d_i\), doubling a positive distance halves its raw weight; with \(1/d_i^p\), doubling it multiplies the raw weight by \(2^{-p}\).
A compact kernel combines gradual decay with a finite cutoff. The bandwidth controls the distance scale: increasing \(h\) makes a fixed distance a smaller fraction of the bandwidth, generally increasing its kernel weight and potentially allowing more observations to contribute. Decreasing \(h\) concentrates the prediction more tightly around the query. This changes the degree of locality, not merely the numerical scale of the weights. The comparison assumes the same distance definition is used throughout; changing how distances are calculated is a separate modeling decision.
The raw weight alone is not the final contribution. A point’s normalized influence depends on every other point’s raw weight through the denominator \(\sum_jw_j\). Consequently, a point can have the same raw weight in two neighborhoods but a different normalized weight if the other neighbors differ.
Common Mistakes
- Forgetting to normalize. A weighted regression prediction divides the weighted outcome total by the total weight, unless weights have already been normalized.
- Confusing raw distance with relative distance. Kernel weights use \(d_i/h\); the bandwidth and distance must be in compatible units.
- Giving a compact kernel positive weight outside its support. For the kernels defined here, \(d_i\geq h\) gives zero weight.
- Assuming all schemes produce the same estimate. Different weight patterns generally change both regression predictions and classification fractions.
- Using inverse distance at zero. The expression \(1/d_i\) is undefined when \(d_i=0\), so the stated formula requires strictly positive distances.
Check Your Understanding
For each problem, calculate the requested weighted estimate before checking the answer.
- A hypothetical regression query has two neighbors: distances 1 and 3, with outcomes \(\$20\) and \(\$50\). Find the uniform prediction over both neighbors, the inverse-distance prediction, and the triangular-kernel prediction with \(h=4\). (Answer: uniform \(\$35.00\); inverse-distance \(\$27.50\); triangular \(\$27.50\).)
- Three hypothetical claim-file records have distances \(0.5,1.5,2.5\) and continuous outcomes \(10,30,50\). Using a triangular kernel with \(h=3\), calculate the prediction. (Answer: \(190/9=\$21.11\) to the nearest cent.)
- Three neighbors have distances \(0.5,1,2\) and class labels X, Y, X. Calculate the inverse-distance estimate of the local probability of X and identify the predicted class. (Answer: \(5/7=0.7143\); class X.)
- Two hypothetical outcomes are 8 and 20 at distances \(0.5\) and \(1.5\). With an Epanechnikov kernel and \(h=2\), calculate the prediction. (Answer: \(130/11=\$11.82\) to the nearest cent.)
- Two observations have distances \(0.5\) and \(2\), outcomes 10 and 40, and a triangular-kernel bandwidth \(h=2\). Calculate the prediction and state why the second observation does not affect it. (Answer: \(\$10.00\); the second observation is at \(d_i=h\), so its kernel weight is zero.)