Noise, Outliers, and Robust KNN Predictions

Why Noise and Outliers Matter in KNN

K-nearest-neighbor (KNN) predictions are local: a prediction is based on the training observations closest to a query point under the chosen distance. That local character is useful when nearby cases have similar outcomes, but it also means that a few unusual observations can have a disproportionate effect. In a hypothetical insurance application, a single incorrectly coded claim category, an unusually large recorded claim, or an extreme predictor value could affect a model’s prediction in quite different ways.

The first step is to identify what is unusual. A mislabeled classification response is not the same problem as an extreme numerical response. Neither is the same as an unusual predictor value, which can change which observations count as neighbors. Increasing $k$ and weighting neighbors by distance can help in some situations, but neither is a universal outlier correction.

Definition: For a query predictor vector $x$, KNN selects the $k$ training observations with the smallest distances to $x$, using a specified predictor scaling and distance measure. In regression, the ordinary KNN prediction is the mean of the selected responses. In classification, the prediction is the class receiving the most votes. A label error is an incorrect categorical response; a response outlier is an unusually extreme numerical response; a predictor outlier is an unusual value or combination of values in the predictors.

KNN does not require a particular parametric distribution for the responses. It does, however, rely on a meaningful distance measure and on the idea that nearby training cases provide useful information about the query. If the predictors are poorly scaled, the data are sparse near the query, or nearby responses are contaminated, that local information may be unreliable.

Label Errors in Classification

In KNN classification, each neighbor contributes a vote for its recorded class. A mislabeled training case contributes a vote for the wrong class. Its effect depends both on how many other neighbors support each class and on its distance if the voting rule uses distance weights. A single incorrect label may decide a small-neighborhood prediction, especially when the vote is otherwise close.

Worked Example: A Mislabeled Point in a Classification Neighborhood

A hypothetical classifier distinguishes claim records in classes A and B. For a new record, the five nearest training cases have distances and recorded classes as follows: $0.2$: B; $0.3$: A (known to be mislabeled, with true class B); $0.4$: B; $0.8$: A; and $1.0$: B. Find the ordinary KNN predictions for $k=1$, $k=3$, and $k=5$. Then find the inverse-distance-weighted vote proportions among the three nearest cases, using weights $w_j=1/d_j$.

Method. For ordinary voting, count each neighbor once. For weighted voting, add the weights separately within each class and divide each class total by the sum of all three weights.

For $k=1$, the sole neighbor is class B at distance $0.2$, so the prediction is B. For $k=3$, the labels are B, A, B: B has two votes and A has one, so the prediction is B. For $k=5$, B has three votes and A has two, so the prediction is again B.

For weighted voting with $k=3$, the weights are $1/0.2=5$, $1/0.3=10/3$, and $1/0.4=2.5$. The total weight is $5+10/3+2.5=65/6$. Class B has weight $5+2.5=7.5=45/6$, and class A has weight $10/3=20/6$. Thus the weighted proportions are $45/65=0.6923$ for B and $20/65=0.3077$ for A, to four decimals. The weighted prediction is B.

Check. The two class proportions sum to $0.6923+0.3077=1.0000$. If the erroneous A label were corrected to B, all three nearest cases would vote B. The method avoids letting this one mislabeled case decide the prediction; it does not make the label error disappear.

This example illustrates a useful but limited effect of neighborhood size. With one neighbor, the prediction depends entirely on that observation. Adding more neighbors can dilute an isolated label error when the surrounding cases support the correct class. But if a larger neighborhood crosses a genuine class boundary or contains many cases from another region, increasing $k$ can instead make the prediction worse. The bias-variance tradeoff remains relevant: larger neighborhoods generally smooth predictions, but may smooth away real local structure.

Extreme Responses in Regression

In ordinary KNN regression, each of the $k$ selected responses contributes equally to the prediction. One extreme response can therefore pull the local mean substantially, particularly when $k$ is small. Distance weighting changes the relative contributions according to proximity; it does not assess whether a response value is plausible. An extreme response that is also very close to the query may receive a large weight.

Result: For positive neighbor distances $d_j$, inverse-distance-weighted KNN regression predicts $$ \hat{m}(x)=\frac{\sum_{j=1}^{k} w_jY_j}{\sum_{j=1}^{k}w_j}, \qquad w_j=\frac{1}{d_j}. $$ The formula assumes none of the selected distances is zero. If a selected distance is zero, the zero-distance case requires a separate convention rather than direct substitution into $1/d_j$.

Worked Example: Weighting a Distant Extreme Response

For a hypothetical query, the three selected observations have distances $0.1$, $0.2$, and $0.9$, with responses $20$, $21$, and $200$, respectively. Treat the response of $200$ as an extreme value. Calculate the ordinary mean, the inverse-distance-weighted prediction, and the median of the three responses.

Ordinary KNN mean. The response total is $20+21+200=241$, so the prediction is $241/3=80.3333\ldots$. To four decimal places, this is $80.3333$.

Inverse-distance weighting. The weights are $10$, $5$, and $1/0.9=10/9$. The weighted response total is $20(10)+21(5)+200(10/9)=200+105+2000/9=4745/9$. The weight total is $10+5+10/9=145/9$. The prediction is therefore $(4745/9)/(145/9)=4745/145=32.7241\ldots$, or $32.7241$ to four decimal places.

Median. The responses in order are $20,21,200$, so their median is $21$.

Check. The weighted prediction lies between the smallest and largest responses, and is much closer to the two ordinary responses than the unweighted mean is. The outlying response is also the farthest neighbor, so inverse-distance weighting limits its influence. The median is less affected by the extreme value in this three-response neighborhood.

A short derivation explains why the ordinary regression prediction is a mean. Suppose the selected responses are $Y_1,\ldots,Y_k$, and choose a single prediction $a$ to minimize their total squared error. The objective is $Q(a)=\sum_{j=1}^{k}(Y_j-a)^2$. Differentiating gives $Q'(a)=-2\sum_{j=1}^{k}(Y_j-a)$. Setting this equal to zero yields $ka=\sum_{j=1}^{k}Y_j$, so $a=\bar{Y}$. Since $Q''(a)=2k>0$, this value is the unique minimum. Squared error gives an extreme response a large contribution because its residual is squared.

For absolute error, the sum $\sum_{j=1}^{k}|Y_j-a|$ is minimized by a median of the selected responses. This gives a possible robust local regression summary: it reduces the effect of a few extreme responses compared with the mean. It is not the ordinary mean-based KNN prediction, and it is not immune to many contaminated responses or to a neighborhood that is otherwise unrepresentative.

When Distance Weighting Helps—and When It Does Not

Distance weighting gives more influence to closer cases, so it can reduce the influence of a distant response outlier, as in the preceding example. The protection depends on the relationship between predictor distance and response contamination. If the extreme response is the closest observation, inverse-distance weighting may give it the greatest influence. A large $k$ may include more ordinary cases, but it does not guarantee that their combined weight will outweigh a very close, very unusual response.

For instance, if the responses $20, 21, 200$ occurred at distances $0.1, 0.2, 0.9$, the outlier is far and receives the smallest weight. If response $200$ instead occurred at distance $0.1$, its inverse-distance weight would be $10$, rather than $10/9$. The same weighting rule would then emphasize, not suppress, the extreme response. Always inspect both the response and the distance when interpreting a weighted prediction.

Predictor Outliers and the Distance Geometry

A predictor outlier can cause trouble even when its response is not extreme. If predictor variables are on different scales, large-scale variables can dominate Euclidean distance. Standardizing predictors using training data is a common way to make scales more comparable, but an extreme training value can inflate a variable’s standard deviation and thereby reduce that variable’s contribution to standardized distances. Thus an outlier can change which cases are neighbors, not merely how much a selected neighbor contributes.

Worked Example: A Predictor Outlier Changes the Nearest Neighbor

A hypothetical regression training set has predictor pairs and responses $A=(2,0)$, response $10$; $B=(0,1.5)$, response $30$; and $C=(100,0)$, response $50$. The query is $x=(0,0)$, and $k=1$. Compare the nearest neighbor using raw Euclidean distance with the nearest neighbor using each predictor standardized by its sample standard deviation calculated from the three training cases.

Raw distances. The distances from the query are $d_A=\sqrt{2^2+0^2}=2$, $d_B=\sqrt{0^2+1.5^2}=1.5$, and $d_C=\sqrt{100^2+0^2}=100$. The nearest case is B, so the raw-distance prediction is its response, $30$.

Standard deviations. For predictor 1, the mean is $(2+0+100)/3=34$. The sum of squared deviations is $(2-34)^2+(0-34)^2+(100-34)^2=1024+1156+4356=6536$. The sample standard deviation is $s_1=\sqrt{6536/2}=\sqrt{3268}$, approximately $57.1664$. For predictor 2, the mean is $(0+1.5+0)/3=0.5$. The sum of squared deviations is $0.25+1+0.25=1.5$, so its sample standard deviation is $s_2=\sqrt{1.5/2}=\sqrt{0.75}$, approximately $0.8660$.

Standardized distances. Since all coordinates are centered by the same training means for every case and the query, the standardized difference on each predictor is the original difference divided by that predictor’s sample standard deviation. The distance to A is $2/s_1$, approximately $0.0350$. The distance to B is $1.5/s_2$, approximately $1.7321$. The distance to C is $100/s_1$, approximately $1.7493$. The nearest case is now A, so the standardized-distance prediction is $10$.

Check. The standardization changed the nearest case from B to A, and therefore changed the $k=1$ prediction from $30$ to $10$. The unusual value $100$ increased the scale estimate for predictor 1, making a difference of $2$ on that predictor count relatively little. This is not a reason to delete an extreme value automatically: first determine whether it is an error or a valid observation, and understand that the chosen scaling defines the neighborhood.

A Practical Reading of Robustness

“Robust” does not mean unaffected by every unusual observation. It means less sensitive to a specified kind of contamination under particular conditions. Larger $k$ can dilute an isolated label error or response outlier when additional neighbors are relevant and reasonably clean. Distance weighting can reduce the contribution of a far neighbor, but can magnify the effect of a close contaminated neighbor. A median can reduce sensitivity to extreme responses, but it does not correct poor predictor distances or erroneous class labels.

When interpreting a KNN prediction, separate three questions: Are the recorded outcomes trustworthy? Are the selected cases genuinely comparable to the query? Does the prediction rule give one observation disproportionate influence? These questions help identify whether the issue lies in labels or responses, in the distance geometry, or in the aggregation rule. This distinction is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Common exam trap: Do not assume that distance weighting automatically protects regression predictions from extreme responses. The weights depend on predictor distance, not on whether a response is unusually large. An extreme response at the smallest distance can dominate a weighted prediction. Also, do not treat every predictor outlier as a response outlier: it can affect the selected neighborhood through the distance calculation.

Check Your Understanding

For each problem, identify the type of contamination before choosing a KNN response.

A query has three nearest class labels, in distance order: C, D, C. The middle label is known to be a recording error and should be C. Find the ordinary $k=1$ and $k=3$ predictions before correction, then the $k=3$ prediction after correction. (Answer: Before correction, $k=1$ predicts C and $k=3$ predicts C by a 2–1 vote; after correction, $k=3$ predicts C unanimously.)
A regression neighborhood has responses $8, 10, 70$ at distances $0.25, 0.5, 1$, respectively. Calculate the ordinary mean and inverse-distance-weighted prediction. Give both to four decimals. (Answer: Ordinary mean $=29.3333$; weighted prediction $=(8(4)+10(2)+70(1))/(4+2+1)=122/7=17.4286$.)
A query’s selected responses are $5, 6, 100$. What is their median, and what is their ordinary mean? Which summary is less affected by the single extreme response in this neighborhood? (Answer: Median $=6$; mean $=111/3=37$; the median is less affected.)
For $k=3$, an extreme response is at distance $0.1$, while two ordinary responses are at distances $0.2$ and $0.4$. Under inverse-distance weights, which observation receives the largest weight? Explain whether the weighting rule protects the prediction from that response merely because the response is extreme. (Answer: The observation at distance $0.1$ receives weight $10$, the largest; no, inverse-distance weighting uses distance, not response size, so it does not protect against a close extreme response.)
A two-predictor training set contains a very large value on predictor 1. The predictors are standardized by their training sample standard deviations before distance calculation. What specific effect can the large value have on distances, and does its presence alone establish that the record should be removed? (Answer: It can inflate predictor 1’s standard deviation and shrink that predictor’s contribution to standardized distances, potentially changing the neighbors; no, the value must be assessed as a possible error or valid extreme observation.)

What You'll Learn