Why Diagnose KNN Beyond Choosing \(k\)?
A small validation error is useful, but it does not by itself explain how a \(k\)-nearest-neighbor (KNN) model is behaving. A training-validation error curve can reveal patterns consistent with overfitting or underfitting. Inspecting the neighbors for particular predictions can reveal a different problem: the distance calculation may be selecting observations for a reason that is not meaningful in context.
This tutorial focuses on diagnosis. It uses a fixed distance rule and a sequence of candidate values of \(k\), then examines both aggregate errors and individual neighborhoods. The examples are hypothetical. No parametric distribution for the predictors or responses is assumed; the calculations use the stated observations and distance rule. These diagnostic skills are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
For a regression model, a common diagnostic is mean squared error (MSE). If the prediction for observation \(i\) is \(\hat{y}_i\) and its observed response is \(y_i\), then the MSE over a set of \(n\) observations is \(n^{-1}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2\). For classification, the misclassification rate is the number of incorrectly classified observations divided by the total number assessed. Compute training and validation errors separately: training error assesses the fitted observations, while validation error assesses held-out observations not used to fit the predictions.
A training error curve is not a substitute for validation. In particular, when a regression observation is allowed to be its own neighbor, a training prediction at \(k=1\) reproduces its response exactly. That fact can make the smallest-\(k\) training error look excellent even when prediction on new observations is poor.
Reading Training-Validation Error Curves
For each candidate \(k\), calculate the same error measure on the training set and on a separate validation set. Plotting both against \(k\), or placing their values in a table, helps separate three broad patterns. These patterns are diagnostic clues, not proofs that a model has a particular defect.
| Observed pattern | Possible interpretation | Diagnostic check |
|---|---|---|
| Low training error and noticeably higher validation error, especially for small \(k\) | Overfitting or high sensitivity to individual observations | Inspect small-\(k\) neighbors and whether a few observations dominate predictions |
| Both errors remain high, including at larger \(k\) | Underfitting, weak predictors, or a poor distance representation | Check whether predictions are overly smoothed and whether the predictors represent relevant similarity |
| Validation error changes sharply across nearby \(k\) values | Possible sensitivity to neighborhood composition or limited validation data | Inspect the identities, distances, and responses or classes of the neighbors |
The validation minimum is a practical selection rule among the tested candidates, not a guarantee that the chosen model is best in every population. If several values of \(k\) have similar validation errors, inspect the curve's overall shape and the neighborhoods. A small difference in one validation result can be sampling noise rather than a meaningful performance difference.
Worked Example: Diagnosing a Regression Error Curve
A hypothetical insurer uses KNN regression to predict a claim-severity score from recorded predictors. For four candidate values of \(k\), the training and validation MSEs are:
| \(k\) | Training MSE | Validation MSE |
|---|---|---|
| 1 | 0 | 16 |
| 3 | 1 | 9 |
| 7 | 4 | 6 |
| 15 | 9 | 8 |
Method. Compare validation MSE across the candidate \(k\) values, then compare each training error with its validation error. The error units are squared score units; there is no distributional assumption in this calculation.
Step 1: Find the validation minimum. The four validation MSEs are 16, 9, 6, and 8. The smallest is 6, attained at \(k=7\). Among these candidates, \(k=7\) is the validation-based choice.
Step 2: Interpret the small-\(k\) gap. At \(k=1\), the training MSE is 0 but the validation MSE is 16, a gap of \(16-0=16\). This is consistent with a highly flexible fit that reproduces training responses but predicts less well on held-out observations.
Step 3: Inspect the larger-\(k\) end. At \(k=15\), training MSE is 9 and validation MSE is 8. Both are higher than the minimum validation MSE of 6. This suggests that averaging over a larger neighborhood may be smoothing away useful local variation.
Check. The selected \(k=7\) has the lowest listed validation MSE, \(6<8<9<16\). The pattern supports a diagnosis of excess flexibility at very small \(k\) and possible oversmoothing at the largest \(k\). It does not establish that either mechanism is the only cause. The common mistake is to choose \(k=1\) because its training MSE is zero; validation error, not in-sample fit, is the relevant comparison for prediction on new cases.
Directly Inspecting Neighbors
An error curve shows what happens across a set of cases; a neighbor check investigates why a particular prediction may be surprising. For a selected query, record the chosen neighbors, their distances, and their responses or class labels. Ask whether these observations are plausibly similar to the query for the purpose of predicting the response.
Feature scale matters because ordinary Euclidean distance squares and adds raw coordinate differences. If one predictor's numerical scale is much larger than another's, its differences can dominate the distance. Standardizing predictors using training-set quantities is one possible remedy, but the choice of scaling should reflect the data and modeling goal. A validation set must not be used to calculate scaling quantities: doing so allows information from the held-out data to influence the fitted procedure.
Worked Example: Detecting a Scale Problem in Neighbor Selection
A hypothetical portfolio analyst predicts a service-cost score from two predictors: annual mileage, measured in thousands of miles, and vehicle age, measured in years. For a query vehicle with predictor vector \((12,4)\), two training vehicles have predictor vectors \(A=(12,7)\) and \(B=(14,4)\). Their scores are 100 and 300, respectively. Suppose the training-set standard deviations are 0.1 thousand miles for mileage and 10 years for age.
Method. Compare Euclidean distances on the raw predictors with distances after standardizing each coordinate by its training-set standard deviation. No distributional assumption is used.
Step 1: Calculate raw distances. For vehicle \(A\), the coordinate differences from the query are \(12-12=0\) and \(7-4=3\). Thus, \(d_A=\sqrt{0^2+3^2}=3\). For vehicle \(B\), the differences are \(14-12=2\) and \(4-4=0\), so \(d_B=\sqrt{2^2+0^2}=2\). On the raw scale, \(B\) is the \(k=1\) neighbor, and the predicted score is 300.
Step 2: Calculate standardized distances. Divide each coordinate difference by the corresponding training standard deviation. For \(A\), the standardized differences are \(0/0.1=0\) and \(3/10=0.3\), giving \(d_A^*=\sqrt{0^2+0.3^2}=0.3\). For \(B\), they are \(2/0.1=20\) and \(0/10=0\), giving \(d_B^*=\sqrt{20^2+0^2}=20\).
Step 3: Compare the neighbor and prediction. After standardization, \(A\) is the \(k=1\) neighbor and the predicted score is 100. The selected observation and prediction have changed substantially.
Check. The raw distances rank \(B\) closer (\(2<3\)); the standardized distances rank \(A\) closer (\(0.3<20\)). The example is deliberately constructed to show how a distance choice can determine the result. The analyst should investigate whether these standard deviations and this scaling reflect meaningful similarity before adopting either prediction. The common mistake is to treat a raw-distance neighbor as automatically meaningful without checking the units and scales of its predictors.
Checking Neighborhood Stability
A neighborhood can also be unstable when two or more observations have similar distances from the query. A small change in a predictor may change which observation is nearest, potentially changing a \(k=1\) prediction abruptly. For larger \(k\), the same change may or may not matter: the key question is whether the entering and leaving neighbors differ in responses or classes.
A practical check is to compare the selected neighbors at the original query with those at a nearby, plausible query. Note which observation enters or leaves and how much the prediction changes. Also check for ties, duplicate predictor vectors, and unusually influential responses. A large prediction change signals sensitivity worth investigating; it does not by itself prove that the model is unusable.
Worked Example: Testing a Classification Neighborhood for Instability
A hypothetical KNN classifier distinguishes two claim-review categories, Red and Blue, using one scaled predictor \(x\). Four training observations have \((x,\text{class})\) values \((9.8,\text{Red})\), \((10.3,\text{Blue})\), \((10.5,\text{Blue})\), and \((12,\text{Red})\). Use absolute distance and majority vote, with no tie in the votes. Compare queries \(x=10.00\) and \(x=10.06\) for \(k=1\) and \(k=3\).
Method. Calculate each absolute distance \(|x-x_i|\), rank the four observations, and compare the classes among the first \(k\).
Step 1: Query \(x=10.00\). The distances in the listed observation order are \(0.20, 0.30, 0.50,\) and \(2.00\). The nearest observation is the Red case at 9.8, so the \(k=1\) prediction is Red. The three nearest classes are Red, Blue, and Blue, so the \(k=3\) prediction is Blue.
Step 2: Query \(x=10.06\). The distances are \(0.26, 0.24, 0.44,\) and \(1.94\). The nearest observation is now the Blue case at 10.3, so the \(k=1\) prediction is Blue. The three nearest classes are Blue, Red, and Blue, so the \(k=3\) prediction is Blue.
Step 3: Diagnose the change. The query moved only \(10.06-10.00=0.06\) units. The \(k=1\) prediction changed from Red to Blue because the two closest cases exchanged order. The \(k=3\) prediction stayed Blue because the majority among its neighbors remained two Blue cases out of three.
Check. The example identifies a local sensitivity at \(k=1\) and a more stable vote at \(k=3\) for these two nearby queries. It does not show that \(k=3\) will be stable everywhere. The common mistake is to inspect only the predicted class and not the identities and distances of the cases that produced it.
A Diagnostic Workflow and Common Traps
Use a consistent sequence of checks when an error curve or prediction looks unexpected:
Use the same error measure and validation observations for every candidate \(k\). Keep validation observations out of fitting and preprocessing.
Find the validation minimum, then assess the training-validation gap and whether errors are high at both small and large \(k\).
For an important or surprising query, list neighbor distances and responses or classes. Check whether the observations make sense as comparables.
Ask whether small query changes alter the neighbor set or prediction, and whether predictor units or scaling choices dominate the distance.
Several interpretation traps recur. First, a zero or very small training error is not evidence of good generalization, particularly for \(k=1\). Second, a validation minimum among the tested candidates does not guarantee that the candidate set, split, or distance definition is appropriate. Third, a high validation error is not automatically proof of underfitting: weak predictors, noisy outcomes, poor scaling, or a small validation set can also contribute. Finally, standardize using training data only, and apply those same training-set scaling quantities to validation and future observations.
Check Your Understanding
For each problem, use the stated distance and error definitions. The scenarios and values are hypothetical.
- A regression model has validation MSEs 12, 7, 5, and 6 for \(k=1,3,5,9\), respectively. Which tested \(k\) minimizes validation MSE, and what is the MSE difference between the \(k=1\) and selected candidate? (Answer: \(k=5\); the difference is \(12-5=7\).)
- A classifier's training misclassification rates at \(k=1,3,7\) are \(0, 0.08, 0.16\), and its validation rates are \(0.24, 0.15, 0.18\). Which candidate has the lowest validation error, and what pattern does the \(k=1\) result suggest? (Answer: \(k=3\); \(k=1\) has a large training-validation gap consistent with overfitting or sensitivity.)
- A query has predictors \((5,2)\). Two training cases are \(A=(5,5)\) and \(B=(7,2)\). Using raw Euclidean distance, which is the \(k=1\) neighbor? If training standard deviations are 10 for the first predictor and 1 for the second, which case is nearest after dividing each coordinate difference by its standard deviation? (Answer: Raw: \(B\), since \(2<3\). Standardized: \(A\), since \(3/1=3\) versus \(2/10=0.2\).)
- For a one-dimensional classifier, a Red observation is at 4.9 and a Blue observation is at 5.2. Compare \(k=1\) predictions at queries 5.00 and 5.08. (Answer: At 5.00, Red is nearest because the distances are 0.10 and 0.20; at 5.08, Blue is nearest because the distances are 0.18 and 0.12. The prediction changes after a query shift of 0.08.)
- A practitioner calculates predictor means and standard deviations using the combined training and validation data, then evaluates validation error. What is the diagnostic concern, and what is the correction? (Answer: The validation predictors influenced preprocessing, creating information leakage; calculate scaling quantities from training data only, then apply those same quantities to validation data.)