Why Integrated KNN Problems Need a Workflow
A K-nearest-neighbors (KNN) prediction is determined by several choices at once: how distances are measured, how predictors are scaled, how many neighbors are included, and how their responses are combined. A mistake in an early choice can change the neighbor set, so a correct vote calculation may still produce an inappropriate prediction. Validation adds another layer: the data used to assess a final model must not influence the choices being assessed.
The calculations in this tutorial use classification. The same distance and validation principles apply to KNN regression, where neighbor responses are averaged or weighted rather than voted on. KNN does not require a particular probability distribution for the predictors or responses. It does require a defined distance, consistently processed predictors, and a specified prediction rule.
A common distance for numeric predictors \(x_1,\ldots,x_p\) is Euclidean distance. Standardized Euclidean distance first centers and scales each predictor using quantities estimated from the training data. The same training means and standard deviations must then be used to transform query, validation, and test observations. If a training standard deviation is zero, division by it is undefined; that predictor must be handled separately, for example by removing it if it is constant.
Here \(s_j\) is the standard deviation of predictor \(j\) in the training set. Standardization does not guarantee that a predictor is useful; it only changes the scale on which differences contribute to distance. The distance metric and any feature weights should therefore be chosen as part of the modeling procedure, not changed after examining final test results.
Why Standardization Changes Distances
The standardized distance formula follows by measuring each coordinate difference in standard-deviation units. If two observations differ by \(s_j\) on predictor \(j\), then their contribution inside the squared distance is \((s_j/s_j)^2=1\). A difference of \(2s_j\) contributes \(4\). Thus a large-unit predictor does not automatically dominate merely because its numerical values are large.
By contrast, raw Euclidean distance adds squared differences in the original units. A numerical difference of 20 can overwhelm a difference of 2 even when the latter is more unusual relative to its predictor’s spread. This is why distance calculations must be matched to the scale and meaning of the predictors.
Worked Example: Scaling Reverses the Nearest Neighbor
A hypothetical insurer classifies an applicant as “Renew” or “Lapse” using two numeric predictors, \(X\) and \(Z\). The training standard deviations are \(s_X=4\) and \(s_Z=20\). A query applicant has \((X,Z)=(10,100)\). Three training applicants are \(A=(12,110)\), labeled Renew; \(B=(18,102)\), labeled Lapse; and \(C=(9,130)\), labeled Renew. Compare raw and standardized Euclidean distances, and find the \(k=1\) prediction under each distance.
Method. Calculate each coordinate difference from the query, square and add for raw distance, and divide each difference by its training standard deviation before squaring for standardized distance.
For \(A\), the coordinate differences are \(2\) and \(10\). Its raw distance is \(\sqrt{2^2+10^2}=\sqrt{104}\approx10.1980\). Its standardized distance is \(\sqrt{(2/4)^2+(10/20)^2}=\sqrt{0.25+0.25}=\sqrt{0.5}\approx0.7071\).
For \(B\), the differences are \(8\) and \(2\). Its raw distance is \(\sqrt{8^2+2^2}=\sqrt{68}\approx8.2462\). Its standardized distance is \(\sqrt{(8/4)^2+(2/20)^2}=\sqrt{4+0.01}=\sqrt{4.01}\approx2.0025\).
For \(C\), the differences are \(-1\) and \(30\). Its raw distance is \(\sqrt{(-1)^2+30^2}=\sqrt{901}\approx30.0167\). Its standardized distance is \(\sqrt{(-1/4)^2+(30/20)^2}=\sqrt{0.0625+2.25}=\sqrt{2.3125}\approx1.5207\).
The raw-distance order is \(B,A,C\), so the \(k=1\) prediction is Lapse. The standardized-distance order is \(A,C,B\), so the \(k=1\) prediction is Renew. For a further check, the two nearest standardized neighbors are both labeled Renew, whereas the two nearest raw-distance neighbors include one of each class. The closest neighbor—and thus the \(k=1\) result—really does change when the scale changes.
Common mistake avoided: choosing a metric because its distance values look familiar, or standardizing with information from outside the training set. The stipulated training standard deviations are used for every standardized calculation here.
Neighborhood Size and Distance-Weighted Voting
The value of \(k\) controls how many training responses contribute to a prediction. A small \(k\) uses a very local neighborhood; a larger \(k\) includes more observations and may smooth local variation. Neither choice is automatically best. It must be assessed using data reserved for model selection, rather than selected by looking at the final test outcomes.
With inverse-distance weights, the total vote for class \(c\) among the \(k\) selected neighbors is the sum of the weights of neighbors in that class. A normalized weighted vote proportion is the class’s weight divided by the total weight. Normalization does not change which class wins, but it expresses the result on a zero-to-one scale.
Here \(N_k(x)\) is the set of \(k\) nearest training observations, \(Y_j\) is neighbor \(j\)'s class, and \(\mathbf{1}(Y_j=c)\) equals 1 when that class is \(c\) and 0 otherwise. These formulas assume the selected neighbor distances are positive. If a neighbor has distance zero, inverse-distance weights are undefined; a standard remedy is to base the prediction on the zero-distance observations alone, with an explicitly stated tie rule.
Worked Example: The Vote Changes When Weights Are Used
A hypothetical claim-triage classifier has five nearest training cases at distances \(0.5,1.0,1.5,2.0,2.5\) from a new case. Their labels, in that order, are A, B, B, A, A. For \(k=3\), find the unweighted prediction, the inverse-distance weighted prediction, and the normalized weighted vote for each class.
Method. First count the three closest labels. Then calculate \(1/d_j\) for those same three neighbors; do not include weights from the fourth or fifth neighbor when \(k=3\).
Among the first three neighbors, class A appears once and class B appears twice. The unweighted vote proportions are \(1/3\) for A and \(2/3\) for B, so the unweighted prediction is B.
The three inverse-distance weights are \(1/0.5=2\), \(1/1.0=1\), and \(1/1.5=2/3\). Thus \(W_A=2\), while \(W_B=1+2/3=5/3\). The total weight is \(2+5/3=11/3\). The normalized weighted vote for A is \(2/(11/3)=6/11\approx0.5455\), and for B it is \((5/3)/(11/3)=5/11\approx0.4545\). The weighted prediction is A.
As a check, the normalized proportions sum to \(6/11+5/11=1\), and the weighted prediction differs from the unweighted prediction because the closest neighbor, labeled A, carries greater weight.
Common mistake avoided: counting the labels and then applying weights to all five listed cases. The specified neighborhood contains only the three nearest cases. Also, a normalized vote is a weighted proportion; it should not be described as a calibrated probability without additional evidence.
Validation Without Leakage
A validation set is used to compare modeling choices, such as candidate values of \(k\). A test set is held aside for a final assessment after those choices have been made. The test set should not be used to choose \(k\), distance metrics, transformations, or feature weights. If a test result influences any such decision, the test set has become part of model development.
Preprocessing can leak information too. Means, standard deviations, imputation values, and any data-driven feature choices must be estimated using the training portion of a split. Apply the resulting fitted transformation to validation or test observations without refitting it on those observations. This rule keeps the validation procedure aligned with how predictions will be made for genuinely new cases.
Worked Example: Tune on Validation, Assess Separately
A hypothetical equipment-monitoring team uses one numeric predictor and labels each observation A or B. Its training data are \((0,A),(3,B),(4,B),(10,A)\). The validation observations are \((1,A),(5,B),(9,A)\). Compare the validation misclassification rates for \(k=1\) and \(k=3\), select \(k\), and then assess it on a separate test set \((2,A),(8,B)\). Use ordinary one-dimensional distance and majority vote.
Method. For each validation point, rank only the four training observations by absolute distance. Record each prediction and compare it with the known validation label. Use validation results to select \(k\); do not use test labels until that selection is complete.
For validation point \(x=1\), the distances to training predictors \(0,3,4,10\) are \(1,2,3,9\). With \(k=1\), the nearest label is A, which is correct. With \(k=3\), the labels are A, B, B, so the prediction is B and is incorrect.
For validation point \(x=5\), the distances are \(5,2,1,5\). With \(k=1\), the nearest label is B, which is correct. With \(k=3\), the three nearest labels are B, B, A, so the prediction is B and is correct.
For validation point \(x=9\), the distances are \(9,6,5,1\). With \(k=1\), the nearest label is A, which is correct. With \(k=3\), the nearest labels are A, B, B, so the prediction is B and is incorrect. Therefore, \(k=1\) has \(0/3=0\) validation errors, while \(k=3\) has \(2/3\) validation errors. Select \(k=1\) based on this validation comparison.
Now assess the selected \(k=1\) on the separate test set. At \(x=2\), the distances to training predictors \(0,3,4,10\) are \(2,1,2,8\); the nearest label is B, so the prediction is wrong for the test label A. At \(x=8\), the distances are \(8,5,4,2\); the nearest label is A, so the prediction is wrong for the test label B. The selected model therefore makes \(2/2=1.0000\) test errors as a proportion. This small test set gives a very imprecise performance assessment, but it remains separate from the tuning decision.
Common mistake avoided: reporting the minimum validation error as though it were an independent final test error. The value \(0/3\) was used to select \(k\), so it is not a separate, untouched assessment of the selected procedure.
A Reliable Integrated-Problem Checklist
In an exam-style calculation, keep the stages separate. First establish which data are training, validation, and test data. Then identify the prescribed distance and scaling. Only after neighbor distances are calculated should you determine the neighborhood and vote. Finally, state what the validation or test result can and cannot support.
Use training observations to fit the KNN procedure and preprocessing. Use validation data for choices such as \(k\); reserve test data for final assessment.
Confirm whether the problem specifies raw or standardized distances, and use training-set scaling quantities consistently.
Calculate distances, order the training cases, and select exactly \(k\) neighbors. Check for tied distances or zero distances that require a stated rule.
Count labels for an unweighted vote, or calculate and sum the specified weights. Normalize only if a vote proportion is requested.
Do not describe a tuning-set minimum as an independent final performance estimate, and do not use test results to revise the model.
These checks matter because KNN is determined directly by the geometry of the observed predictors. A unit mismatch can alter the neighbors; a different \(k\) or weighting rule can alter the vote; and a leaked validation or test result can make an error estimate too optimistic. Explicitly showing the distance ranking, vote, and data role makes the reasoning auditable and is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, identify the distance or voting rule before calculating. Answers are given at the end of each question.
- A training set has standard deviations \(s_X=2\) and \(s_Z=5\). A query is \((2,20)\), and two candidates are \(A=(4,25)\) and \(B=(3,28)\). Calculate both standardized distances and identify the nearer candidate. (Answer: \(d_A=\sqrt{2}\approx1.4142\), \(d_B=\sqrt{2.81}\approx1.6763\); A is nearer.)
- The three nearest cases to a query have labels C, D, D, in that order. What is the unweighted \(k=3\) prediction, and what is the vote proportion for that class? (Answer: D; \(2/3\approx0.6667\).)
- For \(k=2\), the nearest case is class M at distance \(0.5\), and the second is class N at distance \(1\). Under inverse-distance weighting, calculate the normalized vote proportions and prediction. (Answer: M has weight 2, N has weight 1; proportions are \(0.6667\) and \(0.3333\), respectively; predict M.)
- A practitioner computes standardization means and standard deviations using the combined training and validation sets, then compares values of \(k\) on validation data. Identify the error and state the correct procedure. (Answer: preprocessing information from validation data leaked into model fitting; estimate scaling quantities on training data alone and apply those same quantities to validation data.)
- A team tries four values of \(k\), chooses the one with the lowest error on a validation set, and then reports that same lowest validation error as its final independent performance estimate. Explain the problem and name the appropriate separate data role. (Answer: the validation error was used for tuning and is not an independent final assessment; use an untouched test set for final assessment.)