Why Irrelevant Predictors Matter in KNN
K-nearest-neighbor (KNN) methods predict from observations that are close to a query point in predictor space. Consequently, a predictor can affect a KNN model even when it carries no useful information about the response: its coordinate still contributes to the distance calculation. If that coordinate changes which observations count as neighbors, it can change the prediction.
“Irrelevant” does not mean that a variable is uninteresting or has no relationship to anything in the data. For this discussion, it means that the predictor provides no additional information about the response once the useful predictors are known. A variable can be irrelevant in that predictive sense and still vary substantially across observations.
For a numeric predictor set with ordinary Euclidean distance, the squared distance between a query \(x_0\) and an observation \(x_i\) is the sum of squared coordinate differences. A feature that contributes no response information still adds a nonnegative term to that sum. Scaling matters: distances should be calculated on appropriately scaled features when their units or ranges differ. Here, as in the earlier material on feature scaling, all numeric coordinates in the examples are already on the stated common scale.
A Distance Calculation Can Change the Neighbor Order
Worked Example: An Irrelevant Coordinate Reorders the Nearest Neighbors
A hypothetical insurer predicts a claim-handling score from a useful predictor \(X\). A second standardized predictor \(Z\), generated independently of the score once \(X\) is known, is suspected to be irrelevant. For a query with coordinates \((X,Z)=(0,0)\), three training observations have the following coordinates and observed scores. Use \(k=1\) and squared Euclidean distance to compare using \(X\) alone with using both predictors.
| Observation | \(X\) | \(Z\) | Observed score |
|---|---|---|---|
| A | 0.1 | 2.0 | 12 |
| B | 1.8 | 0.0 | 20 |
| C | 1.5 | 0.5 | 30 |
Method: compare squared distances, then use the score of the single nearest observation. With \(X\) alone, the squared distances from the query are:
Observation A is nearest, so the \(k=1\) regression prediction is 12. Now include \(Z\). The squared distances are:
Observation C is now nearest, and the prediction is 30. The added coordinate did not change any observed score; it changed which score is used for the query. Including an irrelevant predictor is not harmless merely because it has no response information.
Check. The full-distance ordering is C, B, A because \(2.50<3.24<4.01\). The prediction of 30 agrees with the score attached to the nearest observation. The common mistake this calculation avoids is treating a coordinate difference as its squared contribution: for A, the \(Z\) contribution is \(2.0^2=4.00\), not 2.0. Using 2.0 would give \(0.01+2.00=2.01\), incorrectly making A appear nearest.
Why Noise Adds to Distance
The effect is not limited to a particular arrangement of points. Suppose an irrelevant coordinate \(Z\) and the query coordinate \(z_0\) are independent draws from a standard normal distribution. The expected squared difference contributed by that coordinate is:
The equality uses independence, zero means, and unit variances. Thus, even though \(Z\) contains no response information, it contributes an expected 2 to the squared distance between two independently drawn points on this scale. With \(q\) independent irrelevant standard-normal coordinates, the expected total contribution is \(2q\). This is a distance calculation, not a guarantee that every neighbor order will change: the actual effect depends on the observed coordinates and on the separation among candidate observations.
The practical point is narrower than “more predictors are always worse.” A predictor that genuinely helps distinguish responses may improve predictions despite changing distances. The question is empirical: does including that feature, or feature set, improve predictive performance on data not used to fit the candidate predictions? The earlier material on feature weights and correlated predictors addresses other ways that coordinates affect distance; here the issue is response-irrelevant variation.
Compare Feature Subsets with Validation Performance
A direct way to assess candidate features is to fit KNN models using different subsets and compare their predictions on validation observations. For regression, a common criterion is validation mean squared error (MSE). If \(y_j\) is the observed response and \(\hat y_j\) is its validation prediction for \(m\) cases, then:
For classification, validation misclassification error is the proportion of cases assigned the wrong class. The metric should suit the task and be chosen before comparing candidate subsets. If both the feature subset and \(k\) are under consideration, compare the candidate combinations under the same validation plan.
Worked Example: Choose a Predictor Set by Validation MSE
A hypothetical modeler predicts a numerical service-cost index. Two KNN regression models have been fitted using only the training data, with the same value of \(k\). Model \(M_X\) uses predictor \(X\); model \(M_{X,Z}\) also uses a candidate predictor \(Z\). Their predictions on four validation cases are given below. Choose between the models using validation MSE.
| Case | Observed \(y_j\) | \(\hat y_j\) from \(M_X\) | \(\hat y_j\) from \(M_{X,Z}\) |
|---|---|---|---|
| 1 | 10 | 12 | 15 |
| 2 | 20 | 18 | 17 |
| 3 | 25 | 24 | 20 |
| 4 | 29 | 30 | 35 |
For \(M_X\). The residuals \(y_j-\hat y_j\) are \(10-12=-2\), \(20-18=2\), \(25-24=1\), and \(29-30=-1\). Their squared values are \(4,4,1,1\). Therefore, the sum of squared errors is \(4+4+1+1=10\), and:
For \(M_{X,Z}\). The residuals are \(10-15=-5\), \(20-17=3\), \(25-20=5\), and \(29-35=-6\). Their squares are \(25,9,25,36\), which sum to \(95\). Thus:
Choose \(M_X\) among these candidates because \(2.5<23.75\). This result supports leaving \(Z\) out for this modeling comparison; it does not prove that \(Z\) is irrelevant in every population or for every model. The common mistake to avoid is selecting the larger predictor set simply because it contains more data. The validation comparison favors the model with lower observed validation MSE, not the model with more coordinates.
Check. The two residual-square totals are 10 and 95, and dividing each by four gives 2.5 and 23.75. The choice follows the specified validation criterion.
Prevent Response Leakage During Feature Selection
Validation performance can guide feature selection, but the data roles must be kept distinct. Training responses are used to fit the KNN predictions. Validation responses are used to compare the candidate models or feature subsets. A separate test set, if available, is reserved for a final evaluation after those choices are made. Repeatedly choosing features based on test-set results turns the test set into part of model selection, so its final performance no longer serves as an independent evaluation.
A related problem is response leakage: a predictor uses information that would not be available at the time the prediction is supposed to be made, or it is constructed using the response being predicted. For example, in a hypothetical claim-severity model made at the time a claim is first reported, the eventual settled payment cannot be used as a predictor. It is determined after the outcome process has unfolded. Its apparent predictive value would not demonstrate a useful pre-settlement feature.
Feature screening must also respect the data split. If a screening rule uses response labels—for example, to rank variables by their association with the response—derive that rule from the training data, not from the full dataset before splitting. Otherwise, information from validation or test responses has already influenced which features are retained. When using cross-validation, screening and fitting must be repeated within each training fold; the held-out fold is used to evaluate that fold’s fitted procedure, not to choose its features in advance.
Worked Example: Use Validation Folds Without Selecting on the Test Set
A team compares two feature sets for a KNN regression model. The team uses three validation folds and records fold MSEs from models fitted within the corresponding training folds. Set A contains the established predictors. Set B adds a candidate feature. After selecting a set, the team evaluates that chosen procedure once on a separate test set.
| Validation fold | MSE for A | MSE for B |
|---|---|---|
| 1 | 1.2 | 0.8 |
| 2 | 1.6 | 2.5 |
| 3 | 1.4 | 2.4 |
Step 1: Average the validation losses. For A, the total is \(1.2+1.6+1.4=4.2\), so its mean fold MSE is \(4.2/3=1.4\). For B, the total is \(0.8+2.5+2.4=5.7\), so its mean fold MSE is \(5.7/3=1.9\).
Step 2: Select using validation results. Since \(1.4<1.9\), choose A under this comparison. The first fold alone favored B, but the pre-specified comparison uses all three folds, not the most favorable single result.
Step 3: Evaluate the selected procedure once. Suppose the separate test-set MSE for A is 1.7. Report that value as the test evaluation; do not switch to B because of a later test-set comparison. The final test result is not used to revise the feature choice.
Check. The mean validation MSEs are 1.4 and 1.9, so A is selected; the given test MSE of 1.7 is reported only after selection. The common mistake avoided is choosing or screening features using test responses and then presenting performance on that same test set as an independent check.
A Practical Feature-Selection Sequence
A defensible comparison makes the prediction target and the candidate feature sets explicit before interpreting the validation results. For each candidate, the distance calculation, scaling, \(k\), and loss criterion should be applied consistently. If these choices are all being tuned, they form a collection of candidate procedures; comparing them on validation data is a model-selection step, not a final unbiased evaluation.
Exclude predictors derived from the future response or otherwise unavailable when the prediction will be made.
For example, compare a baseline set with one or more sets that add or remove plausible predictors. Keep the distance and scaling rules clear.
Fit predictions from training observations and use validation loss to compare the candidates. If screening uses responses, perform it using only the applicable training portion.
After selecting the feature set and other modeling choices, evaluate the selected procedure on test data that did not guide those choices.
Feature selection is not a claim that every retained variable has a causal effect. It is a decision about predictive performance for the chosen model, data split, and loss measure. Validation results can vary across splits, especially when the validation sample is small, so small differences should not be treated as conclusive proof that one feature is universally useful or useless. The central KNN insight remains: a predictor’s contribution to distance and its contribution to response prediction are different things.
Check Your Understanding
For each problem, calculate or identify the requested result before checking the answer.
- A query is at \((0,0)\). Two training observations have coordinates A \((0.4,1.5)\) and B \((1.2,0)\). Using squared Euclidean distance, which is nearest with both coordinates, and which is nearest if the second coordinate is removed? (Answer: With both coordinates, B is nearest: \(d^2(A)=0.16+2.25=2.41\), \(d^2(B)=1.44\). With only the first coordinate, A is nearest: \(0.16<1.44\).)
- For three validation cases, a model has observed responses \(8,12,16\) and predictions \(10,10,19\). Find its validation MSE. (Answer: Residuals are \(-2,2,-3\); squared errors sum to \(4+4+9=17\), so MSE is \(17/3\approx5.6667\).)
- Two candidate classifiers have validation results on 20 cases: model P makes 4 errors and model Q makes 6 errors. Which has lower validation misclassification error, and what is the difference in percentage points? (Answer: P has \(4/20=0.20\); Q has \(6/20=0.30\). P is lower by 10 percentage points.)
- A response-based feature-ranking rule is computed once using all observations, after which the data are split into training and validation sets. Why is this procedure contaminated, and what is the correction? (Answer: The ranking has used validation responses before model comparison, leaking information across the split. Compute response-based screening using the training data only; within cross-validation, repeat screening inside each training fold.)
- Two candidate feature sets have fold MSEs \((2,3,1)\) and \((1,4,3)\). Select by mean fold MSE, then state whether a test-set result should be used to reverse the choice. (Answer: The means are \(6/3=2\) and \(8/3\approx2.6667\), so choose the first set. Do not use test results to reverse the choice; reserve the test set for final evaluation.)