Why Missing Predictors Matter in KNN
K-nearest-neighbor methods choose observations by comparing predictor values. When a predictor is missing, the distance to an observation may be unavailable, or may be calculated using only part of the information. Those choices affect which observations are selected as neighbors and, in turn, can change a regression prediction or classification vote.
There is no universally correct distance rule for missing predictors. Complete-case handling, imputation, and pairwise distance adjustment make different choices about what information is usable. Before applying any of them, predictors should be on comparable scales, for example by standardizing them using training data. The calculations below use predictors already on comparable scales and ordinary Euclidean distance.
The same missingness pattern does not necessarily apply to every training observation. A distance may therefore be based on different predictors for different candidates. This is important: a ranking created from such distances is not always a like-for-like comparison.
Three Ways to Handle Missing Predictors
Complete-case handling excludes training observations that have one or more missing predictors. It produces distances on the same set of predictors for all retained observations, provided the query itself is complete. If a query predictor is missing, complete-case handling alone does not define a distance to any fully observed training row; the query needs a separate strategy, or the analysis cannot proceed under this rule.
Imputation fills in missing predictor values before distances are calculated. For example, a missing numerical predictor might be replaced by its training-set mean or median. The same fitted imputation rule must be applied to the query and training observations as needed. Imputation makes full-coordinate distances possible, but a filled value is an estimate, not an observed value. In particular, mean imputation can place an observation at a value that makes it appear more typical than its observed information warrants.
Pairwise distance adjustment calculates a distance using only coordinates observed for both the query and the candidate, then rescales it to account for the number of coordinates used. With equally weighted, comparable-scale predictors, one common rule is:
Here \(p\) is the total number of predictors, and \(|O_i|\) is the number observed for both observations. If no predictor is jointly observed, the distance is undefined. A distance-weighted version can use predictor weights, but its scaling must be chosen consistently with those weights; the displayed rule is specifically for equally weighted coordinates.
Why the Pairwise Rescaling Can Be Reasonable
The multiplier \(p/|O_i|\) is intended to put a partial-coordinate squared distance on the scale of a full-coordinate squared distance. Its justification relies on a strong condition: the coordinates observed for a pair must act like a representative sample of all \(p\) coordinates. In particular, the rule is most defensible when each coordinate is equally likely to be observed, independently of the coordinate differences being measured.
Write \(a_j=(x_j-x_{ij})^2\) for each coordinate. The full squared distance is \(\sum_{j=1}^{p}a_j\).
If \(m\) of the \(p\) coordinates are observed, and the subset is equally likely to include each coordinate, the average observed squared difference estimates the average across all coordinates: \(\frac{1}{m}\sum_{j\in O_i}a_j\) estimates \(\frac{1}{p}\sum_{j=1}^{p}a_j\).
Multiplying that average by \(p\) gives \(p\left(\frac{1}{m}\sum_{j\in O_i}a_j\right)=\frac{p}{m}\sum_{j\in O_i}a_j\), which motivates the adjusted squared distance.
This argument concerns the squared distance, before taking its square root. It does not guarantee that the adjusted distance equals the true full distance for an individual pair. If some predictors are more likely to be missing, or their missingness depends on their values or on the differences between observations, the observed subset may not represent the full set. The adjustment can then misstate distances. Even under representative missingness, different candidates may have different observed subsets, so their adjusted distances have additional uncertainty.
Worked Example: A Missing Training Predictor Changes the Ranking
A KNN regression model uses two equally scaled predictors. The query is \(x=(0,0)\), and the training observations are \(A=(2,\text{missing})\), \(B=(0,2.5)\), and \(C=(3,0)\). Compare complete-case handling, mean imputation, and pairwise distance adjustment. Identify the nearest usable observation under each method.
Method and complete cases. Observation \(A\) is excluded because a predictor is missing. For \(B\), the distance is \(\sqrt{(0-0)^2+(0-2.5)^2}=\sqrt{6.25}=2.5\). For \(C\), it is \(\sqrt{(0-3)^2+(0-0)^2}=\sqrt{9}=3\). Thus \(B\) is nearest among the complete cases.
Mean imputation. The observed second-predictor values in the training data are 2.5 and 0. Their mean is \((2.5+0)/2=1.25\). Impute \(A=(2,1.25)\). Its distance is \(\sqrt{(0-2)^2+(0-1.25)^2}=\sqrt{4+1.5625}=\sqrt{5.5625}\approx2.358\). The other distances remain \(d_B=2.5\) and \(d_C=3\). Therefore imputation makes \(A\) nearest.
Pairwise adjustment. For \(A\), only the first predictor is observed, so \(|O_A|=1\), \(p=2\), and the observed squared difference is \((0-2)^2=4\). Hence \(d_A^{\text{pair}}=\sqrt{(2/1)(4)}=\sqrt{8}\approx2.828\). Observations \(B\) and \(C\) are complete, so their adjusted distances are 2.5 and 3. Thus \(B\) is nearest under pairwise adjustment.
Check and mistake avoided. The three nearest-neighbor outcomes are not the same: complete-case handling selects \(B\), mean imputation selects \(A\), and pairwise adjustment selects \(B\). The common mistake is to treat \(A\)'s observed difference of 2 as its full distance without either imputing or adjusting; that would report 2 and would not match either of the two stated rules.
Different Overlap Can Reverse Rankings
Pairwise adjustment does not merely fill gaps in a fixed ranking. Since each candidate can have a different number of observed coordinates, the rescaling can change the order. The next example contrasts the unadjusted distance on the observed coordinates with the adjusted distance.
Worked Example: Unequal Overlap Reorders Three Candidates
A query has values \(x=(0,0,0)\) on three equally scaled predictors. Candidate \(A=(1,\text{missing},\text{missing})\), candidate \(B=(0.8,0.8,\text{missing})\), and candidate \(C=(1.5,0,0)\). Calculate each candidate's observed-coordinate distance and pairwise-adjusted distance, then rank the candidates under both rules.
Observed-coordinate distances. For \(A\), only the first coordinate is shared, so the distance is \(\sqrt{1^2}=1\). For \(B\), two coordinates are shared, so the distance is \(\sqrt{0.8^2+0.8^2}=\sqrt{1.28}\approx1.131\). Candidate \(C\) is complete, so its distance is \(\sqrt{1.5^2+0^2+0^2}=1.5\). Ranking by these unadjusted values gives \(A,B,C\).
Pairwise-adjusted distances. Here \(p=3\). Candidate \(A\) has one shared coordinate, so \(d_A^{\text{pair}}=\sqrt{(3/1)(1)}=\sqrt{3}\approx1.732\). Candidate \(B\) has two, so \(d_B^{\text{pair}}=\sqrt{(3/2)(1.28)}=\sqrt{1.92}\approx1.386\). Candidate \(C\) has all three, so \(d_C^{\text{pair}}=\sqrt{(3/3)(2.25)}=1.5\). The adjusted ranking is \(B,C,A\).
Check and mistake avoided. \(A\) is closest under the unadjusted observed-coordinate rule, but farthest under the stated adjustment. The rescaling does not guarantee a better ranking; it changes the scale to account for the amount of overlap. The mistake to avoid is comparing partial distances as if they used the same number of coordinates.
Missing Predictors in the Query
Missingness in a query needs particular care. Complete-case handling of the training data cannot by itself supply the missing query value. Imputing that value creates a complete query, while pairwise adjustment compares each candidate with the query only on their jointly observed predictors. These approaches can select different neighbors, even when all training observations are complete.
Worked Example: Imputing a Query Versus Comparing Shared Predictors
A KNN regression model uses two equally scaled predictors. A new case has query values \(x=(0,\text{missing})\). Three complete training observations are \(A=(0.5,0)\) with response 10, \(B=(2,10)\) with response 40, and \(C=(3,10)\) with response 60. Use \(k=1\). Compare mean imputation of the query's missing predictor with pairwise distance adjustment, and calculate the resulting regression prediction under each method.
Mean-imputed query. The training-set mean of the second predictor is \((0+10+10)/3=20/3\approx6.667\). Impute the query as \((0,20/3)\). For \(A\), \(d_A=\sqrt{(0-0.5)^2+(20/3-0)^2}=\sqrt{0.25+400/9}=\sqrt{1609/36}\approx6.685\). For \(B\), \(d_B=\sqrt{(0-2)^2+(20/3-10)^2}=\sqrt{4+100/9}=\sqrt{136/9}\approx3.888\). For \(C\), \(d_C=\sqrt{(0-3)^2+(20/3-10)^2}=\sqrt{9+100/9}=\sqrt{181/9}\approx4.484\). Candidate \(B\) is nearest, so the \(k=1\) regression prediction is its response, 40.
Pairwise adjustment. The query and each training observation share only the first predictor, so \(|O_i|=1\) and \(p=2\) for all three candidates. The adjusted distances are \(d_A^{\text{pair}}=\sqrt{2(0.5^2)}=\sqrt{0.5}\approx0.707\), \(d_B^{\text{pair}}=\sqrt{2(2^2)}=\sqrt{8}\approx2.828\), and \(d_C^{\text{pair}}=\sqrt{2(3^2)}=\sqrt{18}\approx4.243\). Candidate \(A\) is nearest, so the prediction is 10.
Check and mistake avoided. Imputation gives a prediction of 40, whereas pairwise adjustment gives 10. The arithmetic uses the same training responses, but the methods define distance differently. The mistake is to calculate a full-coordinate distance for a query that has no observed value on the second predictor without first stating how that missing value is handled.
Choosing and Reporting a Rule
The method should be selected as part of the modeling procedure, not chosen separately for each test observation after seeing which answer is preferred. Imputation values and any scaling parameters should be estimated from the training data, then applied consistently to validation or test observations. Otherwise, information from held-out observations can influence the distance calculation.
| Method | What it does | Main consequence |
|---|---|---|
| Complete-case handling | Excludes training observations with missing predictors; requires a complete query for ordinary full-coordinate distances. | Uses a consistent predictor set for retained cases, but discards incomplete observations. |
| Imputation | Fills missing values, such as with a training-set mean or median, before distance calculation. | Allows full-coordinate comparisons, but distances depend on the chosen imputed values. |
| Pairwise adjustment | Uses shared observed predictors and rescales for the number used. | Retains partial information, but rankings may compare distances based on different predictor subsets. |
A sound analysis makes the rule explicit, checks how many coordinates contribute to each distance, and considers whether the missingness pattern makes the rule defensible. If missingness is systematic, pairwise rescaling may not repair the resulting bias. If imputation is used, the imputation rule can affect the neighborhood even when no observed predictor values change. In all cases, examine whether the selected neighbors and predictions are sensitive to the chosen handling method.
Check Your Understanding
For each problem, follow the stated distance rule and compare only the candidates for which that rule defines a distance.
- A query has four equally scaled predictors. A candidate shares two coordinates with it, and the squared differences on those coordinates are 1 and 4. Calculate the pairwise-adjusted distance. (Answer: \(\sqrt{10}\approx3.1623\))
- A complete query is \(x=(0,0)\). Candidate \(A=(1,\text{missing})\), \(B=(2,1)\), and \(C=(0,3)\). Use the mean of observed second-predictor values to impute \(A\), and separately use pairwise adjustment. Which candidate is nearest under each rule? (Answer: The imputation mean is \((1+3)/2=2\); imputed distances are \(d_A=\sqrt{5}\), \(d_B=\sqrt{5}\), and \(d_C=3\), so \(A\) and \(B\) tie. Pairwise distances are \(\sqrt{2}\), \(\sqrt{5}\), and \(3\), so \(A\) is nearest.)
- A query is \((0,\text{missing})\). Two complete training cases have predictors \(A=(0.5,2)\) and \(B=(1.5,6)\). The training-set mean of the second predictor is used to impute the query. Which case is nearest after imputation, and which is nearest under pairwise adjustment? (Answer: The imputed query is \((0,4)\); \(d_A=\sqrt{4.25}\approx2.0616\) and \(d_B=\sqrt{6.25}=2.5\), so \(A\) is nearest. Pairwise distances are \(\sqrt{2(0.5^2)}\approx0.7071\) and \(\sqrt{2(1.5^2)}\approx2.1213\), so \(A\) is also nearest.)
- For the pairwise-adjusted squared distance, explain why multiplying the observed sum of squared differences by \(p/m\), rather than by \(m/p\), gives the proposed scale when \(m\) of \(p\) coordinates are observed representatively. (Answer: The observed average squared difference is the observed sum divided by \(m\); estimating the full total requires multiplying that average by \(p\), giving \((p/m)\) times the observed sum.)
- A candidate has observed squared differences summing to 2 on one shared coordinate, while another has observed squared differences summing to 3 on three shared coordinates. There are three equally scaled predictors in total. Rank them by unadjusted observed distance and by pairwise-adjusted distance. (Answer: Unadjusted distances are \(\sqrt{2}\) and \(\sqrt{3}\), so the first candidate is nearer. Adjusted distances are \(\sqrt{6}\) and \(\sqrt{3}\), so the second candidate is nearer.)