Why Repeated Information Matters in KNN
K-nearest-neighbor (KNN) methods decide which training observations are close to a query point by calculating distances across predictor coordinates. This makes the representation of each observation consequential: if two coordinates carry nearly the same information, both can contribute to the distance. A signal represented twice may then have more influence on the neighbor search than a signal represented once.
The effect is not that correlation automatically makes KNN invalid. Rather, correlation can make the chosen distance treat some directions in the predictor space differently from what the analyst intends. The practical question is whether a repeated or correlated signal should count repeatedly when defining a local neighborhood.
The distance formula adds a squared difference for every coordinate. Consequently, if one predictor is entered twice, its difference from the query is squared twice. Standardizing each column to have mean zero and standard deviation one can put predictors on comparable scales, but it does not remove a duplicate or make correlated columns independent. Two identical standardized columns remain identical.
What an Exact Duplicate Does to Distance
Suppose a query and a training point differ by \(a\) on predictor \(X_1\) and by \(b\) on predictor \(X_2\). With one copy of each predictor, their squared Euclidean distance is \(a^2+b^2\). If \(X_1\) is copied into a third coordinate, the squared distance becomes \(a^2+a^2+b^2=2a^2+b^2\). Thus, duplicating \(X_1\) is equivalent, for this calculation, to giving its squared difference weight 2 rather than weight 1.
This change can alter the ordering of training observations, not just the numerical size of all distances. A point that differs mainly on the duplicated predictor may move farther away relative to a point that differs mainly on another predictor. If that changes the nearest neighbors, KNN regression averages different responses, and KNN classification counts different labels.
Worked Example: An Exact Duplicate Changes the Nearest Neighbor
A hypothetical KNN regression model uses two predictors, \(X_1\) and \(X_2\), recorded on a common standardized scale. The query is \(q=(0,0)\). Three training observations have predictor values and responses \(A=(1,0)\), \(y_A=12\); \(B=(0,1.2)\), \(y_B=30\); and \(C=(0.8,0.8)\), \(y_C=20\). Find the \(k=1\) prediction first with the two predictors, then after adding an exact copy of \(X_1\). Use ordinary Euclidean distance.
Method. Calculate each distance from the query and select the observation with the smallest distance. The response of that observation is the \(k=1\) regression prediction.
With one copy of each predictor:
For \(C\), the squared distance is \(0.8^2+0.8^2=0.64+0.64=1.28\), so \(d(q,C)=\sqrt{1.28}\approx1.1314\). The distances are therefore \(1\) for \(A\), approximately \(1.1314\) for \(C\), and \(1.2\) for \(B\). Observation \(A\) is nearest, giving prediction \(\hat y(q)=y_A=12\).
After copying \(X_1\): Each distance now includes the difference on \(X_1\) twice.
For \(B\), the duplicated coordinate remains zero, so \(d(q,B)=\sqrt{0^2+0^2+1.2^2}=1.2\). For \(C\), the squared distance is \(0.8^2+0.8^2+0.8^2=0.64+0.64+0.64=1.92\), so \(d(q,C)=\sqrt{1.92}\approx1.3856\). The new ordering is \(B\), then \(C\), then \(A\). The \(k=1\) prediction changes to \(y_B=30\).
Check: The duplicated coordinate increases the distances to \(A\) and \(C\), which differ from the query on \(X_1\), but leaves \(B\)'s distance unchanged. This explains the changed nearest observation. A common mistake is to calculate \(A\)'s distance as \(\sqrt{1^2+0^2}=1\) even after adding the copied column; the correct calculation includes both copies and gives \(\sqrt{2}\). The example shows why adding a duplicate is not an innocuous way to store the same information twice.
Correlation Changes Which Directions Look Close
An exact duplicate is the clearest case of repeated information. With correlated but nonidentical predictors, the issue is subtler: a pair of coordinates may vary together, so the cloud of observations tends to lie along a particular direction. Ordinary Euclidean distance treats a given squared displacement in either coordinate equally and does not account for the shape of that cloud.
One way to inspect this issue is a covariance-aware distance. For standardized predictors with covariance matrix \(\Sigma\), the Mahalanobis distance from a query \(q\) to a point \(x\) is \(\sqrt{(x-q)^\mathsf{T}\Sigma^{-1}(x-q)}\), when \(\Sigma\) is invertible. It measures displacement relative to the covariance structure: movement along a direction with high variability is treated differently from movement along a direction with low variability. It is a diagnostic and possible alternative, not an automatic correction for every KNN problem.
For two standardized predictors with correlation \(\rho\), the covariance matrix is \(\Sigma=\begin{pmatrix}1&\rho\\ \rho&1\end{pmatrix}\). When \(|\rho|<1\), its inverse is \(\frac{1}{1-\rho^2}\begin{pmatrix}1&-\rho\\-\rho&1\end{pmatrix}\). This makes the contrast between common and unusual directions explicit.
The calculation does not say that every KNN analysis should replace Euclidean distance with Mahalanobis distance. Estimating a covariance matrix can be unstable when the training sample is limited or predictors are strongly dependent. It does provide a precise way to see that equal Euclidean displacements can have different meanings relative to the observed predictor relationships.
Worked Example: Equal Euclidean Scale, Different Correlation Context
Two predictors have been standardized, and their assumed correlation is \(\rho=0.8\). A query is \(q=(0,0)\). Compare training points \(P=(1,1)\) and \(Q=(1.3,0)\) using Euclidean distance and then the covariance-aware distance. State which point each distance regards as closer.
Euclidean calculation: For \(P\), \(d_E(q,P)=\sqrt{1^2+1^2}=\sqrt{2}\approx1.4142\). For \(Q\), \(d_E(q,Q)=\sqrt{1.3^2+0^2}=\sqrt{1.69}=1.3\). Euclidean distance therefore regards \(Q\) as closer.
Covariance-aware calculation: Here \(1-\rho^2=1-0.8^2=0.36\), so the squared distance for a displacement \((a,b)\) is \((a^2-1.6ab+b^2)/0.36\). For \(P\), \(a=b=1\), giving
For \(Q\), \(a=1.3\) and \(b=0\), giving
The covariance-aware distance regards \(P\) as closer. Its displacement is along the common direction of strongly positively correlated predictors. The Euclidean ranking and the covariance-aware ranking differ because they answer different geometric questions. Check: \(P\)'s Euclidean distance exceeds \(Q\)'s, but its covariance-aware distance is smaller, as the calculations show. A common mistake is to assume that standardizing the two predictors resolves their correlation; both have unit variance here, yet \(\rho=0.8\) still affects the covariance-aware comparison.
Assessing the Effect on a Neighborhood
A distance change matters to a KNN prediction only if it changes which observations enter the neighborhood, or changes their relative weights in a distance-weighted method. For ordinary unweighted \(k\)-nearest neighbors, a useful assessment is to compare neighbor identities and then recompute the local average or vote. A distance comparison alone does not establish that the prediction changes.
The following example keeps the target labels fixed while showing how a correlated predictor structure can change a classification result. The covariance-aware calculation uses the stated covariance matrix, assumed known for this illustration. In practice, the matrix would have to be estimated from training data, and its suitability would need to be considered.
Worked Example: Correlation Changes a KNN Classification Vote
A binary KNN classifier uses two standardized predictors with correlation \(\rho=0.8\). Its query is \(q=(0,0)\). Four labeled training points are \(A=(1,1)\), green; \(B=(1.2,0)\), blue; \(C=(-1.05,-1.05)\), green; and \(D=(0,1.4)\), blue. For \(k=3\), compare the majority-vote prediction using Euclidean distance with the prediction using the covariance-aware distance defined above.
Euclidean distances: For \(A\), \(d_E=\sqrt{1^2+1^2}=\sqrt{2}\approx1.4142\). For \(B\), \(d_E=1.2\). For \(C\), \(d_E=\sqrt{1.05^2+1.05^2}=\sqrt{2.205}\approx1.4850\). For \(D\), \(d_E=1.4\). Thus the three nearest are \(B\), \(D\), and \(A\), in that order. Their labels are blue, blue, and green, so the Euclidean \(k=3\) prediction is blue.
Covariance-aware distances: The squared-distance expression is \((a^2-1.6ab+b^2)/0.36\). For \(A=(1,1)\), this is \(0.4/0.36=1.1111\), giving distance approximately \(1.0541\). For \(B=(1.2,0)\), it is \(1.44/0.36=4\), giving distance \(2\). For \(C=(-1.05,-1.05)\), the numerator is \(1.1025-1.6(1.1025)+1.1025=0.441\). Its squared distance is \(0.441/0.36=1.225\), giving distance approximately \(1.1068\). For \(D=(0,1.4)\), the squared distance is \(1.96/0.36=5.4444\), giving distance approximately \(2.3333\).
The three closest under this distance are \(A\), \(C\), and \(B\). Their labels are green, green, and blue, so the covariance-aware \(k=3\) prediction is green. Check: The Euclidean distances rank \(B,D,A,C\), whereas the covariance-aware distances rank \(A,C,B,D\); neither ordering has a tie at the third neighbor. The resulting votes are therefore two blue versus one green, and two green versus one blue, respectively. A common mistake is to compare only the closest point under each metric and infer the \(k=3\) vote from that comparison. The vote requires identifying all three neighbors.
Practical Checks and Common Traps
When predictors are correlated, the main task is to assess whether the distance calculation is giving repeated information more influence than intended. Useful checks include inspecting predictor definitions for duplicate or near-duplicate measurements, comparing neighbor identities before and after accounting for the relationship, and evaluating the resulting predictions on held-out data. The chosen representation should be judged by predictive performance and the intended meaning of a neighbor, not by a rule that correlation must always be removed.
- Standardization is not decorrelation. Scaling columns to comparable ranges does not stop two columns from carrying substantially the same signal.
- Correlation is not duplication. A correlation near one does not mean the columns match exactly for every observation; exact duplicates have a stronger, directly calculable effect on distance.
- A distance change is not automatically a prediction change. Check whether the set of \(k\) neighbors or their weights actually changes.
- Do not assume a covariance-aware metric is always superior. It depends on the data, how the covariance is estimated, and what notion of similarity is appropriate.
A practical response may be to remove a redundant representation, reduce the combined influence of a group of related predictors, or compare a covariance-aware distance with the original metric. These are choices to assess, not universal fixes. In particular, a correlated predictor can still contain useful information, and removing it without checking the effect may discard signal.
Check Your Understanding
For each problem, identify the distance or neighborhood effect before stating the result.
- A query is \((0,0)\). Two training points are \(U=(0.9,0)\) and \(V=(0,1.1)\). Using Euclidean distance, which point is nearer? If the first coordinate is duplicated, which point is nearer? (Answer: \(U\) is nearer originally; \(V\) is nearer after duplication.)
- A query differs from a training point by \(a=0.6\) on \(X_1\) and \(b=0.8\) on \(X_2\). What is the squared Euclidean distance with one copy of each predictor, and with an exact duplicate of \(X_1\)? (Answer: \(1.00\) originally and \(1.36\) with the duplicate.)
- Two standardized predictors have correlation \(0.5\). For a displacement \((1,1)\), calculate the squared covariance-aware distance using \((a^2-2\rho ab+b^2)/(1-\rho^2)\). (Answer: \(4/3\), approximately \(1.3333\).)
- A \(k=3\) classifier has neighbor labels red, red, and blue under one distance, but red, blue, and blue under another. What are the two majority predictions, and does changing distance necessarily change the vote in this case? (Answer: red under the first distance and blue under the second; yes, the vote changes.)
- Two predictors each have mean zero and standard deviation one, but their correlation is \(0.95\). Has standardization removed their dependence? What is the main distance concern? (Answer: No; they remain strongly correlated, so related information may influence the neighborhood more than intended.)