Why Feature Scaling Changes Nearest Neighbors

Why Units Matter for Nearest Neighbors

A nearest-neighbor method compares observations by measuring distances between their predictor values. If one predictor is measured in dollars and another in years, their numerical scales can be very different. A difference of 500 dollars may overwhelm a difference of 5 years in an unscaled Euclidean distance, even when the dollar difference is not more important for the modeling task. The resulting neighbors can therefore depend on the units in which the predictors happen to be recorded.

Feature scaling changes the contributions of numeric predictors to a distance. Standardization is a common approach: subtract each predictor’s training-set mean and divide by its training-set standard deviation. The transformation does not guarantee that every predictor is equally useful, but it gives predictors measured on different scales a more comparable contribution. This treatment is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: For numeric predictor \(j\), let \(\bar{x}_j\) and \(s_j\) be its mean and sample standard deviation calculated from the training data. The standardized value of observation \(i\) is \(z_{ij}=(x_{ij}-\bar{x}_j)/s_j\), provided \(s_j>0\). A standardized Euclidean distance between observations \(a\) and \(b\), using \(p\) numeric predictors, is \(d(a,b)=\sqrt{\sum_{j=1}^{p}(z_{aj}-z_{bj})^2}\).

The mean and standard deviation used for a predictor must be estimated from the training data. Apply those same estimates to transform any validation, test, or future observations. Do not estimate new scaling parameters from the data being evaluated: doing so lets information from that data influence preprocessing. When a predictor has zero training standard deviation, the formula cannot be applied because it would divide by zero; that predictor is constant in the training data and must be handled separately.

How Standardization Changes a Distance

For a predictor \(j\), subtracting the same mean from two observations does not change their difference:

z_{aj}-z_{bj} =\frac{x_{aj}-\bar{x}_j}{s_j}-\frac{x_{bj}-\bar{x}_j}{s_j} =\frac{x_{aj}-x_{bj}}{s_j}.

This gives a useful expression for standardized Euclidean distance:

d(a,b)=\sqrt{\sum_{j=1}^{p}\left(\frac{x_{aj}-x_{bj}}{s_j}\right)^2}.

Each raw difference is measured in standard-deviation units before it is squared and added. A difference of one training standard deviation contributes \(1\) to the squared distance, whether the predictor is measured in years, dollars, or another numeric unit. The mean sets the transformed coordinate origin, but it cancels from the distance between two observations transformed with the same mean. The standard deviation changes the relative scale of each predictor’s contribution.

Key takeaway: With unscaled Euclidean distance, predictor units determine the numerical size of differences. With standardized Euclidean distance, each numeric difference is divided by that predictor’s training-set standard deviation. This is a scale adjustment, not evidence that the predictors have equal predictive value.

Worked Examples

Worked Example: A Large Unit Can Dominate the Distance

A hypothetical insurer uses age in years and annual mileage in miles as numeric predictors for a nearest-neighbor analysis. A query has raw values \((40,12000)\). Two candidate training observations are A \((42,11000)\) and B \((35,12500)\). The training standard deviations are \(s_1=5\) years and \(s_2=2500\) miles. Compare the raw and standardized Euclidean distances. Which candidate is nearer under each calculation?

Method. First calculate ordinary Euclidean distances from the raw values. Then calculate standardized distances by dividing each predictor difference by its stated training standard deviation.

For candidate A, the raw differences from the query are \(40-42=-2\) years and \(12000-11000=1000\) miles. Therefore,

d_{\text{raw},A} =\sqrt{(-2)^2+1000^2} =\sqrt{4+1000000} =\sqrt{1000004} \approx 1000.0020.

For candidate B, the raw differences are \(40-35=5\) years and \(12000-12500=-500\) miles. Thus,

d_{\text{raw},B} =\sqrt{5^2+(-500)^2} =\sqrt{25+250000} =\sqrt{250025} \approx 500.0250.

On the raw scale, B is nearer. But these calculations combine years and miles directly. For the standardized distances, divide each difference by its corresponding training standard deviation. For A, the standardized differences are \(-2/5=-0.4\) and \(1000/2500=0.4\), so

d_{\text{std},A} =\sqrt{(-0.4)^2+(0.4)^2} =\sqrt{0.16+0.16} =\sqrt{0.32} \approx 0.5657.

For B, the standardized differences are \(5/5=1\) and \(-500/2500=-0.2\). Therefore,

d_{\text{std},B} =\sqrt{1^2+(-0.2)^2} =\sqrt{1+0.04} =\sqrt{1.04} \approx 1.0198.

After standardization, A is nearer. The ordering changed because the raw mileage differences dominated the unscaled calculation. Common mistake avoided: do not describe raw values as standardized, or divide values that have already been standardized by the standard deviations a second time. Here the listed predictor values are raw, so division by the stated standard deviations is required for the standardized distances. Distances are rounded to four decimal places.

Worked Example: Calculate Training Scaling Parameters

A training sample contains three observations with paired numeric predictors \((2,10)\), \((4,20)\), and \((6,30)\). A validation query has raw values \((5,24)\). Two candidate training observations are A \((4,20)\) and B \((6,30)\). Use the training sample means and sample standard deviations to standardize the predictors, then identify the nearer candidate.

Method. Calculate the sample mean and sample standard deviation for each predictor from the three training observations only. For the first predictor, the mean is \((2+4+6)/3=4\). The sum of squared deviations is \((2-4)^2+(4-4)^2+(6-4)^2=4+0+4=8\), so the sample variance is \(8/(3-1)=4\), and the sample standard deviation is \(s_1=2\).

For the second predictor, the mean is \((10+20+30)/3=20\). Its sum of squared deviations is \((10-20)^2+(20-20)^2+(30-20)^2=100+0+100=200\). The sample variance is \(200/2=100\), so \(s_2=10\).

The validation query standardizes to \(((5-4)/2,(24-20)/10)=(0.5,0.4)\). Candidate A standardizes to \(((4-4)/2,(20-20)/10)=(0,0)\), and candidate B standardizes to \(((6-4)/2,(30-20)/10)=(1,1)\).

The distance from the query to A is

d_A=\sqrt{(0.5-0)^2+(0.4-0)^2} =\sqrt{0.25+0.16} =\sqrt{0.41} \approx 0.6403.

The distance from the query to B is

d_B=\sqrt{(0.5-1)^2+(0.4-1)^2} =\sqrt{(-0.5)^2+(-0.6)^2} =\sqrt{0.25+0.36} =\sqrt{0.61} \approx 0.7810.

Candidate A is nearer because \(0.6403<0.7810\). All scaling parameters came from training data; the validation query was transformed using those same parameters. Common mistake avoided: estimating a separate mean or standard deviation from the validation observation would make the preprocessing depend on the data being evaluated. Distances are rounded to four decimal places.

Training-Only Scaling and Information Leakage

Scaling is part of the model-development procedure. If validation or test observations are used to estimate the means or standard deviations, information from those observations enters the fitted preprocessing step. This is a form of leakage even when the response variable is not used. In a validation procedure, fit the scaling parameters separately within each training portion and use them to transform that portion’s validation observations. After choosing a procedure, fit the final preprocessing parameters on the data designated for final model fitting, then apply them unchanged to the held-out test data.

For Euclidean distances, there is a useful qualification: a common shift of all observations on a predictor cancels from pairwise differences. Thus, changing only the estimated mean does not change standardized pairwise distances, provided the same mean is used for both observations. Changing the estimated standard deviation can change those distances, because it changes that predictor’s weight. Training-only preprocessing remains important: it is a consistent leakage-safe rule, and the estimated scales can directly affect the neighbor geometry.

Worked Example: Detect Scaling Leakage

The training observations are \((0,0)\), \((2,10)\), and \((4,20)\). A validation query is \((4,10)\). The candidate training observations are A \((2,0)\) and B \((0,20)\). Calculate standardized distances using training-only sample standard deviations. Then show how including the validation query when estimating the standard deviations changes the distances.

Method. For predictor 1, the training mean is \(2\), and the sample standard deviation is \(2\): the squared deviations sum to \(8\), which divided by \(3-1\) gives variance \(4\). For predictor 2, the training mean is \(10\), and the squared deviations sum to \(200\), giving variance \(100\) and sample standard deviation \(10\).

With training-only scaling, the query standardizes to \((1,0)\), A to \((0,-1)\), and B to \((-1,1)\). Therefore,

d_A=\sqrt{(1-0)^2+(0-(-1))^2} =\sqrt{2}\approx1.4142, \qquad d_B=\sqrt{(1-(-1))^2+(0-1)^2} =\sqrt{5}\approx2.2361.

A is nearer under the valid training-only calculation. Now consider the incorrect procedure of adding the validation query to the data used to estimate the standard deviations. For predictor 1, the four values are \(0,2,4,4\), their mean is \(2.5\), and their squared deviations sum to \(11\). The resulting sample variance is \(11/3\), so \(s_1^2=11/3\). For predictor 2, the four values are \(0,10,20,10\), their mean is \(10\), and their squared deviations sum to \(200\), so \(s_2^2=200/3\).

For query-to-A differences \((2,10)\), the incorrectly scaled squared distance is

d_{A,\text{leak}}^2 =\frac{2^2}{11/3}+\frac{10^2}{200/3} =\frac{12}{11}+\frac{3}{2} =\frac{57}{22}, \qquad d_{A,\text{leak}}=\sqrt{\frac{57}{22}}\approx1.6096.

For query-to-B differences \((4,-10)\), it is

d_{B,\text{leak}}^2 =\frac{4^2}{11/3}+\frac{(-10)^2}{200/3} =\frac{48}{11}+\frac{3}{2} =\frac{129}{22}, \qquad d_{B,\text{leak}}=\sqrt{\frac{129}{22}}\approx2.4215.

The ranking happens to remain A then B in this example, but both distances have changed because the validation data changed the estimated standard deviations. The fact that the answer did not change here does not make the procedure valid: in another data set, the altered relative scales could change the nearest observation. Common mistake avoided: do not use validation or test observations to estimate preprocessing parameters merely because this example’s nearest candidate stayed the same. Distances are rounded to four decimal places.

Practical Checks and Common Traps

Before using standardized distances, check what the supplied values represent. If the problem gives raw predictors and training standard deviations, divide raw pairwise differences by those standard deviations. If it gives already-standardized coordinates, calculate ordinary Euclidean distances between those coordinates; do not standardize them again. Keep the transformation consistent for every training observation and query.

Mixing units in a raw distance. A large numerical scale can dominate, even if that predictor is not substantively more important.
Using the wrong standard deviation. The denominator is the standard deviation for that predictor, not a standard deviation shared across all predictors unless explicitly specified.
Scaling on the whole data set before splitting. Estimate preprocessing parameters from the training portion only.
Ignoring a zero standard deviation. A constant training predictor cannot be standardized by division; it contributes no observed variation in that training data.
Interpreting standardization as feature selection. Standardization changes measurement scale. It does not determine whether a predictor is relevant or whether equal standardized contributions are desirable.

Standardization is especially useful when numeric predictors use different units or have very different spreads. It is not automatically appropriate in every application: a deliberately meaningful scale or a justified feature weighting may call for a different distance construction. The essential exam skill is to identify the scale being used, apply its formula consistently, and ensure preprocessing is fitted without access to validation or test observations.

Check Your Understanding

For each problem, use the stated data and distinguish raw values from standardized values.

A raw query is \((12,400)\). Candidate A is \((14,300)\), and candidate B is \((10,450)\). The training standard deviations are \(2\) and \(50\), respectively. Calculate both raw distances and both standardized distances, and identify the nearer candidate under each metric. (Answer: Raw \(d_A=\sqrt{10004}\approx100.0200\), \(d_B=\sqrt{2504}\approx50.0400\), so B is nearer; standardized \(d_A=\sqrt{5}\approx2.2361\), \(d_B=\sqrt{2}\approx1.4142\), so B is nearer.)
A training sample has predictor values \(1,3,5\) for its first feature and \(10,20,30\) for its second. A new observation is \((7,25)\). Using sample standard deviations, calculate the training means, standard deviations, and standardized new-observation coordinates. (Answer: Means \((3,20)\), sample standard deviations \((2,10)\), standardized coordinates \((2,0.5)\).)
Training rows are \((2,10),(4,20),(6,30)\), and a raw query is \((5,24)\). Candidate A is \((4,20)\), and B is \((6,30)\). Using training-only sample standard deviations, find the two standardized distances and the nearer candidate. (Answer: \(d_A=\sqrt{0.41}\approx0.6403\), \(d_B=\sqrt{0.61}\approx0.7810\); A is nearer.)
A training predictor has values \(2,4,6\), and a validation observation has value \(10\). Calculate the training mean and sample standard deviation, then standardize the validation value using those training estimates. Why must the validation value not be used to refit the estimates? (Answer: Training mean \(4\), sample standard deviation \(2\), validation standardized value \(3\); using validation data to refit preprocessing leaks information from the evaluation data.)
Two observations have raw predictor differences \((3,12)\), and the training standard deviations are \((3,4)\). Calculate their standardized Euclidean distance. If both raw observations are shifted by the same amount on each predictor, does the distance change? (Answer: \(d=\sqrt{(3/3)^2+(12/4)^2}=\sqrt{10}\approx3.1623\); no, a common shift cancels from pairwise differences.)

What You'll Learn