Why Predictor Units Matter
A distance calculation treats each coordinate as a numerical contribution. If one predictor is recorded in dollars and another in years, the dollar differences can dominate the distance even when that is not the intended comparison. Principal components analysis (PCA) and k-means clustering also depend on numerical scales, but in different ways. A tree-based method, by contrast, generally compares possible splits in the ordering of a predictor, not its numerical units.
The practical question is not whether scaling is always required. It is whether the method’s mathematical objective gives the intended weight to each predictor. Scaling can change a model comparison by changing that objective. It does not create new information in the observations, and it cannot be chosen by looking at a held-out test set.
For any preprocessing operation that estimates quantities from data, calculate those quantities using the training portion of the data, then apply the same transformation to validation or future observations. For standardization, a new observation \(x_j\) is transformed using the training mean and standard deviation, not a mean or standard deviation recalculated from the new observations. This keeps the transformation consistent and avoids using information from data that should be held out.
Distance-Based Methods: Scaling Changes the Geometry
For two observations with \(p\) predictors, Euclidean distance is \(d(\mathbf{x},\mathbf{x}')=\sqrt{\sum_{j=1}^{p}(x_j-x_j')^2}\). A nearest-neighbor method uses such distances to decide which training observations are closest to a new case. If predictor \(j\) is multiplied by \(c_j\), its squared difference contributes \(c_j^2(x_j-x_j')^2\). Thus, scaling is equivalent to changing the relative weights of the predictors in the distance calculation.
Worked Example: Which Risk Record Is Nearest?
A hypothetical insurer uses age in years and annual benefit in dollars to find a nearby training record for a new policyholder. The query is age 40 with an annual benefit of $20,000. Three training records are A: (42, $21,000), B: (41, $27,000), and C: (50, $20,500). First compare raw Euclidean distances. Then compare distances after standardizing with training-set means of 44 years and $22,000, and training-set standard deviations of 5 years and $5,000. These summary values are assumed to have been computed from the training data.
Raw distances. For A, the age difference is \(42-40=2\), and the benefit difference is $1,000. Therefore, \(d_A=\sqrt{2^2+1000^2}=\sqrt{1{,}000{,}004}\approx1000.002\). For B, the differences are 1 year and $7,000, so \(d_B=\sqrt{1^2+7000^2}=\sqrt{49{,}000{,}001}\approx7000.000\). For C, the differences are 10 years and $500, so \(d_C=\sqrt{10^2+500^2}=\sqrt{250{,}100}\approx500.100\). The raw-distance ranking is C, A, B; C is the nearest record.
Standardized distances. The query transforms to \(((40-44)/5,(20{,}000-22{,}000)/5{,}000)=(-0.8,-0.4)\). Record A transforms to \((-0.4,-0.2)\), giving differences \((0.4,0.2)\) and distance \(\sqrt{0.4^2+0.2^2}=\sqrt{0.20}\approx0.4472\). Record B transforms to \((-0.6,1.0)\), giving differences \((0.2,1.4)\) and distance \(\sqrt{0.2^2+1.4^2}=\sqrt{2.00}\approx1.4142\). Record C transforms to \((1.2,-0.3)\), giving differences \((2.0,0.1)\) and distance \(\sqrt{2.0^2+0.1^2}=\sqrt{4.01}\approx2.0025\). The standardized ranking is A, B, C; now A is nearest.
Check. The nearest record changes because raw distances are dominated by dollar differences, while standardization measures differences relative to the training-set spread of each predictor. The standardization does not establish that these are the right weights for a particular decision; it makes the weighting choice explicit.
Method and mistake avoided. The method is to calculate distances in both coordinate systems and compare rankings. A common mistake is to divide the dollar difference by the age standard deviation, or to use a new observation to recalculate the standard deviations. Each predictor must use its own training-set scale.
More generally, standardized Euclidean distance has squared form \(\sum_j((x_j-x_j')/s_j)^2\). This gives a one-standard-deviation difference the same squared contribution for each predictor. That may be a useful default, but it is not automatically optimal: predictors can have different relevance, and correlated predictors can contribute overlapping information. The key is to understand which distance the chosen method is using.
PCA: Covariance Depends on Units
PCA finds directions of high variance. If the predictors are centered and have sample covariance matrix \(S\), the first principal component is the direction that maximizes the variance of a unit-length linear combination. Rescaling the variables changes \(S\), so it can change the maximizing direction. In covariance-based PCA, a variable with a numerically large variance can dominate even when that variance mostly reflects the measurement unit.
Worked Example: A Change of Units Rotates the First Component
Suppose two centered training predictors, \(Z_1\) and \(Z_2\), each have sample variance 1 and sample correlation 0.6. Their covariance matrix is therefore \(\begin{pmatrix}1&0.6\\0.6&1\end{pmatrix}\). Now record the first predictor in units ten times as large, so \(X_1=10Z_1\), while \(X_2=Z_2\). Find the first-component direction before and after the unit change.
Before rescaling. The covariance matrix has equal diagonal entries and correlation 0.6. Its eigenvalues are \(1+0.6=1.6\) and \(1-0.6=0.4\). The largest eigenvalue, 1.6, has unit eigenvector \((1,1)/\sqrt{2}\). Thus the first component gives equal coefficients to the two standardized coordinates.
After rescaling. Multiplying the first coordinate by 10 multiplies its variance by 100 and its covariance with the second coordinate by 10. The new covariance matrix is \(\begin{pmatrix}100&6\\6&1\end{pmatrix}\). Its largest eigenvalue is \(\lambda_1=(101+\sqrt{(100-1)^2+4(6^2)})/2=(101+\sqrt{9945})/2\approx100.3623\). For an eigenvector \((a,b)\), the first row of the eigenvector equation gives \(100a+6b=\lambda_1a\), so \(b/a=(\lambda_1-100)/6\approx0.06038\). Normalizing gives a direction approximately \((0.9982,0.0603)\). The first component is now almost entirely the first predictor.
Justification. If \(D\) is the diagonal matrix of positive rescaling factors, the transformed covariance matrix is \(DSD\): each covariance is multiplied by the scale factor for each of its two variables. The eigenvectors of \(DSD\) need not match those of \(S\). If instead each variable is standardized to variance 1, PCA uses the correlation matrix; in this example its first direction remains \((1,1)/\sqrt{2}\), because the correlation remains 0.6.
Check. The underlying association did not change: the correlation is still 0.6. The raw-scale PCA direction changed because the first variable’s variance became 100 times as large. Standardizing changes the question from finding high variance in the original units to finding directions based on relative, unit-free variation.
Method and mistake avoided. The method is to transform the covariance matrix and then identify its leading eigenvector. A common mistake is to assume PCA directions are unaffected by changing units. That is generally false for covariance-based PCA; correlation-based PCA removes the effect of positive linear rescaling, provided each variable has nonzero variance.
Clustering: The Objective Also Uses Distances
For k-means clustering, a partition is chosen to minimize the total within-cluster sum of squared Euclidean distances to cluster means. Consequently, rescaling a predictor changes the objective itself. A partition that is best in one set of units need not be best after a feature is multiplied by a different positive factor. This is not evidence that one partition is intrinsically correct; it shows that the chosen geometry gives different relative importance to the coordinates.
Worked Example: A Unit Change Alters a Two-Cluster Solution
Consider three hypothetical accounts represented by two numeric features: A=(0,0), B=(3,0), and C=(0,2). For \(k=2\), exactly two points form one cluster and the remaining point forms the other. Find the partition minimizing the k-means within-cluster sum of squares, first as recorded and then after the second feature is multiplied by 10.
Original units. If two points at distance \(d\) form a cluster, their mean lies halfway between them, so each point is \(d/2\) from the mean. Their total squared distance to the mean is \(2(d/2)^2=d^2/2\). The pairwise distances are \(d_{AB}=3\), \(d_{AC}=2\), and \(d_{BC}=\sqrt{3^2+2^2}=\sqrt{13}\). The corresponding two-point cluster costs are \(9/2=4.5\), \(4/2=2\), and \(13/2=6.5\). The smallest is 2, so the optimal partition is \(\{A,C\}\) and \(\{B\}\).
Second feature multiplied by 10. The coordinates become A=(0,0), B=(3,0), and C=(0,20). Now \(d_{AB}=3\), \(d_{AC}=20\), and \(d_{BC}=\sqrt{3^2+20^2}=\sqrt{409}\). The pair costs are \(3^2/2=4.5\), \(20^2/2=200\), and \(409/2=204.5\). The smallest is 4.5, so the optimal partition becomes \(\{A,B\}\) and \(\{C\}\).
Check. The chosen grouping changed when the second feature’s unit changed. The arithmetic verifies both optima by comparing all three possible pairs. In a real analysis, the scaling rule should reflect the intended notion of similarity, rather than being selected because it produces a preferred cluster.
Method and mistake avoided. The method is to compute the k-means objective for each possible two-point cluster. A common mistake is to say that a unit conversion cannot affect clustering because the observations themselves have not changed; the objective uses squared distances, so the conversion changes its values and potentially its minimizer.
Why Tree Splits Are Relatively Scale-Invariant
A standard decision tree considers splits of one predictor at a time. For a numeric predictor \(x\), a split at threshold \(t\) sends observations with \(x\leq t\) to one child and those with \(x>t\) to the other. If \(x^*=cx\) for \(c>0\), the same division of observations is obtained at threshold \(t^*=ct\), because \(x\leq t\) exactly when \(cx\leq ct\). Thus the possible partitions of the training cases are unchanged.
Worked Example: Rescaling Does Not Change a Tree Partition
A hypothetical tree uses one predictor with observed training values 1, 2, and 4, and corresponding outcomes 10, 11, and 30. A candidate split at 3 places the first two observations in one leaf and the third in another. Compare the partition and squared-error split cost after recording the predictor in units multiplied by 100.
Original scale. The split \(x\leq3\) places outcomes 10 and 11 together, with mean \((10+11)/2=10.5\), and outcome 30 in the other leaf. The within-leaf sum of squared errors is \((10-10.5)^2+(11-10.5)^2+(30-30)^2=0.25+0.25+0=0.5\).
Rescaled predictor. The predictor values become 100, 200, and 400. A threshold of 300 gives the same two leaves. Their outcome means and squared-error cost are unchanged, so the cost remains 0.5.
Check. The threshold value changes from 3 to 300, but the assigned cases and the split criterion do not. With a positive scale factor, the ordering of predictor values is preserved, so a tree search over all candidate partitions has the same candidate case divisions.
Method and mistake avoided. The method is to map each threshold to the corresponding threshold in the new units. A common mistake is to conclude that every modeling method is unaffected by rescaling because trees often are. This result concerns ordinary axis-aligned split searches and positive rescaling; it does not imply invariance for every transformation, implementation constraint, or model family.
Tree invariance is relative, not absolute. Multiplying a predictor by a negative constant reverses its order, but the same partition can still be represented by reversing the inequality. Nonlinear transformations, restricted sets of candidate cut points, rounding, or implementation-specific limits can alter the available splits. Also, scaling may matter elsewhere in a pipeline even when the tree’s ideal split partitions are unchanged—for example, if another method or a distance-based step uses the same predictors.
A Practical Comparison
| Method | What scaling changes | Practical implication |
|---|---|---|
| Nearest neighbors | The distance contribution of each predictor | Choose units or weights that define meaningful proximity |
| Covariance-based PCA | The covariance matrix and usually its principal directions | Check whether raw variance or relative variation is intended |
| K-means | The squared-distance objective and potentially the partition | Assess whether distances represent the intended similarity |
| Axis-aligned trees | Threshold numbers, but not the partitions under positive rescaling | Scaling is usually unnecessary solely to preserve tree splits |
These distinctions help interpret a model comparison. If scaling changes nearest neighbors, PCA directions, or clusters, the methods are responding to changed geometry, not to changed outcomes. If tree partitions remain the same, that does not mean the tree and the distance-based methods are comparing predictors on an equal basis; their objectives differ. For a fair predictive comparison, preprocessing must be applied consistently within the training process, and any choice among transformations must be evaluated without using test data to make that choice.
Check Your Understanding
For each problem, identify the scaling effect and verify the numerical comparison.
- A query has predictors (2 years, $10,000). Candidate R differs by (1 year, $2,000), and candidate S differs by (3 years, $500). Which is closer under raw Euclidean distance? (Answer: S; \(d_R=\sqrt{4{,}000{,}001}\approx2000.000\), while \(d_S=\sqrt{250{,}009}\approx500.009\).)
- A two-variable covariance matrix is \(\begin{pmatrix}4&2\\2&1\end{pmatrix}\). The first predictor is multiplied by 3. What is the new covariance matrix? (Answer: \(\begin{pmatrix}36&6\\6&1\end{pmatrix}\).)
- Three points are U=(0,0), V=(4,0), and W=(0,3), with \(k=2\). Which pair minimizes the k-means within-cluster sum of squares? (Answer: U and W; pair costs are \(UV:8\), \(UW:4.5\), and \(VW:12.5\).)
- A tree splits a predictor at 8, sending values at most 8 left. The predictor is multiplied by 5. What threshold gives the same training partition, and why? (Answer: 40; positive multiplication preserves the ordering and \(x\leq8\) exactly when \(5x\leq40\).)
- Two predictors are standardized using training means 10 and 50 and training standard deviations 2 and 5. A new case has values 14 and 45. What are its standardized coordinates? (Answer: \((2,-1)\), since \((14-10)/2=2\) and \((45-50)/5=-1\).)