Why Feature Scaling Changes K-Means

Why Scale Matters in K-Means

K-means forms clusters by measuring distances between observations. That makes the scale of each feature consequential: a difference of 10 units on one feature can outweigh a difference of 1 unit on another, even when the smaller numerical difference is more meaningful for the analysis. As a result, changing a feature's units can change which observations are grouped together.

This is not a numerical quirk in the algorithm. It follows directly from the distance measure K-means uses. Scaling changes the geometry of the data, and K-means uses that geometry both to assign observations to clusters and to calculate cluster centers. The examples below show how those effects arise.

Definition: For observations with $p$ numerical features, let $x_{ij}$ be the value of feature $j$ for observation $i$, and let $\mu_{kj}$ be feature $j$ of the center of cluster $k$. The squared Euclidean distance from observation $i$ to that center is $$ d^2(\mathbf{x}_i,\boldsymbol{\mu}_k)=\sum_{j=1}^{p}(x_{ij}-\mu_{kj})^2. $$ K-means assigns observations to centers using Euclidean distance, equivalently squared Euclidean distance, and seeks a partition minimizing the within-cluster sum of squares (WCSS), the sum of these squared distances to the assigned centers.

This setup assumes that the inputs are numerical features for which Euclidean distance is appropriate. It does not prescribe that their raw units are comparable or that every feature should receive equal substantive importance. Those are modeling decisions. K-means itself treats the numerical contributions in the distance formula as given.

How a Scale Factor Changes a Distance

Suppose feature $j$ is multiplied by a factor $a_j$. The difference between an observation and a center on that feature is then multiplied by $a_j$, so its squared contribution to distance is multiplied by $a_j^2$. If feature $j$ is divided by 10, its squared contribution becomes one hundredth as large. If it is multiplied by 3, its contribution becomes nine times as large.

d_{\mathrm{scaled}}^2(\mathbf{x}_i,\boldsymbol{\mu}_k) =\sum_{j=1}^{p}a_j^2(x_{ij}-\mu_{kj})^2

This equation explains why scaling one feature can change which center is nearest: it changes that feature's relative contribution to every distance comparison. It also changes the objective K-means is minimizing. If every feature is multiplied by the same positive constant, all distances are multiplied by that constant and the nearest-center comparisons do not change. Scaling only selected features is different: it changes their relative influence.

Justification: In the transformed data, feature $j$ has value $a_jx_{ij}$, and the corresponding center has value $a_j\mu_{kj}$. Their difference is $a_j(x_{ij}-\mu_{kj})$. Squaring gives $a_j^2(x_{ij}-\mu_{kj})^2$. Summing over features gives the displayed scaled-distance formula. Since the nearest center is the one with the smallest sum, changing the factors can change the assignment.

Assignments and Cluster Structure Can Change

The next example holds the starting centers fixed while comparing raw and scaled distances. It shows that a scale change can do more than alter the reported WCSS: it can reorganize the observations into different clusters and produce different centroids.

Worked Example: A Scale Change Reorganizes Four Observations

Four observations have two features: $A=(0,0)$, $B=(0,4)$, $C=(10,0)$, and $D=(10,4)$. Start with centers at $A$ and $D$, and assign each observation to its nearer center. First use the raw features. Then divide the first feature by 10 and repeat the assignments and centroid updates. State the method and compare the resulting clusters.

Method. Compute squared distances from each observation to both centers, assign to the smaller one, and calculate each new center as the arithmetic mean of its assigned observations. Using squared distances gives the same assignments as using distances because the square root preserves their ordering.

Raw-scale assignments. For $A$, the squared distances to $A$ and $D$ are $0$ and $10^2+4^2=116$, so $A$ goes to the first center. For $B$, they are $4^2=16$ and $10^2=100$, so $B$ also goes to the first center. For $C$, they are $10^2=100$ and $4^2=16$, so $C$ goes to the second center. Finally, $D$ has squared distances $116$ and $0$, so it goes to the second center. The clusters are $\{A,B\}$ and $\{C,D\}$.

Raw-scale centroid update. The first centroid is $( (0+0)/2,(0+4)/2 )=(0,2)$. The second is $( (10+10)/2,(0+4)/2 )=(10,2)$. Each observation in either cluster is 2 units from its centroid in the second feature, so each squared distance is 4. The WCSS is $4+4+4+4=16$.

Scaled assignments. After dividing the first feature by 10, the observations become $A=(0,0)$, $B=(0,4)$, $C=(1,0)$, and $D=(1,4)$; the starting centers are now $A=(0,0)$ and $D=(1,4)$. Observation $A$ remains with the first center, at squared distances 0 and 17. For $B$, the squared distances are 16 to $A$ and 1 to $D$, so it goes to the second center. For $C$, they are 1 to $A$ and 16 to $D$, so it goes to the first center. Observation $D$ remains with its own center, at squared distances 17 and 0. The clusters are now $\{A,C\}$ and $\{B,D\}$.

Scaled centroid update and check. In scaled coordinates, the new centroids are $(0.5,0)$ and $(0.5,4)$. Each point is 0.5 from its assigned centroid in the first feature, giving squared distance 0.25 per point and WCSS $4(0.25)=1$. In the original units, the centroids correspond to $(5,0)$ and $(5,4)$; the first-feature differences are still 0.5 in scaled coordinates. Thus the assignments and centroids differ from the raw-scale result, as the changed distance contributions predict.

The two WCSS values, 16 and 1, use different distance scales, so they are not a direct comparison of clustering quality. They are checks of the objective under each chosen metric. The common mistake this example avoids is to treat a lower numerical WCSS after rescaling as proof that the scaled clustering is inherently better.

Units Can Set the Implicit Feature Weights

A feature's numerical scale can be determined by its unit of measurement. Expressing a monetary value in dollars rather than thousands of dollars multiplies that feature by 1,000. If that is the only change, its squared contribution to Euclidean distance is multiplied by $1{,}000^2$. The physical data have not changed, but the metric used by K-means has.

Worked Example: Changing Currency Units Changes the Nearest Center

An observation has coordinates $(0,\$0)$. Two candidate centers are $C_1=(0,\$100)$ and $C_2=(2,\$0)$, where the first coordinate is a score and the second is a monetary amount. Determine the nearest center first using dollars, then using thousands of dollars. Explain why the result changes.

Method in dollars. The squared distance to $C_1$ is $(0-0)^2+(0-100)^2=10{,}000$, so its distance is 100. The squared distance to $C_2$ is $(0-2)^2+(0-0)^2=4$, so its distance is 2. The observation is assigned to $C_2$.

Method in thousands of dollars. The monetary coordinates become 0, 0.1, and 0. The squared distance to $C_1$ is $0^2+(0-0.1)^2=0.01$, so its distance is 0.1. The squared distance to $C_2$ remains $2^2+0^2=4$, so its distance is 2. The observation is now assigned to $C_1$.

Check. The dollar-to-thousands conversion multiplied the monetary feature by 0.001, so its squared distance contribution was multiplied by $0.001^2=0.000001$. The assignment changed from $C_2$ to $C_1$ because the score difference now matters more relative to the monetary difference. The common mistake is to assume that changing units cannot affect a clustering method; here, the numerical distance calculation shows exactly why it can.

Scaling Changes the Objective, Not Just the Picture

Because K-means minimizes WCSS, scaling affects both the assignment step and the quantity being minimized. The following calculation isolates the effect on the objective for one fixed cluster. It also shows why WCSS values computed under different feature scales should not be compared as though they came from the same metric.

Worked Example: A Feature Multiplied by Two Has Four Times the Weight

A cluster contains $P_1=(0,0)$, $P_2=(2,0)$, and $P_3=(0,1)$. Calculate its centroid and WCSS, then multiply the first feature by 2 and recalculate in the transformed coordinates. Identify the method and verify the effect of the scale factor.

Method in the original coordinates. The centroid is the coordinate-wise mean: $( (0+2+0)/3,(0+0+1)/3 )=(2/3,1/3)$. The squared distance from $P_1$ to the centroid is $(2/3)^2+(1/3)^2=5/9$. For $P_2$, it is $(4/3)^2+(1/3)^2=17/9$. For $P_3$, it is $(2/3)^2+(2/3)^2=8/9$. Therefore the WCSS is $5/9+17/9+8/9=30/9=10/3$.

After scaling the first feature. The transformed points are $(0,0)$, $(4,0)$, and $(0,1)$. Their centroid is $(4/3,1/3)$. The squared distances are $17/9$ for the first point, $65/9$ for the second, and $20/9$ for the third. For example, the second distance is $(4-4/3)^2+(0-1/3)^2=(8/3)^2+(1/3)^2=65/9$. The new WCSS is $(17+65+20)/9=102/9=34/3$.

Check. Before scaling, the sum of squared deviations in the first feature is $4/9+16/9+4/9=24/9=8/3$; the second-feature sum is $1/9+1/9+4/9=6/9=2/3$. Their total is $10/3$. Multiplying the first feature by 2 multiplies its contribution by 4, so the transformed total must be $4(8/3)+2/3=34/3$, matching the direct calculation. The common mistake is to multiply that contribution by 2 instead of by $2^2=4$.

Choosing a Scale Is a Modeling Decision

K-means does not decide whether a one-unit difference in one feature should count as much as a one-unit difference in another. That depends on the units and on the purpose of the analysis. For example, a hypothetical insurer might cluster policies using both an annual premium amount and a customer-service score. If premium is recorded in dollars while the score lies between 1 and 5, raw squared distances may be driven largely by premium amounts. This could be appropriate if premium differences should dominate, but not if the analyst intended both features to contribute comparably.

A common response is to rescale features, for example by dividing by their observed ranges or standard deviations. Such choices are not neutral: each gives the features a particular relative influence and can change the clusters. Standardizing is one possible choice, not a guarantee that a resulting partition is substantively correct. The scale should reflect the analytical goal, and the resulting clusters should be interpreted in context.

A frequent exam trap is to infer that K-means is “scale-free” because it uses the same Euclidean formula for every feature. The formula is identical, but the inputs to it are numbers in particular units. Another trap is to confuse the numeric magnitude of a feature with its importance. A large raw range may reflect a unit choice, not a deliberate decision that the feature deserves more weight.

Inspect the feature scales.
Record the units and typical ranges of the numerical inputs before interpreting distances.

Calculate the distance contributions.
For each feature, compare its squared difference from a candidate center; do not assume the contributions are comparable.

Make the scale choice explicit.
Decide whether raw units, a chosen rescaling, or a deliberate weighting matches the intended notion of similarity.

Interpret the resulting solution under that metric.
Assignments, centroids, and WCSS all depend on the distances used. A WCSS from one scaling cannot be directly ranked against a WCSS from another scaling as if the metrics were unchanged.

Key takeaway: In K-means, a feature multiplied by $a$ contributes $a^2$ times as much squared distance. Scaling can therefore change nearest-center assignments, centroid locations, and cluster structure. The appropriate scale is a modeling choice about what counts as similar, not a cosmetic adjustment.

Understanding this dependence is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. In a calculation, check the actual units and squared contributions before accepting an assignment or comparing WCSS values.

Check Your Understanding

For each problem, compare the stated distances or objective values carefully; remember that a scale factor is squared in squared Euclidean distance.

An observation is $q=(2,2)$, with candidate centers $C_1=(0,3)$ and $C_2=(5,2)$. Find its nearest center using the raw coordinates. Then divide the first feature of every coordinate by 10 and find the nearest center again. (Answer: Raw: $C_1$, since the squared distances are 5 and 9. Scaled first feature: $C_2$, since the squared distances are 1.04 and 0.09.)
For $q=(0,0)$, compare centers $C_1=(3,1)$ and $C_2=(1,2)$. Determine the nearest center before and after multiplying the second feature of all coordinates by 2. (Answer: Before scaling, $C_2$ is nearer: squared distances 10 and 5. After scaling, $C_1$ is nearer: squared distances 13 and 17.)
A cluster consists of $(1,0)$ and $(5,0)$. Find its centroid and WCSS. Then multiply the first feature by 3 and give the new centroid and WCSS in transformed coordinates. (Answer: Original centroid $(3,0)$, WCSS $8$; transformed centroid $(9,0)$, WCSS $72$.)
An observation is $q=(0,0)$, and two centers are $C_1=(0,50)$ and $C_2=(1,0)$, where the second feature is measured in dollars. Which center is nearer? What happens if that feature is instead measured in thousands of dollars? (Answer: In dollars, $C_2$ is nearer, with distances 1 versus 50. In thousands of dollars, $C_1$ is nearer, with distances 0.05 versus 1.)
One feature is multiplied by 4 while all other features are unchanged. By what factor does that feature's squared-distance contribution change? If its original contribution was 3 and the other features together contributed 7, what is the new total squared distance? (Answer: Its contribution is multiplied by 16, becoming 48; the new total is 55.)

What You'll Learn