Why an Extreme Observation Matters
K-means represents each cluster by its arithmetic-mean centroid and assigns observations according to squared Euclidean distance. As a result, a single observation far from the rest can move a centroid substantially. That movement can change the boundary between clusters, alter which observations are grouped together, and change the within-cluster sum of squares.
An extreme observation is not automatically an error. It may be a valid, rare profile, or it may reflect a recording mistake or a feature measured on an unsuitable scale. The statistical issue is that k-means uses means and squared distances, both of which give observations far from a centroid substantial influence.
For one-dimensional observations, squared Euclidean distance is simply squared difference. For vectors, the squared distance is the sum of the squared coordinate differences. In either case, a point far from its cluster’s centroid can contribute a large amount to WCSS.
Why the Mean Moves Toward an Outlier
Suppose a cluster has \(n\) observations with centroid \(\bar{x}\), and we add an observation \(z\). The new centroid is
The second expression makes the shift explicit: it is \((z-\bar{x})/(n+1)\), in the direction of the added observation. A large separation \(z-\bar{x}\) produces a large shift; a larger cluster size reduces the shift for a given separation.
The mean is also the point that minimizes the sum of squared distances within a cluster. To see why in one dimension, let \(c\) be any proposed center and write \(x_i-c=(x_i-\bar{x})+(\bar{x}-c)\). On summing the squares, the cross term vanishes because \(\sum_i(x_i-\bar{x})=0\). Thus
The second term is nonnegative and is zero at \(c=\bar{x}\), so the mean minimizes the sum. The same argument applies coordinate by coordinate for vector observations. This explains both why k-means updates to arithmetic means and why an extreme point affects its fitted center.
Worked Example: A Single-Cluster Center and Its WCSS
A hypothetical risk-profile data set has one numerical feature, recorded for five profiles as \(2,3,4,5,6\). First find the centroid and WCSS. Then add a valid but extreme profile with value \(26\), and calculate the new centroid and WCSS. These are fixed feature values; no interest rate, mortality assumption, or probability distribution is involved.
Before adding the extreme profile. The centroid is
The squared deviations are \(4,1,0,1,4\), so the WCSS is \(4+1+0+1+4=10\).
After adding the profile with value \(26\). There are now six observations, and the centroid is
The shift is \(23/3-4=11/3\), or about \(3.6667\). The new WCSS is
As a check, the centroid shift formula gives \((26-4)/6=22/6=11/3\), matching the direct calculation. The method is to recompute the mean using all observations, then sum squared deviations from that updated mean. A common mistake is to report the original WCSS of \(10\) after adding the new observation; WCSS must be calculated for the full updated data set and its new centroid.
Centroid Shifts Can Change Assignments
K-means alternates between assigning each observation to its nearest centroid and updating each centroid to the mean of its assigned observations. In one dimension, the boundary between two centroids is their midpoint. If one centroid moves, that midpoint moves too, so an observation can switch clusters even when its own value has not changed.
Worked Example: An Outlier Changes the Lloyd Updates
Consider one-dimensional observations \(0,4,6,10,30\), with initial centroids \(0\) and \(10\). Trace the assignment and update steps until the assignments stabilize, and calculate the final WCSS. Then compare with the data set that omits \(30\). All feature values are deterministic; no interest rate, mortality assumption, or probability distribution is involved.
Initial assignment. Using the initial centers, \(0\) and \(4\) are closer to \(0\), while \(6\), \(10\), and \(30\) are closer to \(10\). There are no ties. The first clusters are therefore \(\{0,4\}\) and \(\{6,10,30\}\).
First centroid update. The centers become
Their boundary is \(\bigl(2+46/3\bigr)/2=26/3\approx8.6667\). Since \(6<26/3\), observation \(6\) now belongs to the first cluster. The assignments become \(\{0,4,6\}\) and \(\{10,30\}\).
Second centroid update. The new centers are \(10/3\) and \(20\), and their boundary is \(\bigl(10/3+20\bigr)/2=35/3\approx11.6667\). Observation \(10\) is now closer to \(10/3\), so the clusters become \(\{0,4,6,10\}\) and \(\{30\}\).
Third update and check. The centers are \(5\) and \(30\). The boundary is \((5+30)/2=17.5\), so all four observations \(0,4,6,10\) remain in the first cluster and \(30\) remains in the second. The assignments have stabilized.
The final WCSS is
Without \(30\), the same initial centers assign \(\{0,4\}\) and \(\{6,10\}\). Their updated centers are \(2\) and \(8\), and the boundary \(5\) leaves those assignments unchanged. The WCSS for that separate data set is \((4+4)+(4+4)=16\). This illustrates that including an extreme observation can change the assignments of other points, not just the centroid of the cluster receiving the extreme point.
The Lloyd updates do not increase WCSS from one iteration to the next for a fixed data set, but adding an observation creates a different data set. Therefore, the WCSS values \(52\) and \(16\) are not a like-for-like comparison of the algorithm’s performance. The key result here is the assignment change: the outlier pulls a center far enough to move the boundary across observations \(6\) and \(10\).
Outlier Effects in More Than One Dimension
For a vector-valued observation, each coordinate contributes to the centroid shift. An observation can therefore pull a center in a direction that is not captured by looking at just one feature. WCSS also adds squared deviations across coordinates, so a large displacement in even one direction can have a substantial effect.
Worked Example: A Two-Feature Centroid Shift
Three hypothetical profiles have feature vectors \((1,1)\), \((3,1)\), and \((2,3)\). Find their centroid and WCSS, then add the profile \((14,7)\) and recalculate both quantities. The coordinates are deterministic measurements; no interest rate, mortality assumption, or probability distribution is involved.
Original centroid and WCSS. Average each coordinate separately:
The three squared distances to this centroid are \(13/9\), \(13/9\), and \(16/9\). Thus the original WCSS is \(42/9=14/3\).
Updated centroid. After adding \((14,7)\), the coordinate means are
The center has moved by \((5-2,3-5/3)=(3,4/3)\). To check the new WCSS directly, the squared distances from \((5,3)\) are \(20\), \(8\), \(9\), and \(97\), respectively. Their sum is \(134\). The method is to update both coordinate means and then calculate each full squared Euclidean distance. A common mistake is to calculate the centroid using only the coordinate in which the outlier looks most extreme; k-means uses all included features in the distance calculation and in the centroid.
Interpreting Sensitivity Carefully
These calculations show a mathematical property of k-means, not a rule that all extreme observations should be removed. Before changing a data set, determine whether an unusual value is a recording error, a valid rare case, or a consequence of the feature scale. Removing a valid profile merely because it affects a center can conceal meaningful structure.
A few checks help keep an outlier analysis on track:
- Check the observation and its units. A value recorded in the wrong unit or entered incorrectly can look extreme for reasons unrelated to the population.
- Check the feature scale. K-means uses Euclidean distances; a feature with large numerical values can dominate distances if features have not been placed on appropriate scales.
- Recalculate assignments, not only the center. A shifted centroid can move a boundary and reassign other observations, as the one-dimensional example demonstrated.
- State which observations define each WCSS. WCSS depends on the data set, the assignments, and the centroids. Values computed on different data sets should not be treated as directly comparable evidence of better fit.
An outlier’s influence is relative to the cluster size and the distance between the observation and the center. The centroid shift formula quantifies this: for fixed cluster size, greater separation means greater movement; for fixed separation, a larger cluster means less movement. Squared distances make the effect on WCSS especially pronounced.
Check Your Understanding
For each problem, treat the listed feature values as deterministic observations and use arithmetic-mean centroids and squared Euclidean distance.
- A one-cluster data set contains \(3,5,8\). Add the observation \(20\). What is the new centroid and how far has it shifted from the original centroid? (Answer: New centroid \(9\); shift \(11/3\).)
- A cluster contains \(2,4,6\). Find its original WCSS, then add \(10\) and find the updated centroid and WCSS. (Answer: Original WCSS \(8\); updated centroid \(5.5\); updated WCSS \(35\).)
- For observations \(0,2,4,9,24\), initialize two one-dimensional centers at \(0\) and \(10\). After the first assignment and centroid update, what are the clusters and centers? (Answer: Clusters \(\{0,2,4,9\}\) and \(\{24\}\); centers \(3.75\) and \(24\).)
- Three profiles are \((0,1)\), \((2,1)\), and \((1,2)\). Add \((9,8)\). What is the updated centroid? (Answer: \((3,3)\).)
- A cluster has \(n=5\), centroid \(12\), and an added observation \(42\). Using the centroid shift formula, how far and in which direction does the center move? (Answer: It moves \(5\) units in the increasing direction, to \(17\).)