Why Outliers Need Special Attention in Clustering
Clustering groups observations according to their similarities across selected features. An observation that is far from most of the data can therefore matter in two ways: it may receive an unusual cluster assignment, and it may change the distances or summaries used to describe other observations. The effect depends on the features, their scales, the distance measure, and the clustering procedure. A point is not automatically a data error merely because it is far away.
The earlier material on scaling predictors before distance-based clustering and on Euclidean distance establishes two important checks: compare observations on appropriately prepared features, and interpret a distance in the feature space actually used. A plot or distance summary can flag a record for review, but neither alone establishes why the record is unusual.
Visual and Distance-Based Diagnostics
Start with the geometry of the features used for clustering, not with a list of records that happen to have large raw values. A boxplot can reveal an unusually high or low value in one feature. A scatterplot can show whether a record is far from the multivariate pattern, even when none of its individual feature values looks exceptional. For more than two features, pair plots or other suitable visual summaries can help, but they cannot show every direction in the full feature space.
Distance summaries provide a numerical companion to these plots. For example, examine each observation's distance to its nearest other observation, or its distance to a representative profile for a group. A very large nearest-neighbor distance suggests isolation relative to the chosen features and distance. It does not prove a mistake: a valid small subpopulation may also be isolated. Always calculate these distances after the feature preparation and missing-value treatment intended for the analysis.
A one-feature boxplot commonly uses the interquartile range, \(IQR=Q_3-Q_1\), where \(Q_1\) and \(Q_3\) are the first and third quartiles. Under the usual Tukey-fence convention, values below \(Q_1-1.5IQR\) or above \(Q_3+1.5IQR\) are flagged for inspection. This is a screening convention, not a deletion rule. Quartile algorithms can differ, particularly in small samples, so state the convention when a numerical fence matters.
Worked Example: Flagging an Extreme Value With a Boxplot
A hypothetical service team records a scaled workload index for seven branches: \(-1, 0, 1, 1, 2, 2, 10\). Using Tukey hinges, determine whether 10 is flagged and compare its position with the median.
Method. Order the values as given. The median is the fourth value, \(1\). Excluding the median, the lower half is \(-1,0,1\), so \(Q_1=0\); the upper half is \(2,2,10\), so \(Q_3=2\).
The interquartile range is \(IQR=2-0=2\). The upper fence is \(Q_3+1.5IQR=2+1.5(2)=5\), and the lower fence is \(Q_1-1.5IQR=0-3=-3\). Since \(10>5\), the value 10 is flagged; the other six values fall between \(-3\) and \(5\).
Its distance from the median is \(10-1=9\), compared with a median of 1 for the group. The calculation supports investigating that branch and checking a plot, but it does not justify deleting it. The common mistake is to treat a boxplot flag as proof of bad data; it is a prompt for context and verification.
For multiple features, a distance can summarize how far a point lies from a selected reference. If the reference is a profile with coordinates \(c_1,\ldots,c_p\), Euclidean distance for observation \(x=(x_1,\ldots,x_p)\) is \(\sqrt{\sum_{j=1}^{p}(x_j-c_j)^2}\). This formula assumes numeric coordinates on a meaningful, prepared scale. If one feature has a much wider scale than another, it can dominate the result; categorical features also require an appropriate representation and distance. A distance number should therefore be interpreted alongside the feature definitions and scaling choices, not as a scale-free measure of abnormality.
Assessing Influence on Cluster Results
A useful influence check compares the analysis with and without a flagged point, or compares the original record with a verified correction if one is available. Look beyond whether the extreme observation itself forms a small or separate group. Ask whether ordinary observations change assignments, whether group profiles move substantially, and whether the substantive description of the groups changes. Keep the same feature definitions and distance choices in each comparison; otherwise, the effect of the outlier is mixed with the effect of changing the analysis.
The reason a faraway observation can affect a mean-based profile follows directly from the definition of an arithmetic mean. Suppose \(n\) observations have mean \(\mu\), and one value \(z\) is added. The new mean is the sum of all \(n+1\) values divided by \(n+1\):
Thus the shift is \(\mu'-\mu=(z-\mu)/(n+1)\): it grows with the point's distance from the original mean and shrinks as the group size grows. In a multi-feature profile, this relationship applies coordinate by coordinate. Since a distance squares coordinate differences, one very large deviation can also contribute disproportionately to squared-distance calculations. This explains why a single point can pull a mean-based profile and affect which other points appear closer to it. The argument explains a possible mechanism of influence; it does not imply that every clustering method uses means or responds in the same way.
Worked Example: Checking Whether One Record Changes Assignments
A hypothetical two-feature analysis has two existing group profiles at \((1,0)\) and \((7,0)\). Four ordinary observations are \((0,0)\), \((2,0)\), \((6,0)\), and \((8,0)\). A newly reviewed observation is \((100,0)\). Using Euclidean distance to the stated profiles, first determine the ordinary assignments, then assess a sensitivity calculation in which the second profile is recalculated as the mean of its original two observations plus the extreme point.
Method. All points lie on the horizontal axis, so Euclidean distance is the absolute difference between first coordinates. Before adding the extreme point, the distances to profiles 1 and 2 are, respectively: for \((0,0)\), \(1\) and \(7\); for \((2,0)\), \(1\) and \(5\); for \((6,0)\), \(5\) and \(1\); and for \((8,0)\), \(7\) and \(1\). The first two are closer to profile 1, and the last two are closer to profile 2. The new point is \(99\) units from profile 1 and \(93\) from profile 2, so it is closer to profile 2.
For the requested sensitivity calculation, the second profile becomes the mean of coordinates \(6,8,100\): \(c'_2=(6+8+100)/3=114/3=38\). The first profile remains at 1. The equal-distance boundary is their midpoint, \((1+38)/2=19.5\). The ordinary points at 6 and 8 are now closer to profile 1: their distances to the two profiles are \(5\) and \(32\), then \(7\) and \(30\). Both have changed their nearest-profile assignment. The extreme point remains closer to profile 2, at distances \(99\) and \(62\).
This sensitivity calculation shows how an extreme observation can alter the meaning of a mean-based profile and affect other assignments. It is not a claim that every clustering procedure updates profiles in this way. The common mistake is to report only that the extreme point joins one group; the relevant influence check also asks what happened to the ordinary observations.
Outlier Candidate or Data Error?
A distance flag answers a statistical question—“is this record unusual relative to these features and this data set?”—not a data-quality question—“was it recorded correctly?” To distinguish a meaningful extreme from an error, check allowed ranges, units, data definitions, source records, and whether the value is supported by other information. A decimal-place error, a unit mismatch, or an incorrectly joined record may create a huge distance. Conversely, a rare but valid customer or claim profile may be important precisely because it is unusual.
Do not automatically remove a valid extreme simply to make clusters look more compact. Removing it can hide a real segment or understate the range of the population. If the record is confirmed erroneous, correct it from a reliable source where possible; if it cannot be corrected, document the treatment and compare conclusions with and without it. If it is valid, consider whether the selected features and distance reflect the business question, and report how much the grouping depends on that record.
Worked Example: Investigating an Extreme Profile
A hypothetical planning team compares two features for service accounts: account age in years and annual service visits. For this diagnostic, reference means are 50 years and 4 visits, and the corresponding standard deviations are 10 years and 2 visits. Two records are flagged: Account A is \((82,18)\), and Account B is \((820,18)\). Calculate their Euclidean distances after standardizing each feature by its stated mean and standard deviation, then decide what the statistics can and cannot establish.
Method. For a feature value \(x_j\), standardize it as \((x_j-\text{mean}_j)/\text{standard deviation}_j\). For Account A, the standardized coordinates are \((82-50)/10=3.2\) and \((18-4)/2=7\). Its distance from the reference profile \((0,0)\) is \(\sqrt{3.2^2+7^2}=\sqrt{10.24+49}=\sqrt{59.24}\approx7.6968\).
For Account B, the standardized coordinates are \((820-50)/10=77\) and \((18-4)/2=7\). Its distance is \(\sqrt{77^2+7^2}=\sqrt{5929+49}=\sqrt{5978}\approx77.3175\). The distances are rounded to four decimal places. Both records are unusual relative to the stated reference; Account B is much farther away, largely because its age coordinate is 77 standard deviations above the reference mean.
The calculation does not prove Account A is valid or Account B is wrong. A very high service-visit count may be unusual but possible, while an age of 820 years is implausible under the stated account-age definition. The appropriate next step for Account B is to check the original record, units, and field definition; for Account A, check the source and context before treating it as a genuine extreme. The common mistake is to infer validity or invalidity from distance alone.
A Practical Review Sequence
Check that the features, scaling, and distance are suitable for the question. An extreme raw value may be unremarkable after appropriate scaling, or a scale mismatch may make one feature dominate.
Use feature plots and, where feasible, plots of pairs of features. Look for both isolated values and unusual combinations that a separate one-feature check could miss.
Compare the flagged record's distances with those of ordinary records under the same prepared features. Treat a large distance as a diagnostic, not a verdict.
Check source data and definitions. Compare assignments and profiles with the record included and excluded, or with a verified correction, while holding other analysis choices fixed.
State whether the point was retained, corrected, or excluded and why. Report whether the substantive cluster interpretation changed.
In R, plots such as boxplots and scatterplots can support the visual review, while distance objects and summaries can help identify unusually isolated rows. Keep row identifiers attached to the distances so a flagged value can be traced back to its source. Do not rely on a plot alone for high-dimensional data or on a single threshold as a universal rule: the appropriate evidence depends on context, feature preparation, and the consequences of the decision.
For an actuarial analysis, an unexplained extreme record can distort segmentation and therefore affect how results are communicated or used. A defensible analysis distinguishes a statistical flag from a data-quality finding, records the sensitivity check, and acknowledges uncertainty when the record cannot be verified. This diagnostic discipline is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, state what the calculation shows and what additional judgment or check is needed.
- A hypothetical feature has ordered values \(2,3,3,4,5,6,15\). Using Tukey hinges and excluding the median when splitting the halves, find \(Q_1\), \(Q_3\), the upper fence, and whether 15 is flagged. (Answer: \(Q_1=3\), \(Q_3=6\), \(IQR=3\), upper fence \(=10.5\); 15 is flagged for investigation, not automatically removed.)
- Two fixed profiles on one scaled feature are at 0 and 10. A record is at 13. Find its distances to both profiles and its nearer profile. What does this establish? (Answer: distances are 13 and 3, so it is nearer the profile at 10; this does not establish whether its value is valid.)
- A group of four observations has mean 5 on a feature. Add one observation with value 20. Calculate the new mean and the shift. (Answer: new mean \(=(4(5)+20)/5=8\); shift \(=3\).)
- Two standardized features have reference means 20 and 100 and standard deviations 5 and 20. For a record \((30,140)\), find its standardized coordinates and Euclidean distance from the reference profile. (Answer: coordinates \((2,2)\); distance \(=\sqrt{2^2+2^2}=\sqrt{8}\approx2.8284\).)
- A flagged record is confirmed as valid, but removing it changes the assignment of several ordinary records. Give two items that should be reported or investigated before interpreting the groups. (Answer: report the assignment/profile sensitivity with the record retained and removed; investigate whether the feature choices, scaling, or distance make the grouping unduly dependent on this valid extreme.)