Why Clustering Results Need Troubleshooting
A clustering algorithm returns a grouping under specified choices: selected variables, a distance or dissimilarity measure, an algorithm, and—in some methods—a tuning parameter or chosen number of clusters. The output is therefore conditional on those choices. A clean-looking plot does not establish that the groups are real, stable, or useful for the question at hand.
Troubleshooting means asking whether the grouping is driven by a defensible pattern or by an avoidable feature of the analysis. This tutorial focuses on four checks: scale, outliers, cut choices, and assignment stability. These checks help an analyst distinguish a useful exploratory summary from a result that should be qualified or reconsidered. This approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Whether Scale Is Driving the Result
For complete numeric observations compared using Euclidean distance, the distance between observations \(i\) and \(j\) is the square root of the sum of their squared coordinate differences. This definition has an important practical consequence: a variable with large numerical units or a wide spread can dominate the distance, even if it is not substantively more important.
If a feature is measured in thousands while another is measured in tenths, the first feature can contribute much more to this sum. A common diagnostic is to compare results on the original measurements with results after scaling numeric features, for example by centering each feature and dividing by its standard deviation. Scaling is not automatically correct: it changes the relative influence of features. The analyst must decide whether equalizing variation is sensible for the particular purpose.
Here, \(\bar{x}_r\) and \(s_r\) are the sample mean and sample standard deviation of feature \(r\). The second expression follows because centering cancels in a difference: \((x_{ir}-\bar{x}_r)-(x_{jr}-\bar{x}_r)=x_{ir}-x_{jr}\). Dividing by \(s_r\) then measures the difference in units of that feature’s standard deviation. Thus standardizing changes the distance by assigning a contribution proportional to \(1/s_r^2\) to feature \(r\). It does not merely change the display units.
Worked Example: A Change in Scale Reverses the Nearest Neighbor
Three hypothetical service regions have two recorded features, \(X_1\) and \(X_2\). Their coordinates are \(A=(0,0)\), \(B=(1,100)\), and \(C=(10,0)\). Suppose the analyst’s chosen reference scales are \(s_1=10\) and \(s_2=1000\). Determine which region is closest to \(A\) before and after scaling.
Method. Calculate Euclidean distances from \(A\) using the original coordinates, then calculate the distances after dividing each feature difference by its stated scale.
Original scale, \(A\) to \(B\): \(d(A,B)=\sqrt{(1-0)^2+(100-0)^2}=\sqrt{1+10{,}000}=\sqrt{10{,}001}\approx100.005\).
Original scale, \(A\) to \(C\): \(d(A,C)=\sqrt{(10-0)^2+(0-0)^2}=\sqrt{100}=10\). Thus \(C\) is closer to \(A\) than \(B\) is on the original scale.
Scaled distance, \(A\) to \(B\): \(d_z(A,B)=\sqrt{(1/10)^2+(100/1000)^2}=\sqrt{0.01+0.01}=\sqrt{0.02}\approx0.1414\).
Scaled distance, \(A\) to \(C\): \(d_z(A,C)=\sqrt{(10/10)^2+(0/1000)^2}=\sqrt{1+0}=1\). Now \(B\) is closer to \(A\). The nearest-neighbor ordering has changed because the feature contributions changed, not because any region’s recorded values changed.
Check and trap avoided. The result is consistent with the scaled differences: \(A\) and \(B\) differ by only \(0.1\) units on each scaled coordinate, while \(A\) and \(C\) differ by \(1\) on the first. A common mistake is to standardize automatically and then claim the new grouping is objectively better. Scaling should instead be justified by the meaning and role of each feature; the stated reference scales here are assumptions for this hypothetical calculation.
In practice, inspect the units, ranges, and spreads of the features used in the distance calculation. In R, compare the clustering inputs before and after a deliberate scaling choice, and record the choice so the analysis can be reproduced. If conclusions change substantially, report that sensitivity rather than presenting only the preferred result.
Check Whether an Outlier Dominates
An outlier can pull a cluster center, create a small isolated group, or dominate a distance-based objective. The effect is especially clear for a one-dimensional \(k\)-means objective, which sums squared distances from observations to their assigned cluster centers. If a cluster contains values \(x_1,\ldots,x_n\), its mean \(\bar{x}\) minimizes \(\sum_i(x_i-c)^2\) over possible centers \(c\). Squaring makes a large deviation contribute disproportionately.
To see why this identity holds, write \(x_i-c=(x_i-\bar{x})+(\bar{x}-c)\), square, and sum. The cross-term is \(2(\bar{x}-c)\sum_i(x_i-\bar{x})=0\), because deviations from the mean sum to zero. The remaining extra term is \(n(\bar{x}-c)^2\), which is nonnegative and is zero only at \(c=\bar{x}\). This proves the minimization claim. It also explains why an extreme observation can strongly affect both the mean and the objective.
Worked Example: One Extreme Value Changes the Cluster Summary
A hypothetical portfolio has a single feature measuring a service-use score. Four observations have scores \(2,4,6,8\). A fifth observation with score \(100\) is later found. Compare the one-cluster mean and sum of squared deviations before and after adding the fifth observation.
Method. For a one-cluster \(k\)-means summary, use the mean as the center and sum squared deviations from it. Separately compare medians to see whether the center summary is sensitive to the extreme value.
Without the score of 100: the mean is \((2+4+6+8)/4=20/4=5\). The squared deviations are \((2-5)^2=9\), \((4-5)^2=1\), \((6-5)^2=1\), and \((8-5)^2=9\). Their sum is \(9+1+1+9=20\). The median is \((4+6)/2=5\).
With the score of 100: the mean is \((2+4+6+8+100)/5=120/5=24\). The squared deviations are \(484,400,324,256,\) and \(5{,}776\), respectively, because the deviations are \(-22,-20,-18,-16,\) and \(76\). The sum is \(484+400+324+256+5{,}776=7{,}240\). The median is now \(6\).
Check and trap avoided. The mean moves from \(5\) to \(24\), and the squared-deviation total grows from \(20\) to \(7{,}240\). The median moves only from \(5\) to \(6\). This does not prove that the score of \(100\) is an error: it may be a valid rare observation. A common mistake is to delete it solely because it looks extreme. First check data quality and context; then assess how the result changes with and without it, or consider a method whose objective is less sensitive to extreme distances. State the method used rather than treating one summary as universally robust.
A useful diagnostic is to inspect the observations that contribute most to the distance or within-cluster variation, then verify their values and units. If an outlier is valid, it may still be important to the analysis. The question is whether it represents a meaningful distinct case or is controlling a grouping that was intended to summarize the bulk of the observations.
Check Whether a Cut Is Being Treated as Certain
Hierarchical clustering produces a nested sequence of merges. A horizontal cut through its dendrogram determines how many groups are retained: cutting before a merge keeps its branches separate, while cutting above it joins them. The dendrogram records the sequence and heights of merges, but it does not by itself select a uniquely correct cut. The interpretation depends on the scale of the merge heights, the intended use, and whether nearby cut choices give materially different profiles.
Worked Example: Neighboring Cuts Give Different Levels of Detail
A hypothetical hierarchical clustering of six branches produces these merges in order: \(A\) with \(B\) at height \(1.0\); \(C\) with \(D\) at \(1.1\); \(E\) with \(F\) at \(1.2\); the \(AB\) group with the \(CD\) group at \(1.3\); and the \(ABCD\) group with \(EF\) at \(1.4\). Find the groups at cut heights \(1.25\), \(1.35\), and \(1.45\).
Method. At each cut, retain every merge that occurs at or below the cut height. Merges above the cut have not yet joined their branches.
Cut at \(1.25\): the first three merges have occurred, but the merges at \(1.3\) and \(1.4\) have not. The groups are \(\{A,B\}\), \(\{C,D\}\), and \(\{E,F\}\): three groups.
Cut at \(1.35\): the merge at \(1.3\) has also occurred, but the merge at \(1.4\) has not. The groups are \(\{A,B,C,D\}\) and \(\{E,F\}\): two groups.
Cut at \(1.45\): all five merges have occurred. There is one group, \(\{A,B,C,D,E,F\}\).
Check and trap avoided. Each answer follows from the stated merge heights. In this example the heights are close together, so the dendrogram does not show a large separation between the last few merges. A common mistake is to report “the” correct number of clusters without explaining the cut or checking whether another defensible cut changes the group profiles. Report the chosen height or group count and explain the purpose of the chosen level of detail.
The cut should be considered alongside other evidence, such as cluster sizes, interpretability, and internal indices discussed in earlier material. Those measures can help compare candidate solutions, but they do not turn a cut into a fact about nature. If two reasonable cuts tell different stories, describe the difference or qualify the conclusion.
Check Whether Assignments Are Stable
A useful grouping should not depend excessively on small perturbations in the data or analysis. Stability can be assessed by repeating the clustering under resampling or other reasonable perturbations, then comparing the resulting memberships. Labels such as “1” and “2” are arbitrary and may switch between runs, so compare groups by their members, not by their numeric labels.
The value is between 0 and 1. A value of 1 means the sets are identical; a value of 0 means they share no observations. It measures membership overlap, not whether the cluster is useful or substantively valid. When comparing repeated runs, match each reference cluster to the most overlapping group in the other run, and use a consistent comparison procedure.
Worked Example: Summarizing Assignment Consistency Across Runs
In a reference clustering, one group is \(G=\{A,B,C\}\). Five perturbed reruns produce corresponding groups \(H_1=\{A,B,C\}\), \(H_2=\{A,B,C\}\), \(H_3=\{A,B,D\}\), \(H_4=\{A,C\}\), and \(H_5=\{A,B,C,E\}\). Calculate each Jaccard similarity and their average.
Method. For each run, count the shared members and the distinct members in the combined set, then apply \(J(G,H)=|G\cap H|/|G\cup H|\).
For \(H_1\), intersection and union are both \(\{A,B,C\}\), so \(J(G,H_1)=3/3=1\). The same calculation gives \(J(G,H_2)=1\).
For \(H_3\), the intersection is \(\{A,B\}\) and the union is \(\{A,B,C,D\}\), so \(J(G,H_3)=2/4=0.5\).
For \(H_4\), the intersection is \(\{A,C\}\) and the union is \(\{A,B,C\}\), so \(J(G,H_4)=2/3\approx0.6667\).
For \(H_5\), the intersection is \(\{A,B,C\}\) and the union is \(\{A,B,C,E\}\), so \(J(G,H_5)=3/4=0.75\).
The average is \((1+1+0.5+0.6667+0.75)/5=3.9167/5\approx0.7833\). The group has substantial but imperfect overlap across these five runs.
Check and trap avoided. The average uses the five stated similarities, and the exact values sum to \(47/12\), giving an average of \(47/60\approx0.7833\). A common mistake is to compare label numbers directly; a group called “2” in one run may be the same group as “1” in another. Another is to call this score proof of validity. It is only a summary of membership consistency under the chosen perturbations and matching rule.
Interpret Cluster Labels as Exploratory Descriptions
Clusters are constructed from selected inputs and analysis choices. Their labels do not establish that the population contains those categories, that the groups have a causal explanation, or that every member shares a defining trait. For example, a hypothetical customer grouping based on claim frequency and average payment might be described as “higher-frequency, lower-average-payment observations” if that is what its profile shows. It should not be called a proven “high-risk type” without evidence that supports that interpretation.
A careful report names the variables and method, describes the observed profiles, and states relevant limitations. It is also important to note whether conclusions were sensitive to scaling, outliers, cut choices, or perturbations. If the analysis is used to support decisions about people or groups, consider whether input variables or resulting assignments could encode or amplify bias. A descriptive cluster label should not substitute for a validated decision rule.
Check feature definitions, units, missingness handling, and unusual values before interpreting a grouping.
Compare defensible scaling choices, cut choices, or perturbations; record when memberships or profiles change.
Summarize observed profiles and limitations. Treat labels as exploratory descriptions, not proven natural categories.
Check Your Understanding
For each problem, identify the relevant diagnostic and state what conclusion is justified.
- Two features are measured in dollars and years. A distance-based clustering changes markedly when dollars are converted to thousands of dollars. What should the analyst check before choosing a version? (Answer: Check whether feature units or spreads are driving the distances and whether scaling is substantively justified; the change alone does not establish which clustering is preferable.)
- A one-dimensional cluster contains values \(3,5,7\) and a verified value \(45\). Calculate its mean. (Answer: \((3+5+7+45)/4=60/4=15\).)
- A dendrogram has completed merges at heights \(0.8\), \(1.0\), \(1.2\), and \(1.6\). At a cut of \(1.1\), how many of those merges have occurred? (Answer: Two merges, those at \(0.8\) and \(1.0\); the merges at \(1.2\) and \(1.6\) have not occurred.)
- A reference cluster is \(\{P,Q,R,S\}\), and a matched group in a perturbed run is \(\{P,Q,R,T\}\). Calculate their Jaccard similarity. (Answer: The intersection has 3 members and the union has 5, so \(J=3/5=0.6\).)
- A clustering is stable across repeated runs but the resulting groups do not have clear, defensible profiles on the recorded variables. What can the analyst conclude? (Answer: Membership is consistent under the tested perturbations, but stability alone does not establish that the groups are meaningful, useful, or natural categories.)