Why Cluster Shape Matters
Clustering assigns observations to groups using their measured features, without using a response variable to define those groups. The method can produce very different partitions depending on the geometry it favors. A useful choice is not simply the one with the most familiar name: it is the one whose distance measure and grouping rule are compatible with the structure being sought.
K-means favors groups that are compact around a mean. Hierarchical clustering builds nested groups from pairwise distances, and the linkage rule determines how distances between groups are measured. These distinctions matter when clusters are unequal in size, elongated, connected by intermediate observations, or contaminated by an outlier. As in the earlier material on scaling, feature units matter: distance calculations should use features placed on a defensible common scale when their units or ranges differ.
The standard Euclidean version of k-means uses squared Euclidean distance. Its objective gives each observation a contribution proportional to its squared distance from its cluster centroid. Hierarchical methods need a rule for distance between clusters, called linkage. Single linkage uses the smallest pairwise distance between observations in the two clusters; complete linkage uses the largest; average linkage uses the average of all cross-cluster pairwise distances. Ward linkage merges clusters to produce the smallest increase in within-cluster sum of squares, and is therefore closely aligned with a compact-cluster objective.
Neither approach automatically discovers a uniquely correct grouping. The number of clusters, the features, their scales, the distance, and the linkage all affect the result. The examples below focus on how to reason about those choices, not on using cluster membership as evidence of an outcome or causal effect.
Why the K-Means Centroid Is the Mean
The centroid in k-means is not an arbitrary representative. For a fixed cluster, its mean minimizes the sum of squared distances to a representative point. In one dimension, let the cluster values be \(x_1,\ldots,x_m\), and let \(c\) be any proposed center. Write \(\bar{x}=m^{-1}\sum_i x_i\). Expanding around the mean gives the identity below.
To see why, write \(x_i-c=(x_i-\bar{x})+(\bar{x}-c)\), square, and sum. The cross-term is \(2(\bar{x}-c)\sum_i(x_i-\bar{x})=0\), because deviations from the mean sum to zero. The remaining additional term \(m(\bar{x}-c)^2\) is nonnegative and is zero only at \(c=\bar{x}\). Applying the same argument coordinate by coordinate establishes the result in multiple dimensions. This also explains a limitation: a mean can be pulled strongly by an extreme observation.
Worked Example: Compact Groups of Unequal Size
A hypothetical portfolio segmentation uses one already-scaled numeric feature. The observed values, in ascending order, are \(0,1,2,8,9,10\). For \(K=2\), compare the k-means objective across every possible contiguous split, then compare the result with the intuitive separation between the two compact groups.
Method. In one dimension, a globally optimal k-means partition can be taken as contiguous in sorted order: if two clusters have ordered means, assigning a value between the means to the farther cluster cannot improve squared distance. Thus it is sufficient here to calculate the five possible cuts between adjacent observations.
For the cut after the first value, the clusters are \(\{0\}\) and \(\{1,2,8,9,10\}\). The second cluster has mean \(30/5=6\) and sum of squares \(25+16+16+9+16=82\)? Check the terms carefully: \((1-6)^2=25\), \((2-6)^2=16\), \((8-6)^2=4\), \((9-6)^2=9\), and \((10-6)^2=16\). Their sum is \(70\), so the total objective is \(70\).
For the cut after the second value, \(\{0,1\}\) has mean \(0.5\) and sum of squares \(0.25+0.25=0.5\). The other cluster \(\{2,8,9,10\}\) has mean \(29/4=7.25\), and sum of squares \(27.5625+0.5625+3.0625+7.5625=38.75\). The total is \(39.25\).
For the cut after the third value, \(\{0,1,2\}\) has mean \(1\) and sum of squares \(1+0+1=2\). The cluster \(\{8,9,10\}\) has mean \(9\) and sum of squares \(1+0+1=2\). The total is \(4\).
The data are symmetric around \(5\), so the cut after the fourth has the same objective as the cut after the second, \(39.25\), and the cut after the fifth has the same objective as the cut after the first, \(70\). The minimum is \(4\), attained by \(\{0,1,2\}\) and \(\{8,9,10\}\).
Check. Each selected cluster has three observations one unit apart, so each contributes \(2\) to within-cluster sum of squares. The combined value \(2+2=4\) agrees with the direct calculation. The mistake to avoid is reporting \(102\) for either extreme cut: the five-point cluster at the first cut has mean \(6\) and sum of squares \(70\), not \(102\).
This example shows why k-means can describe compact groups even when their sizes differ: the objective does not require equal numbers of observations per cluster. It does, however, reward a partition by total squared error. A larger, more dispersed group may contribute substantially to that total and may be divided if doing so reduces the objective enough. K-means also requires a specified \(K\); comparing shape suitability is separate from deciding how many clusters to use.
How Linkage Changes Hierarchical Clusters
A hierarchical method does not minimize the k-means objective unless a particular method, such as Ward linkage, is used. Instead, at each agglomerative step it merges the pair of clusters with the smallest linkage distance. Single linkage can join groups through a short chain of nearby observations, while complete linkage demands that even the most separated cross-cluster pair be close. Average linkage lies between these two rules in how it summarizes cross-cluster distances.
Worked Example: A Bridge Between Two Compact Groups
Consider five two-dimensional observations: \(A_1=(0,0)\), \(A_2=(0,1)\), \(B_1=(3,0)\), \(B_2=(3,1)\), and a bridge point \(M=(1.5,0.5)\). Use Euclidean distance and compare agglomerative single and complete linkage at a distance threshold of \(1.6\). A threshold cut retains merges made at distances no greater than that value.
Method. First calculate the within-pair distances: \(d(A_1,A_2)=1\) and \(d(B_1,B_2)=1\), so each pair has merged by threshold \(1.6\). The distance from \(M\) to each of the four other points is \(\sqrt{1.5^2+0.5^2}=\sqrt{2.5}\), approximately \(1.5811\).
Under single linkage, the distance from \(M\) to the \(A\) pair is the smallest of its distances to the pair members, namely \(\sqrt{2.5}\). The same is true for the \(B\) pair. At threshold \(1.6\), the bridge is connected to both compact pairs: the resulting hierarchy has joined all five observations into one cluster.
Under complete linkage, the distance from \(M\) to the \(A\) pair is the largest of its distances to the two members. Both distances are \(\sqrt{2.5}\), so that linkage is approximately \(1.5811\); the bridge can join the \(A\) pair. The complete-linkage distance between this enlarged cluster and the \(B\) pair is the largest cross-cluster distance. The distance from either \(A\) point to the farthest \(B\) point is \(\sqrt{3^2+1^2}=\sqrt{10}\), approximately \(3.1623\), which exceeds \(1.6\). Thus \(B_1,B_2\) remain a separate cluster at the threshold.
Check. At the stated cut, single linkage produces one connected cluster, while complete linkage produces two. The bridge is close to each group individually, but the compact groups remain far apart under the complete-linkage rule. The example illustrates why single linkage is useful for connected or irregular shapes but can also exhibit chaining: a sequence of short links can connect groups whose endpoints are far apart.
Average linkage would compare the mean of all distances across two clusters, rather than only the shortest or longest. It may be a reasonable compromise, but it does not remove the need to think about shape: a mean cross-distance can still conceal a long, thin structure. Ward linkage is more directly suited to compact groups because it chooses the merge with the smallest increase in within-cluster sum of squares. It is not a general solution for arbitrary shapes, and its usual interpretation depends on squared Euclidean geometry.
Outliers and the Objective Being Optimized
An outlier may form a singleton cluster, pull a centroid away from the main observations, or change which observations are grouped together. In k-means, the squared distance makes an observation far from its centroid particularly influential. In hierarchical clustering, an outlier can remain isolated until a high merge distance, but the linkage rule determines how that distance is evaluated. Single linkage uses distance to the nearest member; complete linkage uses distance to the farthest member. Neither rule makes an outlier harmless.
Worked Example: An Extreme Observation in a One-Dimensional Group
A hypothetical exposure score has values \(0,1,2,3,4,5,20\). With \(K=2\), find the globally best contiguous k-means split. Then describe how single linkage treats the final observation when the distance threshold is \(14.9\).
Method. For a set of values, compute its sum of squares around the mean using \(\sum x_i^2-(\sum x_i)^2/m\), where \(m\) is the number of values. The total sum of squares of all seven values is \(455-35^2/7=455-175=280\). For a two-cluster split, the within-cluster objective is the total sum of squares minus the between-cluster contribution \(m_1m_2(\bar{x}_1-\bar{x}_2)^2/7\).
For cuts after the first, second, third, fourth, and fifth observations, the within-cluster objectives are respectively \(1505/6\), \(1339/6\), \(196\), \(497/3\), and \(245/2\). For example, after the third observation, the groups are \(\{0,1,2\}\) and \(\{3,4,5,20\}\). Their means are \(1\) and \(8\), and their within-cluster sums of squares are \(2\) and \(25+16+9+144=194\), giving \(196\).
For the cut after the sixth observation, the first cluster \(\{0,1,2,3,4,5\}\) has mean \(15/6=2.5\) and sum of squares \(2.5^2+1.5^2+0.5^2+0.5^2+1.5^2+2.5^2=17.5\). The singleton \(\{20\}\) contributes zero. Comparing all six values, \(17.5\) is the minimum, so the global k-means split isolates \(20\).
For single linkage, the distance from \(20\) to its nearest observation is \(20-5=15\). At threshold \(14.9\), the consecutive observations \(0\) through \(5\) have joined through distances of \(1\), but \(20\) has not joined them. The resulting groups are therefore \(\{0,1,2,3,4,5\}\) and \(\{20\}\).
Check. The k-means objective for the six-point group plus singleton is \(17.5+0=17.5\), which is less than the objective for every other contiguous cut listed. The linkage calculation also checks against the gap: \(15>14.9\). The squared-distance objective makes the isolated value expensive to keep with the main group, so k-means can favor giving it its own cluster.
Choosing a Method for the Geometry
A practical comparison should start with the geometry the analysis is intended to represent. K-means and Ward linkage are natural candidates when groups are expected to be compact around their means and Euclidean distances are meaningful. Single linkage may be appropriate when connected, elongated, or irregular shapes are plausible, but its chaining behavior deserves attention. Complete linkage tends to limit the diameter of clusters, making it less likely to join groups through a narrow bridge; the trade-off is that an elongated, genuinely coherent group may be split.
| Method or choice | What it favors | What to watch for |
|---|---|---|
| K-means | Clusters with small squared distances to their centroids | Outliers, noncompact shapes, and dependence on a specified \(K\) |
| Single linkage | Groups connected by short nearest-neighbor links | Chaining through bridge observations |
| Complete linkage | Groups whose farthest cross-cluster observations are relatively close | Splitting elongated groups; sensitivity to distant points |
| Average linkage | Groups with small average cross-cluster distances | Its average can mask very short or very long connections |
| Ward linkage | Merges causing small increases in within-cluster sum of squares | Compact Euclidean clusters rather than arbitrary shapes |
Before comparing results, confirm what the distance means for the chosen features. A feature measured in thousands can dominate one measured between zero and one if the data are not scaled appropriately. Scaling should be justified by the task and performed without using an outcome to define unsupervised groups. A numerical feature that is merely a code or label should not automatically be treated as a meaningful Euclidean coordinate.
Finally, a partition is a description of similarity under particular choices, not evidence that clusters differ in a future outcome. Cluster interpretation and outcome evidence are distinct questions. When a clustering will inform a risk decision, examine the feature definitions, distance, scaling, outliers, and linkage alongside the resulting groups. This method-choice reasoning is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, identify the relevant objective or linkage rule before choosing a partition.
- For the values \(1,2,3,9,10,11\), a two-cluster k-means fit is proposed as \(\{1,2,3\}\) and \(\{9,10,11\}\). Compute its within-cluster sum of squares and state why it is a compact split. (Answer: \(4\), with \(2\) in each cluster.)
- Two clusters each contain two points: \(A=\{(0,0),(0,1)\}\) and \(B=\{(4,0),(4,1)\}\). Find their single-linkage and complete-linkage distances using Euclidean distance. (Answer: single linkage \(4\); complete linkage \(\sqrt{17}\), approximately \(4.1231\).)
- On the line, observations are \(0,1,2,3,4\). Under single linkage, what is the largest threshold at which all five have not yet become one cluster, assuming merges occur at distances no greater than the threshold? (Answer: any threshold less than \(1\) leaves five singletons; at threshold \(1\), all five connect, so there is no largest threshold below \(1\).)
- A group consists of \(0,1,2,3\), and a fifth observation is \(15\). Calculate the sum of squares for the four-point group around its mean, then give the single-linkage distance from \(15\) to that group. (Answer: mean \(1.5\), sum of squares \(5\), and single-linkage distance \(12\).)
- A proposed segmentation has a long chain of near-neighbor observations connecting two dense clouds. Which linkage is most likely to join the clouds early through that chain, and which linkage is more likely to keep them separate until their farthest cross-cluster points are close? (Answer: single linkage is most likely to join through the chain; complete linkage waits for the farthest cross-cluster distance to be small.)