When Clusters Are Unequal, Overlapping, Or Non-Spherical

Why Cluster Shape and Density Matter

Clustering methods do not discover a unique set of natural groups simply because data have been supplied. Each method applies a particular rule for deciding which observations belong together. If that rule does not match the structure of the data, a mathematically valid result can still be a poor description of the problem.

For example, k-means assigns each observation to its nearest centroid and updates the centroids to reduce within-cluster squared distances. This is most suitable when groups are reasonably compact and separated. It can struggle when a meaningful group is elongated, when groups overlap, or when a compact group lies near a much more dispersed one. Hierarchical methods also make choices through their linkage rules: single linkage can connect long chains, while complete and Ward linkage favor more compact groups in different ways.

These are limitations, not automatic reasons to reject a method. Cluster meaning depends on the analysis purpose and the observed data. A candidate should diagnose the mismatch, state what the method is doing, and consider an alternative only when it better represents the intended structure. This reasoning is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: A cluster is non-spherical when its observations form a shape that is not well described as a compact ball around a centroid, such as a long band or curved arc. Clusters have unequal density when the local concentration of observations differs substantially between groups. Clusters overlap when their feature values occupy some of the same region, so membership cannot be separated perfectly from those features alone.

What Centroid-Based Clustering Prefers

For observations \(x_1,\ldots,x_n\) and a proposed partition into \(K\) clusters \(C_1,\ldots,C_K\), k-means minimizes the within-cluster sum of squares:

\operatorname{WSS}=\sum_{k=1}^{K}\sum_{i\in C_k}\lVert x_i-\bar{x}_k\rVert^2, \qquad \bar{x}_k=\frac{1}{|C_k|}\sum_{i\in C_k}x_i.

Here \(\bar{x}_k\) is the centroid of cluster \(k\), and the distance is Euclidean distance in the feature space used for clustering. The objective penalizes observations that are far from their assigned centroid, with large deviations receiving especially strong weight because they are squared. A long, spread-out group can therefore have a large WSS even if it is a meaningful group. Dividing that group into two parts may reduce WSS more than keeping it intact and assigning a separate compact group its own centroid.

This preference is sometimes described using a probabilistic interpretation. If observations are treated as arising from groups with the same spherical covariance, equal prior probabilities, and hard assignment to one group, the log-likelihood of the assignments differs from negative WSS only by a positive scale factor and constants. Under that simplified model, smaller WSS corresponds to larger likelihood. This interpretation explains the geometry k-means favors; it is not a claim that real clusters must satisfy those assumptions.

Important distinction: K-means does not require all clusters to have the same number of observations. It can return clusters of different sizes. Its main geometric preference is for groups whose points are close to their respective centroids; unequal size or spread can make that preference a poor match for the intended grouping.

Worked Example: A Long Group Competes With a Compact Group

A hypothetical feature set contains seven two-dimensional observations. Five form a long horizontal band, \(A=\{(-10,0),(-5,0),(0,0),(5,0),(10,0)\}\), and two form a compact group, \(B=\{(0,8),(0,9)\}\). Compare the WSS for the intuitive partition \(A,B\) with a different two-cluster partition. Use Euclidean distance and squared distance.

Method. Calculate each partition’s centroid and sum of squared distances. This tests whether the k-means objective necessarily preserves the intuitive elongated group.

For \(A\), the centroid is \((0,0)\). The squared distances are \(100,25,0,25,100\), so its WSS is \(250\). For \(B\), the centroid is \((0,8.5)\), and the squared distances are \(0.25\) and \(0.25\), giving WSS \(0.5\). Thus the intuitive partition has total WSS \(250+0.5=250.5\).

Now consider \(C=\{(-10,0),(-5,0)\}\) and \(D=\{(0,0),(5,0),(10,0),(0,8),(0,9)\}\). The centroid of \(C\) is \((-7.5,0)\), so its WSS is \(6.25+6.25=12.5\).

The centroid of \(D\) is \((3,3.4)\): its mean \(x\)-coordinate is \(15/5=3\), and its mean \(y\)-coordinate is \(17/5=3.4\). For the \(x\)-coordinates, the sum of squares is \(125\), so the sum of squared deviations is \(125-5(3^2)=80\). For the \(y\)-coordinates, the sum of squares is \(145\), so the sum of squared deviations is \(145-5(3.4^2)=145-57.8=87.2\). Therefore \(D\)'s WSS is \(80+87.2=167.2\), and the alternative partition has total WSS \(12.5+167.2=179.7\).

Check. The alternative has lower WSS: \(179.7<250.5\). This does not prove it is the globally optimal k-means solution; it shows that even this alternative improves the objective over the intuitive grouping. The method’s common trap is to read a lower WSS as proof that the resulting groups are more meaningful. The calculation only establishes which partition better satisfies the WSS objective.

How Linkage Choices Affect Unequal or Elongated Groups

Hierarchical clustering builds groups by merging observations or existing groups according to a linkage rule. The distance between two groups is therefore not a single universal quantity. Single linkage uses the smallest cross-group distance; complete linkage uses the largest; average linkage uses the average of cross-group distances; Ward linkage chooses merges based on their increase in within-cluster variation.

Single linkage can preserve a curved or elongated structure because successive nearby observations can join. But it is also vulnerable to chaining: a sequence of intermediate observations may link two groups that otherwise appear well separated. Complete linkage resists such chains by considering the most distant pair, but it tends to favor compact groups and can split long structures. Ward linkage also favors compactness because it minimizes the increase in within-cluster variation at each merge. None of these rules is universally best.

Worked Example: A Linkage Cut Changes the Interpretation

Consider two one-dimensional groups, \(A=\{0,0.1,0.2\}\) and \(B=\{0.6,0.7,0.8\}\). Calculate the single-linkage distance, complete-linkage distance, and Ward increase for merging the groups. Then decide whether a cut at distance \(0.5\) merges them under the first two rules.

Method. For single linkage, take the minimum cross-group distance. For complete linkage, take the maximum. For Ward linkage, calculate the increase in total within-cluster sum of squares.

The closest cross-group pair is \(0.2\) and \(0.6\), so the single-linkage distance is \(0.6-0.2=0.4\). The farthest pair is \(0\) and \(0.8\), so the complete-linkage distance is \(0.8\). At a cut of \(0.5\), single linkage joins the groups because \(0.4\leq0.5\), while complete linkage does not because \(0.8>0.5\).

For Ward linkage, each group has size \(3\), and their centroids are \(0.1\) and \(0.7\). The increase from merging two groups is \( \frac{n_A n_B}{n_A+n_B}(\bar{x}_A-\bar{x}_B)^2\). Here it equals \(\frac{3\cdot3}{6}(0.1-0.7)^2=1.5(0.36)=0.54\). To check directly, the merged centroid is \(0.4\); the six squared deviations sum to \(0.16+0.09+0.04+0.04+0.09+0.16=0.58\). The separate-group WSS is \(0.02+0.02=0.04\), and \(0.58-0.04=0.54\), as required.

Check. Single linkage joins at this cut, while complete linkage does not; Ward’s merge adds \(0.54\) to WSS. The common mistake is to treat a dendrogram cut as though all linkage rules used the same notion of group distance. The merge criterion must be interpreted in the context of the selected linkage.

Density Differences and Overlap

Density-based methods define groups through local neighborhoods rather than distance to a centroid. In DBSCAN, a point is a core point if its neighborhood of radius \(\varepsilon\) contains at least the specified number of observations, MinPts, counting the point itself. Core points connected through neighboring core points form a cluster; nearby non-core points may be border points, and points not assigned to a cluster are treated as noise.

This approach can identify irregularly shaped groups and flag isolated observations without requiring a centroid for each group. Its important limitation is that a single \(\varepsilon\) may not suit groups with very different densities. A small value can omit a sparse but meaningful group; a larger value can connect groups that should remain separate. Alternatives such as OPTICS or HDBSCAN examine density structure across multiple neighborhood scales rather than relying on one fixed radius, though their results still require interpretation.

Worked Example: One Neighborhood Radius Cannot Preserve Both Groups

A hypothetical one-dimensional feature has a dense set \(D=\{0,0.1,0.2,0.3,0.4\}\) and a more widely spaced set \(S=\{1.0,1.6,2.2,2.8,3.4\}\). Run the DBSCAN logic with MinPts \(=3\), first using \(\varepsilon=0.25\), then \(\varepsilon=0.65\). Include the observation itself when counting neighbors.

Method. Check neighborhood counts and whether points are density-connected. The purpose is to diagnose how the radius trades off between recognizing sparse points and keeping groups separate.

At \(\varepsilon=0.25\), each point in \(D\) has at least two other points within the radius. For example, \(0.0\) has neighbors \(0.1\) and \(0.2\), and \(0.4\) has neighbors \(0.2\) and \(0.3\). Thus all five dense points meet MinPts and form a connected cluster. Each point in \(S\) is \(0.6\) from its nearest other point in \(S\), so each has only itself in its neighborhood. None meets MinPts; these points are noise at this setting.

At \(\varepsilon=0.65\), consecutive points in \(S\) are only \(0.6\) apart. The first sparse point, \(1.0\), has \(0.4\), \(1.0\), and \(1.6\) in its neighborhood, so it is core. The interior sparse points also have at least three neighbors when the point itself is counted. The final point, \(3.4\), is not core, but it is within \(0.65\) of the core point \(2.8\), so it is a border point. Crucially, \(0.4\) and \(1.0\) are also only \(0.6\) apart. The radius therefore connects the dense and sparse sets into one density-connected cluster.

Check. The smaller radius identifies \(D\) but leaves \(S\) as noise; the larger radius joins both sets. In this example, a single DBSCAN radius does not both include the sparse group and keep it separate from the dense one. The trap is to conclude from one parameter setting that no density-based method could work; the correct conclusion is that this fixed-radius configuration does not represent both density scales well.

Overlap creates a different challenge. If two populations occupy some of the same feature space, no hard boundary can assign every observation correctly based only on those features. A Gaussian mixture model represents each group with a probability distribution, often a multivariate normal distribution, and calculates a probability of membership for each observation. Unlike a hard k-means assignment, this can preserve uncertainty about points in a shared region.

Definition: In a Gaussian mixture model with \(K\) components, an observation has density \(f(x)=\sum_{k=1}^K\pi_k f_k(x)\), where \(f_k\) is the density for component \(k\), \(\pi_k\) is its mixing proportion, and the proportions are nonnegative and sum to one. The posterior membership probability for component \(k\) is \(\tau_k(x)=\pi_k f_k(x)/f(x)\). A hard label can be created by choosing the component with the largest \(\tau_k(x)\), but the probabilities retain information about ambiguity.

Worked Example: Quantifying an Overlapping Assignment

Suppose a hypothetical one-dimensional feature is modeled by two normal components with equal prior proportions \(0.5\), means \(0\) and \(2\), and variance \(1\) for each component. For an observation \(x=0.5\), calculate both posterior membership probabilities. Use the normal density \(f(x)=\frac{1}{\sqrt{2\pi}}\exp[-(x-\mu)^2/2]\) for variance \(1\).

Method. Calculate each component’s density, multiply by its prior proportion, and divide by the sum of the weighted densities. The shared normalizing constant cancels in the posterior ratio.

For the component with mean \(0\), \((0.5-0)^2/2=0.125\), so its density is approximately \(0.3989e^{-0.125}=0.3521\). For the component with mean \(2\), \((0.5-2)^2/2=1.125\), giving density approximately \(0.3989e^{-1.125}=0.1295\). The weighted values are \(0.5(0.3521)=0.17605\) and \(0.5(0.1295)=0.06475\); their sum is \(0.24080\).

The posterior probability for the first component is \(0.17605/0.24080\approx0.7311\); for the second it is \(0.06475/0.24080\approx0.2689\). The probabilities sum to \(1.0000\). The hard assignment would choose the first component, but a probability of \(0.7311\) is not certainty. The common mistake is to report the chosen label as though the observation were unambiguously assigned.

A Practical Method-Selection Checklist

Use the structure and purpose of the analysis to guide method choice. A method that captures an unusual shape may be less useful if the intended output requires every observation to have a group label; a method that produces probabilities may be helpful when uncertainty itself matters.

Inspect shape and scale.
Look for long bands, curved patterns, groups with very different spreads, and regions where observations from candidate groups overlap. Consider whether features need scaling before distances are interpreted.

State the method’s rule.
For k-means, describe nearest-centroid assignment and WSS minimization. For hierarchical clustering, name the linkage. For DBSCAN, specify the neighborhood radius and MinPts. Do not treat the algorithm name alone as an explanation.

Match the rule to the observed structure.
Consider k-means for compact groups; density-based methods for irregular shapes with meaningful density separation; and a probabilistic mixture when overlapping distributional components and uncertain membership are relevant.

Check whether the output answers the question.
Review profiles, assignments, and sensitivity to choices. A lower objective or a visually neat plot does not by itself establish that the clusters are substantively useful.

For hierarchical results, also ask whether a different linkage would alter the merge pattern or the interpretation of a cut. For density-based results, check whether a radius that includes sparse observations also joins groups that should remain distinct. For overlapping groups, report that membership is uncertain rather than implying a perfectly clear separation.

Common Mistakes and Key Takeaways

Assuming k-means requires equal cluster sizes. It does not; its objective instead favors low squared distance to centroids and can penalize dispersed or elongated groups.
Calling every non-spherical pattern a clustering failure. First decide whether the shape is meaningful for the question and whether the selected distance and features represent it appropriately.
Treating linkage rules as interchangeable. Single, complete, average, and Ward linkage define inter-group separation differently and can yield different dendrograms.
Assuming one DBSCAN setting is universally suitable. A radius that handles one density may fail at another, and parameter sensitivity is part of the diagnosis.
Interpreting a hard assignment as certainty. Overlapping groups can support a most-likely label while still having substantial uncertainty.
Choosing an alternative solely because it gives a preferred picture. State the method’s assumptions and verify that its output is relevant to the intended use.

Key takeaway: Choose and interpret a clustering method by how its assignment rule responds to shape, density, and overlap. K-means and hierarchical methods can be useful even when groups are not identical in size, but their geometric preferences must be checked; density-based and probabilistic alternatives address different structures and have their own limitations.

Check Your Understanding

For each problem, identify the method behavior or calculation that best supports a defensible interpretation.

A candidate two-cluster partition has WSS \(84\). Another partition has WSS \(67\), but it divides a long operational segment and groups its end with a compact group. Which partition has the smaller k-means objective, and what does that comparison not establish? (Answer: The second partition has the smaller objective, \(67<84\); it does not establish that its groups are more meaningful.)
Two one-dimensional groups have ranges \([0,0.3]\) and \([0.8,1.1]\). What are the single- and complete-linkage distances between them? (Answer: Single linkage is \(0.8-0.3=0.5\); complete linkage is \(1.1-0=1.1\).)
In DBSCAN with MinPts \(=4\), a point has three other observations within \(\varepsilon\). Is it a core point when the point itself is counted, and how many observations are in its neighborhood? (Answer: Yes; its neighborhood contains \(4\) observations including itself.)
A two-component mixture gives an observation posterior probabilities \(0.42\) and \(0.58\). Which component receives the hard assignment, and what is an important qualification? (Answer: The second component; the assignment is uncertain because its posterior probability is only \(0.58\).)
A sparse chain becomes connected to a dense group only after DBSCAN’s radius is increased. What limitation should be reported, and what kind of alternative could be investigated? (Answer: A single fixed radius may not represent both density scales without merging the groups; investigate a multi-scale density method such as OPTICS or HDBSCAN.)

What You'll Learn