Why Choose the Number of Clusters?
K-means requires the analyst to specify the number of clusters, \(k\), before fitting a solution. A small \(k\) gives a simple partition but may leave substantial variation within clusters. A larger \(k\) can reduce that variation, but it also creates more groups to interpret and use. The elbow method helps organize this trade-off by comparing the within-cluster sum of squares for several candidate values of \(k\).
The method is useful when the plot shows a noticeable bend: before the bend, increasing \(k\) reduces within-cluster variation substantially; after it, the additional reductions are smaller. The bend is evidence for a reasonable choice, not proof that one value of \(k\) is objectively correct. The method is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Define the Quantity on the Elbow Plot
Suppose the observations are represented by vectors \(z_1,\ldots,z_n\) in a \(p\)-dimensional feature space, using the variables and distance scale chosen for the clustering analysis. For a fixed \(k\), a k-means partition assigns each observation to one of \(k\) nonempty clusters. The centroid of cluster \(C_j\) is the component-wise mean of the observations assigned to that cluster.
Here, \(\bar z_j\) is the centroid of \(C_j\), and the minimization is over partitions into \(k\) nonempty clusters. In practice, the k-means algorithm searches for a good solution by alternating assignment and centroid-update steps. Because this procedure can stop at a local minimum, a software run may not find the global minimum \(W_k\). It is therefore common to use multiple starts and retain the solution with the lowest WCSS for each candidate \(k\).
An elbow plot places candidate values of \(k\) on the horizontal axis and their corresponding WCSS values on the vertical axis. Since WCSS depends on the feature scale and the observations being clustered, candidate values should be calculated on the same prepared data, using the same variables, transformations, and scaling. Changing the data preparation partway through the comparison makes the plotted values difficult to compare.
Why WCSS Cannot Increase with More Clusters
For the best achievable solutions, adding a cluster cannot make WCSS larger. To see why, start with a feasible \(k\)-cluster partition and consider splitting one of its clusters into two nonempty subsets. Such a split is possible when \(k<n\), because at least one cluster must contain two or more observations. If both subsets are temporarily represented by the original cluster centroid, the total squared distance for those observations is unchanged. Allowing each new subset to have its own centroid can only maintain or reduce that total, since the mean minimizes the sum of squared distances to a single point.
This is a statement about the best achievable objective values. It does not guarantee that every individual algorithm run will show a decrease: a run for \(k+1\) may end at a poor local solution, while a run for \(k\) may happen to find a better one. Multiple starts help make the observed sequence more comparable.
Read the Elbow as a Pattern of Diminishing Returns
A useful way to read the plot is to calculate the reduction in WCSS from each added cluster. Define the reduction at step \(k\) as \(W_{k-1}-W_k\). Large reductions mean that the additional cluster substantially improves the k-means objective on the chosen scale. A bend is plausible where these reductions begin to become much smaller. There is no universal numerical threshold that defines “much smaller,” so the interpretation requires judgment.
Worked Example: Comparing Successive Reductions
A hypothetical service team applies k-means to the same prepared set of numeric records for each candidate cluster count. The lowest WCSS found across multiple starts is shown below. Use the elbow method to discuss a defensible choice.
| Number of clusters \(k\) | Best WCSS \(W_k\) | Reduction from previous \(k\) |
|---|---|---|
| 1 | 150 | — |
| 2 | 62 | 150 − 62 = 88 |
| 3 | 26 | 62 − 26 = 36 |
| 4 | 12 | 26 − 12 = 14 |
| 5 | 4 | 12 − 4 = 8 |
Step 1: Calculate each reduction. Moving from \(k=1\) to \(k=2\), WCSS falls by \(150-62=88\). The next reductions are \(62-26=36\), \(26-12=14\), and \(12-4=8\).
Step 2: Read the pattern. The reductions diminish from 88 to 36 to 14 to 8. The improvement is largest when moving from one to two clusters, and each later cluster gives a smaller reduction. In this sequence, \(k=3\) is a defensible elbow choice: after \(k=3\), the reductions are 14 and then 8, suggesting less additional improvement than the earlier reductions. However, the decline is gradual rather than a sharp, unambiguous bend.
Check. The WCSS values decrease at every step, as expected. Choosing \(k=3\) is a reasoned interpretation of the diminishing returns, not a claim that another analyst must choose the same value.
Method and mistake avoided. The method is to compare successive WCSS reductions, then assess where the pattern changes. Do not mistake the largest reduction for the elbow automatically: the first reduction, 88, is largest, but that fact alone does not identify a bend.
A Bend Is a Heuristic, Not a Rule
An elbow plot is most persuasive when there is a clear change in slope. In many data sets, however, the curve bends gradually, has more than one plausible bend, or continues to fall without a distinctive change. The elbow method does not provide a formal test that certifies a particular \(k\). It helps compare the cost in WCSS of using fewer clusters with the diminishing benefit of adding more.
A percentage reduction can sometimes help describe the scale of an improvement, but it does not create a universal decision rule. For example, if WCSS falls from 80 to 60, the absolute reduction is \(80-60=20\), and the percentage reduction relative to 80 is \(20/80=0.25\), or 25%. Whether 25% is important depends on the full curve and the analysis context; it is not a standard elbow threshold.
Worked Example: When the Elbow Is Not Clear
For a different hypothetical data set, repeated starts give the following best WCSS values. The analyst wants to decide whether the curve supports a clear choice between \(k=2\) and \(k=4\).
| \(k\) | Best WCSS | Reduction from previous \(k\) |
|---|---|---|
| 1 | 100 | — |
| 2 | 70 | 100 − 70 = 30 |
| 3 | 51 | 70 − 51 = 19 |
| 4 | 39 | 51 − 39 = 12 |
| 5 | 31 | 39 − 31 = 8 |
| 6 | 25 | 31 − 25 = 6 |
Step 1: Confirm the reductions. They are 30, 19, 12, 8, and 6. For instance, the reduction from \(k=3\) to \(k=4\) is \(51-39=12\), and the reduction from \(k=5\) to \(k=6\) is \(31-25=6\).
Step 2: Evaluate the shape. The reductions steadily get smaller, but there is no single dramatic change that clearly separates large gains from negligible ones. The plot could be described as showing diminishing returns, but the table alone does not establish that \(k=2\), \(k=3\), or \(k=4\) is uniquely correct.
Step 3: State the conclusion at the right strength. A suitable conclusion is that a range of small cluster counts may be reasonable, and the elbow evidence is ambiguous. The analyst should not claim a definitive elbow based solely on these values.
Check. The reductions sum to \(30+19+12+8+6=75\), which equals the total fall from \(100\) at \(k=1\) to \(25\) at \(k=6\). This provides an arithmetic check.
Method and mistake avoided. The method is to describe the shape and strength of the bend, including when it is weak. Do not force a precise answer just because the method’s name suggests a sharp elbow must exist.
Compare Reliable Candidate Fits
K-means can converge to different local solutions from different starting centroids. Thus, a candidate’s WCSS should not be treated as a stable benchmark if it came from only one start, especially when a poor initialization is possible. For each \(k\), a practical comparison uses several starts and records the lowest WCSS found. This does not prove that the global minimum has been reached, but it reduces dependence on one initialization.
Worked Example: Choosing the Best Result Across Starts
A hypothetical analyst compares \(k=2\) and \(k=3\). Three starts for \(k=2\) produce WCSS values 84, 80, and 82. Three starts for \(k=3\) produce 61, 65, and 59. Which results should be plotted, and what can the analyst conclude?
Step 1: Retain the best run for each \(k\). For \(k=2\), the minimum is \(\min(84,80,82)=80\). For \(k=3\), the minimum is \(\min(61,65,59)=59\).
Step 2: Calculate the observed reduction. The best observed WCSS falls by \(80-59=21\) when moving from \(k=2\) to \(k=3\). Relative to the \(k=2\) value, this is \(21/80=0.2625\), or 26.25%.
Step 3: State what the comparison does and does not show. Plot 80 for \(k=2\) and 59 for \(k=3\), rather than plotting one arbitrarily selected run. These results show an observed WCSS reduction of 21. By themselves, they do not show whether \(k=3\) is an elbow: that judgment requires the pattern across additional candidate values of \(k\).
Check. The selected values are no larger than the other starts at their respective \(k\), and \(59<80\). The arithmetic is consistent with the nonincreasing objective.
Method and mistake avoided. The method is to use the best WCSS found for each candidate after repeated starts. Do not compare a best run for one \(k\) with a poor run for another, or conclude that one reduction defines an elbow without examining the surrounding curve.
Common Calculation Trap
A common error is to confuse the size of a WCSS reduction with the change between successive reductions. In the first worked example, the reductions are 88, 36, 14, and 8. It would be incorrect to say that the largest change is from 36 to 14: the absolute changes between successive reductions are \(88-36=52\), \(36-14=22\), and \(14-8=6\). The largest of those changes is 52, between the reductions of 88 and 36.
But this arithmetic correction does not turn the largest change into a mechanical selection rule. The elbow method assesses the overall curve and its diminishing returns. For that example, \(k=3\) remains a defensible, though not uniquely determined, choice because the reductions diminish from 88 to 36 to 14 to 8. A sound explanation gives the correctly calculated pattern and acknowledges the judgment involved.
Practical Interpretation
Before interpreting an elbow plot, check that the candidate solutions were fitted to the same observations and the same prepared variables. Because WCSS is based on squared distances, it is sensitive to the scale and representation of the features. A change in scaling or variable selection can alter the curve, so the elbow applies to the chosen analysis setup rather than to the data in the abstract.
When reporting a choice, describe both the evidence and its limitation. For example: “The reductions in WCSS become smaller after \(k=3\), so three clusters are a reasonable elbow-based choice; the curve does not identify a uniquely correct number.” If the curve has no visible bend, say so rather than inventing one. An elbow plot is a tool for transparent judgment, not a substitute for judgment.
Check Your Understanding
For each question, use the stated WCSS values as the best values found for the candidate cluster counts.
- A fit has WCSS values 90, 50, 30, and 22 for \(k=1,2,3,4\). Calculate the three successive reductions. (Answer: 40, 20, and 8.)
- A second fit has WCSS values 120, 84, 60, 45, and 37 for \(k=1,2,3,4,5\). What are the successive reductions, and is there a sharply established elbow? (Answer: 36, 24, 15, and 8; the reductions diminish, but the sequence does not establish a uniquely sharp elbow.)
- For \(k=2\), three starts give WCSS 73, 69, and 71; for \(k=3\), they give 54, 58, and 52. Which values should be used as the best observed WCSS, and what is their reduction? (Answer: 69 for \(k=2\), 52 for \(k=3\), a reduction of 17.)
- For WCSS values 160, 92, 55, and 40 at \(k=1,2,3,4\), calculate the reductions and give a cautious elbow interpretation. (Answer: reductions are 68, 37, and 15; \(k=3\) is a defensible choice because the gains have diminished, but the elbow is not uniquely determined.)
- A colleague says that the WCSS at \(k=4\) is 28 and at \(k=5\) is 34, so adding a cluster worsened the best achievable objective. What is the appropriate response? (Answer: the best achievable WCSS cannot increase with \(k\); investigate whether inconsistent data preparation, different fitting conditions, or a suboptimal local solution caused the reported increase.)