Choosing K In A Worked Clustering Case

Choosing K Requires More Than One Diagnostic

K-means requires the analyst to specify the number of clusters, $K$, before fitting the model. A larger $K$ can always divide the observations into more groups and reduce within-cluster variation, but that reduction alone does not show that the extra groups are useful. A practical choice weighs the improvement in compactness against the separation and interpretability of the resulting clusters.

This tutorial compares candidate values using within-cluster sum of squares (WCSS), an elbow plot, average silhouette widths, and cluster sizes. These diagnostics answer related but different questions. WCSS measures compactness; silhouettes assess how well observations fit their assigned clusters relative to alternatives; cluster sizes show whether the solution divides the data into usable groups. No one diagnostic automatically determines the answer.

Definition: For $n$ numerical observations $z_1,\ldots,z_n$ and a k-means partition into $K$ nonempty clusters $C_1,\ldots,C_K$, the within-cluster sum of squares is $$ W_K=\sum_{k=1}^{K}\sum_{i\in C_k}\lVert z_i-\bar{z}_k\rVert^2, $$ where $\bar{z}_k$ is the centroid (the coordinate-wise mean) of cluster $C_k$. The distance and the input features must be held consistent when comparing candidate $K$ values.

K-means assigns each observation to a cluster and estimates each centroid to minimize WCSS, subject to the selected value of $K$. In a multiple-feature data set, the squared distance is the sum of squared coordinate differences. Feature scaling and the choice of inputs therefore affect the result; the candidate solutions being compared should use the same input representation and scaling.

What the Elbow Can and Cannot Show

For a fixed data set and feature representation, WCSS cannot increase as $K$ increases, if the solutions are optimized globally. Adding clusters gives the model more flexibility. In practice, k-means can settle at a local solution, so the computed WCSS for a particular run may not attain the global minimum. A fair comparison uses a consistent fitting procedure, such as multiple initializations and the best WCSS found for each candidate $K$.

An elbow plot places $K$ on the horizontal axis and WCSS on the vertical axis. A noticeable bend may indicate that adding clusters beyond that point yields smaller incremental reductions in WCSS. The bend is a visual judgment, not a hypothesis test or a uniquely defined optimum. Some data sets have no clear elbow; in that case, the other diagnostics and the purpose of the analysis matter more.

Hold the data representation fixed.
Compare the same observations, numerical features, and scaling across candidate values of $K$.

Review the WCSS reductions.
Record how much WCSS falls when moving from one candidate $K$ to the next, and look for a reduction that becomes less substantial.

Check silhouettes and sizes.
Ask whether observations are well assigned and whether the resulting clusters are large enough to be useful.

State a qualified choice.
Explain which candidate offers a reasonable balance and mention any diagnostic that gives a reason for caution.

Silhouette Widths as a Separation Check

For observation $i$, let $a(i)$ be its average distance to the other observations in its own cluster. For each other cluster, calculate the average distance from $i$ to the observations in that cluster; let $b(i)$ be the smallest of those averages. The silhouette width is

s(i)=\frac{b(i)-a(i)}{\max\{a(i),b(i)\}}.

A silhouette is between $-1$ and $1$. A value near 1 means the observation is, on average, closer to its own cluster than to the nearest alternative. A value near 0 indicates little separation, and a negative value means the observation is, on average, closer to another cluster. For a singleton cluster, the silhouette is conventionally set to 0 because there are no other observations in that cluster from which to calculate $a(i)$.

The mean silhouette for a candidate solution is the arithmetic average of the observation-level silhouettes. It summarizes assignment quality, but it does not show whether one small cluster is poorly assigned. Cluster-specific means and sizes can reveal that detail. Silhouette widths also depend on the distance measure and feature representation, so compare candidates under the same choices.

Justification: WCSS is nonincreasing as $K$ grows because splitting a cluster can preserve its original centroid-based fit or improve it. If a cluster with centroid $\bar{z}$ is divided into two nonempty groups with sizes $n_1,n_2$ and centroids $\bar{z}_1,\bar{z}_2$, the sum of squares for the original cluster equals the two new within-cluster sums plus $$ n_1\lVert\bar{z}_1-\bar{z}\rVert^2+n_2\lVert\bar{z}_2-\bar{z}\rVert^2. $$ The added terms are nonnegative. Thus the split cannot increase the within-cluster sum of squares; optimizing the new centroids and assignments cannot make the best achievable value larger.

Worked Comparisons

Worked Example: A Clear Elbow and a Weak Extra Cluster

A hypothetical analyst clusters six one-dimensional measurements $0,1,2,8,9,10$, using ordinary Euclidean distance. Consider $K=1,2,3,$ and $4$. Compare WCSS and silhouettes, then recommend a practical value.

Method. First calculate WCSS for the stated partitions. Then compare the $K=2$ partition $\{0,1,2\},\{8,9,10\}$ with the $K=3$ partition $\{0,1\},\{2\},\{8,9,10\}$. The example illustrates why a lower WCSS does not automatically justify a larger $K$.

At $K=1$, the centroid is $(0+1+2+8+9+10)/6=5$. Therefore,

$$ W_1=25+16+9+9+16+25=100. $$

At $K=2$, the centroids are $1$ and $9$. The within-cluster sum of squares is $(1+0+1)+(1+0+1)=4$. At $K=3$, the centroids are $0.5$, $2$, and $9$, giving $W_3=(0.25+0.25)+0+(1+0+1)=2.5$. At $K=4$, use $\{0,1\},\{2\},\{8,9\},\{10\}$; the two pairs each contribute $0.5$, so $W_4=1$. The sequence is $100,4,2.5,1$. The reduction from $K=1$ to $K=2$ is 96, much larger than the subsequent reductions of 1.5 and 1.5.

For $K=2$, the silhouettes for observations $0,1,2$ are calculated as follows. For 0, $a(0)=(1+2)/2=1.5$, $b(0)=(8+9+10)/3=9$, and $s(0)=1-1.5/9=0.8333$. For 1, $a(1)=1$, $b(1)=(7+8+9)/3=8$, giving $s(1)=0.8750$. For 2, $a(2)=1.5$, $b(2)=(6+7+8)/3=7$, giving $s(2)=1-1.5/7=0.7857$. By symmetry, the silhouettes for 8, 9, and 10 are $0.7857,0.8750,$ and $0.8333$. Thus the mean is

\bar{s}_{K=2}=\frac{2(0.8333+0.8750+0.7857)}{6}\approx 0.8313.

For the stated $K=3$ partition, $s(0)=1-1/2=0.5000$, since $a(0)=1$ and the nearest other cluster is the singleton $\{2\}$, at distance 2. For 1, $a(1)=1$ and $b(1)=1$, so $s(1)=0$. Observation 2 is a singleton, so its silhouette is 0 by convention. For 8, $a(8)=1.5$, $b(8)=6$, and $s(8)=0.7500$. For 9, $a(9)=1$, $b(9)=7$, and $s(9)=6/7\approx0.8571$. For 10, $a(10)=1.5$, $b(10)=8$, and $s(10)=0.8125$. The overall mean is

\bar{s}_{K=3}=\frac{0.5000+0+0+0.7500+0.8571+0.8125}{6}\approx 0.4866.

Conclusion and check. $K=2$ is the practical choice here: it captures the large gap in the measurements, has a much larger mean silhouette, and avoids making the observation 2 a singleton cluster. The $K=3$ partition has the lower WCSS, as expected, but its silhouettes show that the extra cluster does not provide a compelling improvement. A common mistake is to choose the largest candidate $K$ because its WCSS is smallest; the correction is to interpret the WCSS reduction alongside assignment quality and cluster sizes. The calculations check: the two $K=2$ clusters each have size 3, and their WCSS contributions sum to 4.

Worked Example: Weighting Cluster-Level Silhouette Summaries

A hypothetical data set has ten observations represented by the same numerical features for all candidate solutions. For $K=2$, the two cluster sizes are 5 and 5, and their mean silhouettes are 0.54 and 0.50. For $K=4$, the cluster sizes are 3, 3, 2, and 2, with mean silhouettes 0.80, 0.70, 0.20, and $-0.10$, respectively. Suppose WCSS is 38 for $K=2$ and 20 for $K=4$. Calculate the overall silhouette averages and recommend a candidate.

Method. The overall mean is an observation-weighted average of cluster means. A cluster containing three observations contributes three times its mean silhouette, not the same total weight as a cluster containing one observation.

For $K=2$, the weighted mean is $(5(0.54)+5(0.50))/10=(2.70+2.50)/10=0.5200$. For $K=4$, it is

\frac{3(0.80)+3(0.70)+2(0.20)+2(-0.10)}{10} =\frac{2.4+2.1+0.4-0.2}{10} =\frac{4.7}{10}=0.4700.

Conclusion and check. The $K=4$ solution reduces WCSS from 38 to 20, but its overall mean silhouette is lower than for $K=2$, and one cluster has a negative mean silhouette. $K=2$ is better supported by these summaries. The weighted numerator for $K=4$ is 4.7, and the cluster sizes sum to $3+3+2+2=10$, so the average is consistent. A common mistake is to add the weighted terms incorrectly or to average the four cluster means equally; use each cluster size as its weight.

Worked Example: Combining the Elbow, Silhouettes, and Cluster Sizes

An analyst fits k-means to a hypothetical set of 24 records using the same scaled numerical features for each candidate. The best WCSS found across repeated initializations, mean silhouette, and cluster sizes are shown below. Choose a practical $K$, explaining how the diagnostics work together.

K	WCSS	Mean silhouette	Cluster sizes
2	96	0.46	12, 12
3	61	0.53	8, 8, 8
4	48	0.49	2, 8, 7, 7
5	39	0.42	2, 2, 6, 7, 7

Step 1: Read the WCSS reductions. From $K=2$ to $K=3$, WCSS falls by $96-61=35$. The next reductions are $61-48=13$ and $48-39=9$. The diminishing reductions suggest a bend around $K=3$, although the elbow alone does not prove that $K=3$ is best.

Step 2: Compare silhouettes. The mean silhouette is largest at $K=3$, at 0.53. It declines to 0.49 at $K=4$ and 0.42 at $K=5$, so the additional divisions do not improve average separation in this set of candidate solutions.

Step 3: Inspect sizes. The $K=3$ solution has three equal clusters of eight observations. The $K=4$ and $K=5$ solutions include clusters of size two, which may be less useful in an application that needs substantial groups. Small clusters are not automatically wrong, but they warrant investigation.

Conclusion and check. Choose $K=3$ as the practical candidate: it is near the elbow, has the best mean silhouette, and yields balanced cluster sizes. This is a reasoned choice rather than a universal rule; a different objective could make a small group important. The counts check: each row sums to 24, and WCSS decreases at every step. A common mistake is to treat any decrease in WCSS as evidence for adding another cluster. The correction is to ask whether that decrease is accompanied by better separation or more useful groups.

Practical Interpretation and Common Traps

A defensible choice of $K$ is a balance, not a mechanical rule. State what each diagnostic indicates and identify any conflict. For example, an elbow may favor a smaller value while the silhouette peaks at a larger one; a candidate may have a high overall silhouette but also a very small cluster. Explain which consideration matters for the intended use rather than hiding the disagreement.

Do not compare WCSS from different feature scalings. The scale of the inputs changes distances and therefore changes WCSS.
Do not expect WCSS to select $K$ by itself. Its decrease with more clusters is built into the optimization problem.
Do not average cluster means without weighting. The overall mean silhouette weights every observation equally, so cluster means must be weighted by their cluster sizes.
Do not ignore a negative or low cluster-level silhouette. A favorable overall average can conceal a group whose observations may fit another cluster better.
Do not call a small cluster automatically invalid. Check whether it is meaningful for the analysis and whether it is stable and well separated before rejecting it.

Key takeaway: Compare candidate values of $K$ on the same numerical representation. Use WCSS to assess compactness and the elbow to describe diminishing returns; use silhouettes to assess assignments and cluster sizes to judge practical usefulness. Then justify a choice using the combined evidence.

Check Your Understanding

For each problem, use the stated diagnostics and show enough arithmetic to verify the final choice.

A fixed feature representation gives WCSS values 80, 35, 23, and 18 for $K=1,2,3,4$. Which candidate is the most plausible elbow based on the successive reductions? (Answer: $K=2$; the reductions are 45, 12, and 5, with the largest drop before $K=2$.)
An observation has average within-cluster distance $a(i)=2.4$ and nearest-other-cluster average distance $b(i)=6$. Calculate its silhouette. (Answer: $s(i)=(6-2.4)/6=0.6000$.)
A candidate solution has three clusters of sizes 4, 3, and 1, with mean silhouettes 0.60, 0.30, and $-0.20$. Find the overall mean silhouette. (Answer: $(4(0.60)+3(0.30)+1(-0.20))/8=3.1/8=0.3875$.)
For one scaled data set, $K=2$ has WCSS 50, mean silhouette 0.41, and sizes 6, 6; $K=3$ has WCSS 34, mean silhouette 0.50, and sizes 4, 4, 4; $K=4$ has WCSS 27, mean silhouette 0.46, and sizes 1, 3, 4, 4. Which is the best-supported practical choice from these summaries? (Answer: $K=3$, which has the highest mean silhouette and balanced sizes; $K=4$ includes a singleton and a lower mean silhouette.)
An application requires every cluster to contain at least three observations. Candidate $K=2$ has sizes 8, 8 and mean silhouette 0.48; $K=3$ has sizes 5, 5, 6 and mean silhouette 0.55; $K=4$ has sizes 2, 3, 4, 7 and mean silhouette 0.60. Which candidate meets the size requirement and has the higher silhouette among eligible candidates? (Answer: $K=3$; $K=4$ violates the minimum-size requirement, while $K=3$ has a higher silhouette than eligible $K=2$.)

What You'll Learn