What Cluster Validation Can Establish
A clustering algorithm returns groups, but the existence of an output does not establish that the groups are meaningful. Validation measures summarize particular properties of a partition: whether points are close to members of their assigned cluster, whether the result agrees with available labels, or whether the result changes under perturbations. These are different questions, and no single score certifies that a data set contains objectively true clusters.
This tutorial distinguishes internal validation, which uses the observed features and chosen dissimilarities, from external validation, which compares a clustering with labels supplied independently of that clustering. The silhouette is an internal measure. The adjusted Rand index (ARI) and confusion tables are external comparison tools. Their meanings depend on the data, distance, labels, and clustering procedure being evaluated.
Silhouette: Within-Cluster Fit Versus Nearest-Cluster Separation
Suppose \(n\) observations have been assigned to \(K\) clusters, with \(K\geq 2\), and a dissimilarity \(d(i,j)\) has been chosen for every pair. For observation \(i\), let \(a(i)\) be its mean dissimilarity to the other observations in its own cluster. For any other cluster \(C\), calculate the mean dissimilarity from \(i\) to members of \(C\). Let \(b(i)\) be the smallest of those other-cluster means. Thus \(a(i)\) describes within-cluster closeness, while \(b(i)\) measures closeness to the nearest alternative cluster.
When the denominator is positive, \(s(i)\) lies between \(-1\) and \(1\). A value near 1 means the point is, on average, much closer to its own cluster than to the nearest alternative. A value near 0 means the two distances are similar. A negative value means \(a(i)>b(i)\): on average, the point is closer to another cluster than to its assigned cluster. This is a warning about that point's assignment, not proof that it belongs elsewhere.
For a singleton cluster, there are no other members from which to calculate \(a(i)\). A common convention sets its silhouette to 0; software conventions should be checked when singleton clusters occur. If \(a(i)=b(i)=0\), the formula itself has a zero denominator, so an implementation must specify how that degenerate case is handled. The overall mean silhouette is the arithmetic mean of the pointwise silhouettes. It weights every observation equally, so large clusters contribute more to the mean than small ones.
Worked Example: Calculating and Averaging Silhouettes
Consider four hypothetical profiles assigned to two clusters, \(C_1=\{x_1,x_2\}\) and \(C_2=\{x_3,x_4\}\). Suppose their chosen dissimilarities are as follows.
| \(x_1\) | \(x_2\) | \(x_3\) | \(x_4\) | |
|---|---|---|---|---|
| \(x_1\) | 0 | 2 | 5 | 6 |
| \(x_2\) | 2 | 0 | 3 | 4 |
| \(x_3\) | 5 | 3 | 0 | 1 |
| \(x_4\) | 6 | 4 | 1 | 0 |
For \(x_1\), its within-cluster mean is \(a(x_1)=2\), and its mean dissimilarity to \(C_2\) is \(b(x_1)=(5+6)/2=5.5\). Hence \(s(x_1)=(5.5-2)/5.5=0.6364\). For \(x_2\), \(a(x_2)=2\) and \(b(x_2)=(3+4)/2=3.5\), so \(s(x_2)=(3.5-2)/3.5=0.4286\).
For \(x_3\), \(a(x_3)=1\), while the mean dissimilarity to \(C_1\) is \((5+3)/2=4\). Thus \(s(x_3)=(4-1)/4=0.7500\). For \(x_4\), \(a(x_4)=1\), \(b(x_4)=(6+4)/2=5\), and \(s(x_4)=(5-1)/5=0.8000\). The overall mean, calculated from the unrounded values, is
The positive mean summarizes reasonably good separation relative to within-cluster dissimilarity for these four profiles. It does not show that two is the uniquely correct number of clusters, that the chosen dissimilarity is biologically appropriate, or that the clusters will reproduce in new data.
External Comparison: Contingency Tables and the Adjusted Rand Index
When independent labels are available, a contingency table records how the label groups intersect the cluster assignments. Write \(n_{ij}\) for the number of observations in reference group \(i\) and cluster \(j\), \(a_i=\sum_j n_{ij}\) for reference-group totals, \(b_j=\sum_i n_{ij}\) for cluster sizes, and \(n=\sum_{ij}n_{ij}\). The table reveals where agreement and disagreement occur, but its row and column names need not correspond: cluster labels such as “1” and “2” are arbitrary and can be permuted without changing the partition.
For two observations, ask whether they are grouped together in both partitions, separated in both, or treated differently by the two partitions. The Rand index is the fraction of pairs on which the two partitions agree. Its unadjusted value can appear high simply because many pairs are separated, especially when there are many small groups. The ARI corrects the number of jointly grouped pairs for its expected value under a chance model that holds the two sets of group sizes fixed.
The adjustment follows from the fixed-margin chance model. If the reference group sizes and cluster sizes are fixed and memberships are paired at random, the probability that a given pair is together in the reference partition is \(R/T\), and the probability it is together in the clustering is \(C/T\). Under random pairing, the expected number together in both is therefore
Subtracting this expected count centers the numerator at zero for this chance model. The denominator scales by the average of the two partitions’ separate pair counts, \((R+C)/2\), with the same expected count subtracted. Thus ARI is 1 for identical partitions (up to relabeling) when the expression is defined, is near 0 for agreement at the fixed-margin chance level, and can be negative when agreement is below that level. It is an agreement index, not a p-value or the probability that the labels are correct.
Worked Example: Computing an Adjusted Rand Index
Suppose eight hypothetical profiles have the following cross-tabulation of an external two-class annotation and a two-cluster solution.
| Reference label | Cluster 1 | Cluster 2 | Total |
|---|---|---|---|
| Class A | 3 | 1 | 4 |
| Class B | 1 | 3 | 4 |
| Total | 4 | 4 | 8 |
The number of pairs together in both partitions is \(A=\binom{3}{2}+\binom{1}{2}+\binom{1}{2}+\binom{3}{2}=3+0+0+3=6\). The row totals give \(R=\binom{4}{2}+\binom{4}{2}=12\), and the column totals give \(C=12\). Also, \(T=\binom{8}{2}=28\). Substitution gives
The clustering agrees with the reference partition on some pairs, but its ARI is only 0.1250 after adjusting for the fixed-margin chance expectation. This value does not mean that 12.5% of profiles were classified correctly. It also does not establish that the reference annotation is a biological ground truth.
Reading Confusion Tables Without Overstating Agreement
A confusion table is useful because it shows the pattern of external agreement, not just a single summary. If reference labels have substantive names but clusters are numbered, the table itself does not imply that “Cluster 1” means “Class A.” Any calculation of classification accuracy must state how cluster labels were matched to reference labels. One-to-one matching, such as assigning cluster names to maximize the diagonal total, differs from purity, which assigns each cluster to its largest reference group and can let multiple clusters map to the same group.
Worked Example: Interpreting a Confusion Table
In a hypothetical set of 20 profiles, an external annotation has two classes. A clustering returns two groups, and the table is:
| Reference label | Cluster 1 | Cluster 2 | Total |
|---|---|---|---|
| Class A | 8 | 2 | 10 |
| Class B | 1 | 9 | 10 |
| Total | 9 | 11 | 20 |
If Cluster 1 is matched to Class A and Cluster 2 to Class B, the diagonal count is \(8+9=17\), so the matched accuracy is \(17/20=0.8500\), or 85.0%. Under this naming, Class A recall is \(8/10=0.8000\), and Class B recall is \(9/10=0.9000\). These are conditional proportions relative to the chosen reference labels and this explicit label matching.
The table is not a proof that the clusters recover a natural two-class structure. It shows that this partition aligns substantially with the annotation under the stated matching. For a non-classification clustering comparison, an ARI is often preferable because it is invariant to permutations of cluster names and does not require a diagonal label convention. Different external measures answer different questions, so the table should remain available for inspection.
Assumptions, Diagnostics, and Limits
The silhouette requires a meaningful dissimilarity between observations and a valid cluster assignment with at least two clusters. Its interpretation is strongest when the chosen dissimilarity represents the scientific notion of profile similarity and is applied consistently. In high-dimensional genomic data, scale, normalization, correlated features, and the distance definition can substantially affect these values. A silhouette is not a property of a partition alone: changing the dissimilarity can change both \(a(i)\) and \(b(i)\), and therefore the score.
The ARI requires a set of paired observations with both partitions defined for the same \(n\) units. Its chance adjustment is conditional on the observed group-size margins; it is not a universal null model for biological structure. If one partition is all singletons or both partitions have degenerate margins, the ARI denominator may be zero. In that case the displayed formula is undefined, and software may adopt conventions that should be reported rather than silently interpreted as ordinary ARI values.
Useful diagnostics include examining pointwise silhouettes and cluster-wise distributions rather than only the overall mean, identifying negative or near-zero values, and checking whether small clusters or singletons dominate interpretation. For external comparisons, inspect the full contingency table and group sizes; a single score can hide a class that is poorly represented or a cluster that mixes several labels. In either setting, repeat interpretation under scientifically defensible preprocessing and dissimilarity choices. Large changes signal dependence on analytical choices, not a reason to select whichever score looks most favorable.
These measures also differ from cluster stability. Stability under resampling asks whether assignments persist when the data are perturbed; silhouette asks about relative separation in the observed geometry; ARI asks about agreement between two partitions. A stable partition can consistently reproduce an uninformative grouping, and a high silhouette can describe a geometrically separated partition with no meaningful external interpretation. Conversely, an external label can be incomplete or noisy, so disagreement with it does not by itself invalidate a clustering.
Choosing and Reporting Validation Measures
Use internal and external measures according to the question, rather than treating them as interchangeable rankings. The table summarizes their distinct roles.
| Measure | Inputs | What it summarizes | Important limitation |
|---|---|---|---|
| Silhouette | Assignments and chosen dissimilarities | Within-cluster closeness relative to nearest alternative | Depends on geometry; cannot verify biological meaning |
| Confusion table | Assignments and reference labels | Where the two partitions agree or differ | Named accuracy requires a stated label matching rule |
| Adjusted Rand index | Two partitions on the same observations | Pairwise agreement adjusted for fixed group-size margins | Depends on the reference partition and its chance model |
| Resampling stability | Clusterings from perturbed samples | Reproducibility of assignments under perturbation | Reproducibility is not substantive validity |
A concise report should name the clustering and dissimilarity, state the number and sizes of clusters, describe the reference labels if used, give the precise measure and any label-matching rule, and interpret the result narrowly. For a silhouette, report whether the value is an observation-level score or an overall mean. For ARI, make clear that it compares two partitions and give the observations included in both. Avoid reporting several scores as though the highest one identifies an objectively true partition.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- An observation has \(a(i)=4\) and \(b(i)=3\). Calculate its silhouette and interpret its sign.
- What does an ARI near zero mean, and why is it not a p-value?
- A two-cluster solution has 9 observations in each cluster. The external classes each also have 9 observations, and the contingency table is \(\begin{smallmatrix}9&0\\0&9\end{smallmatrix}\). What is the ARI?
- Why can a confusion-table accuracy be misleading if the cluster labels have not been matched to reference labels?
Solutions. 1. \(s(i)=(3-4)/4=-0.2500\). The point is, on average, closer to another cluster than to its assigned cluster, which flags a potentially questionable assignment but does not prove reassignment is correct. 2. It indicates agreement around the expected level under the ARI's fixed-margin chance model. It is not a p-value because it is a descriptive index, not a probability calculated from a sampling distribution under a hypothesis test. 3. The partitions are identical, so \(A=2\binom{9}{2}=72\), \(R=C=72\), and \(T=\binom{18}{2}=153\). Substitution gives \(\operatorname{ARI}=(72-72^2/153)/(72-72^2/153)=1\). 4. Cluster numbers are arbitrary: swapping Cluster 1 and Cluster 2 leaves the partition unchanged but can change the table's diagonal and apparent accuracy. State the matching rule or use a label-permutation-invariant measure such as ARI.