One Clustering, Several Different Questions
A clustering can look convincing by one criterion and weak by another. Cells may form compact groups in a chosen expression-space distance, yet cut across neighborhoods in a cell graph. They may agree poorly with supplied annotations but recur under small data perturbations. These outcomes are not necessarily calculation errors: the metrics are answering different questions.
The useful question is not simply which score is largest. It is what property does each score assess, and what does its disagreement with another score reveal? Internal indices assess geometry under a chosen representation and distance. Graph diagnostics assess connectivity in a specified graph. Reference-label agreement assesses correspondence to labels supplied independently of the clustering. Perturbation stability assesses reproducibility under a specified change to the data or analysis. None alone establishes that a partition is biologically correct.
Internal Geometry: Silhouette Width
The silhouette width summarizes how close an observation is, on average, to its assigned cluster relative to its nearest alternative cluster. For cell \(i\), let \(a(i)\) be its average distance to the other cells in its assigned cluster. For each other cluster, calculate the average distance from \(i\) to that cluster; let \(b(i)\) be the smallest of those averages. The silhouette width is
For a singleton cluster, conventions vary; a common convention sets its silhouette to zero. Otherwise, \(s(i)\) lies between \(-1\) and \(1\). A value near 1 means the assigned cluster is closer on average than the nearest alternative. A value near 0 indicates comparable distances, and a negative value means the nearest alternative cluster is closer on average. The mean silhouette is the average of \(s(i)\) over the cells being assessed.
This summary depends on the representation, scaling, distance, and partition. A high mean silhouette indicates separation in that chosen geometry; it is not a universal measure of biological validity. Earlier material on distance geometry shows why a distance that distorts relevant structure can also distort an internal score.
Worked Example: A Silhouette Ranking That Favors One Partition
Consider eight hypothetical cells on a one-dimensional representation at coordinates \(0,1,2,3,10,11,12,13\). Partition \(G\) places the first four cells together and the last four together. Partition \(R\) groups \(\{0,2,10,12\}\) and \(\{1,3,11,13\}\). Use absolute distance and the silhouette definition above.
For \(G\), the within-cluster mean distance \(a(i)\), in order for the first block, is \(2,\frac{4}{3},\frac{4}{3},2\). For example, the endpoint at 0 has distances 1, 2, and 3 to its other cluster members, so \(a(0)=(1+2+3)/3=2\). The corresponding nearest-other-cluster mean distances are \(11.5,10.5,9.5,8.5\). Thus the four silhouettes are
The other block has the same four widths in reverse order. Therefore \(\bar{s}_G=(0.8261+0.8730+0.8596+0.7647)/4\approx0.8309\).
For \(R\), the cell at 0 has \(a(0)=(2+10+12)/3=8\), while its average distance to the other cluster is \((1+3+11+13)/4=7\). Its width is \((7-8)/8=-0.125\). Repeating the calculation for coordinates 2, 10, and 12 gives \(-0.175,-0.2500,-0.3125\), respectively; the other four cells have the same widths. Hence \(\bar{s}_R=(-0.125-0.175-0.2500-0.3125)/4=-0.215625\), approximately \(-0.2156\). On this representation and distance, silhouette strongly favors \(G\).
Graph Diagnostics: Connectivity Across a Cut
A cell graph represents cells as vertices and selected relationships as edges, often based on neighborhood similarity. A graph diagnostic asks whether a proposed cluster is well connected internally relative to its connections outside the cluster. One common measure is conductance. For a vertex set \(C\), let \(\operatorname{cut}(C,C^c)\) be the total edge weight joining \(C\) to its complement, and let \(\operatorname{vol}(C)\) be the sum of degrees (or weighted degrees) of vertices in \(C\). For an undirected graph, define
A small conductance means relatively little edge weight crosses the boundary compared with the volume of the smaller side. It is a graph-based diagnostic, not a distance-based compactness score. Its meaning depends on how the graph was built, including the neighborhood rule, edge weights, and any pruning.
Worked Example: A Geometric Split Can Cut Through Graph Neighborhoods
Take a hypothetical graph made of two disjoint four-vertex cliques. Each vertex has degree 3. Consider a partition \(G\) that puts two vertices from each clique in each of its two clusters. Within each clique, the split cuts \(2\times2=4\) edges, so the total cut is \(4+4=8\). Each cluster has four vertices and volume \(4(3)=12\), as does its complement. Its conductance is therefore
In contrast, the partition into the two whole cliques has cut weight 0 and volume 12 on each side, so its conductance is \(0/12=0\). The graph strongly favors keeping each clique intact. If an internal distance index favors the split \(G\), the disagreement says that the distance geometry and graph connectivity encode different structures, or that the graph construction may not reflect the geometry used by the internal index. It does not by itself identify which structure is scientifically preferable.
Reference-Label Agreement: What Was Supplied as a Comparator?
When reference labels are available, one can measure agreement between those labels and a clustering. The adjusted Rand index (ARI) compares pairs of cells: it rewards pairs placed together in both partitions or apart in both, then adjusts the raw agreement for the expected value under a chance-partition model with the same cluster-size margins. Its value is 1 for identical partitions, is near 0 for agreement at the chance-adjusted baseline, and can be negative when agreement is below that baseline. The ARI is invariant to arbitrary names of clusters.
For precision, let \(n_{ij}\) be the number of cells in reference group \(i\) and predicted cluster \(j\), with row totals \(a_i\), column totals \(b_j\), and \(n\) total cells. The standard ARI is
This comparison assumes the reference labels are meaningful for the question and were not simply created from the same clustering result. Labels may be coarse, incomplete, or based on a different biological definition. Therefore, low ARI can expose disagreement with the reference scheme without proving the clustering is useless; high ARI establishes correspondence to those labels, not that every cell is correctly characterized in every sense.
Worked Example: The Reference Labels Favor the Lower-Silhouette Partition
Use the eight cells from the silhouette example, but suppose the hypothetical reference labels are exactly partition \(R\). The reference groups and predicted clusters \(G\) each contain four cells, and each intersection contains two. Thus \(\sum_{ij}\binom{n_{ij}}{2}=4\), while each set of row and column totals gives \(2\binom{4}{2}=12\). There are \(\binom{8}{2}=28\) cell pairs, so the chance-adjustment term is \(12(12)/28=36/7\). Substitution gives
Partition \(R\) matches the reference partition exactly, so its ARI is 1. The metrics therefore disagree: silhouette favors \(G\), while reference-label agreement favors \(R\). The reason is not mysterious: the geometric compactness of the two partitions and their correspondence to the supplied labels are distinct properties. The appropriate interpretation depends on whether those labels are an independent, relevant comparator for the scientific question.
Perturbation Stability: Reproducibility Under a Defined Change
Stability compares cluster assignments across runs after a specified perturbation, such as cell subsampling, feature perturbation, or a change in initialization. The perturbation must be stated because stability is always stability under that perturbation. For two partitions of the same cells, an agreement measure such as ARI can summarize assignment similarity; across many runs, one may summarize pairwise agreement. If runs contain different cells, comparisons require a stated matching or common-cell procedure rather than treating absent cells as disagreements.
A stable result is reproducible under the tested changes. It need not have high silhouette, low conductance, or high agreement with reference labels. Conversely, an unstable result may signal a boundary-sensitive structure, a rare group, or an analysis pipeline sensitive to small changes. These are diagnostic possibilities, not conclusions that can be selected from the stability score alone.
Worked Example: Stability Can Favor a Different Partition
Suppose six hypothetical cells are clustered in two runs of an analysis under small feature perturbations. In run 1, partition \(P\) is \(\{1,2,3\}\mid\{4,5,6\}\); in run 2, it is \(\{1,2,4\}\mid\{3,5,6\}\). The contingency table has entries \(2,1;1,2\). Each row and column total is 3, and there are \(\binom{6}{2}=15\) pairs. The cell-pair sum is \(4\), and the chance term is \(6(6)/15=2.4\). Thus
Now suppose an alternative partition \(Q\) is identical in the two perturbed runs. Its pairwise ARI is 1, so it is more stable under these particular perturbations. That comparison says nothing by itself about whether \(Q\) has better geometric separation or corresponds better to an independent reference. It says only that its assignments changed less in this experiment.
How to Interpret Disagreement
A disagreement is most useful when it is traced back to the inputs and targets of the metrics, rather than settled by declaring one score the winner. Internal indices such as silhouette, Calinski–Harabasz, and Davies–Bouldin summarize aspects of within- and between-cluster geometry, but they can rank partitions differently because their formulas emphasize different summaries. Graph diagnostics depend on the graph, while label agreement depends on the reference scheme. Stability depends on which perturbations were applied and how runs were compared.
A practical comparison makes these distinctions explicit:
Record whether it evaluates geometric separation, graph connectivity, reference-label correspondence, or reproducibility.
Confirm the distance and representation, graph construction, reference-label provenance, and perturbation design are appropriate to the question.
Inspect cluster sizes, cell-level silhouettes, graph boundaries, label cross-tabulations, and which cells switch across runs. A single aggregate score can hide a localized failure.
Describe which criteria support each partition and which assumptions make each comparison informative. Do not convert a collection of distinct scores into an unqualified claim of validity.
Diagnostics should also be checked for computational or design failures: a graph with disconnected components can make some cuts trivially small; a reference label with very different granularity can depress agreement; and a perturbation that barely changes the input may make stability look reassuring without testing the relevant sensitivity. Likewise, apparent differences in internal scores can arise from a distance geometry that does not preserve the structure of interest.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- If a cell has \(a(i)=3\) and \(b(i)=2\), what is its silhouette, and what does its sign indicate?
- A graph cut has weight 6, and the smaller of the two cluster volumes is 18. What is its conductance?
- Why can a clustering with a higher silhouette have a lower ARI against reference labels?
- Two partitions on the same cells have ARI 1 under every tested perturbation. What has been established, and what has not?
- Name one reason a low graph conductance could be uninformative about biological validity.
Solutions. (1) \(s(i)=(2-3)/3=-1/3\); the negative value means the nearest other cluster is closer on average than the assigned cluster. (2) \(\phi=6/18=1/3\). (3) Silhouette assesses separation under a chosen distance, whereas ARI assesses pairwise correspondence to the reference partition; the two partitions can encode different structures. (4) The assignments were identical under those tested perturbations; this does not establish geometric separation, external-label correspondence, or biological correctness. (5) For example, disconnected graph components can yield a zero cut, or the graph construction may not represent the relationships relevant to the biological question.