Why Stability Must Be Read Across Cluster Counts
A clustering solution can appear reproducible at one value of \(k\) and much less reproducible at another. Looking at only one candidate cluster count therefore leaves out an important part of the evidence: how does agreement change as the analysis asks for more or fewer groups? A stability profile summarizes that pattern across a prespecified set of candidate values of \(k\).
The profile is not a score that proves a particular number of clusters is correct. It describes how consistently a specified analysis pipeline recovers partitions under a specified perturbation scheme. A very coarse partition can be easy to reproduce, even when it is not useful for the scientific purpose. The introductory material on consensus clustering covers related summaries of repeated assignments; here the focus is on comparing stability across candidate cluster counts.
Each partition in a comparison must refer to the same units, or to units on which both runs have assignments. One common design is to fit the method on perturbed training data and then assign the same held-out reference samples in every run. Another is to compare assignments for the samples shared by a pair of resamples. The comparison rule must be fixed in advance; changing the evaluation samples or assignment procedure across \(k\) can make differences in the profile reflect design changes rather than cluster-count changes.
For a fair profile, keep the resampling unit, perturbation scheme, preprocessing rules, clustering algorithm, tuning choices, and agreement metric consistent across candidate \(k\). The resampling unit should respect the dependence structure of the data, as established in the earlier material on choosing a resampling unit. If the same patient contributes repeated observations, for example, resampling observations independently would not represent independent patient-level perturbations.
Computing and Displaying a Profile
A standard choice for \(M\) is the adjusted Rand index (ARI). It compares two partitions using pairs of observations, adjusts for agreement expected under a chance model, and is invariant to arbitrary cluster labels. For a contingency table with \(n_{ab}\) observations assigned to cluster \(a\) in one partition and cluster \(b\) in the other, let \(n_{a\cdot}\) and \(n_{\cdot b}\) be the row and column totals, and let \(n\) be the total number of observations. Then
An ARI of 1 indicates identical partitions; a value near 0 indicates agreement about as large as expected under the index's chance-adjustment model; negative values indicate less agreement than that model's expectation. The adjustment does not make ARI a universal measure of biological validity, nor does it guarantee that scores from every possible partition structure have identical practical meaning.
Worked Example: Computing Agreement for One Pair of Runs
Suppose two hypothetical runs assign the same eight omics samples to two clusters. Their contingency table is
| Run 2: cluster 1 | Run 2: cluster 2 | Run 1 total | |
|---|---|---|---|
| Run 1: cluster 1 | 3 | 1 | 4 |
| Run 1: cluster 2 | 1 | 3 | 4 |
| Run 2 total | 4 | 4 | 8 |
The within-cell pair count is \(3+0+0+3=6\). The corresponding row-total and column-total pair counts are each \( \binom{4}{2}+\binom{4}{2}=12\), and the total number of sample pairs is \(\binom{8}{2}=28\). Therefore,
The two runs agree on many assignments, but the adjusted agreement is modest: \(0.1250\), not close to 1. This is one pairwise comparison, not yet the stability profile for \(k=2\); a profile averages comparisons across all run pairs at each candidate \(k\).
For a plot, put candidate \(k\) on the horizontal axis and the selected mean agreement \(\widehat{S}(k)\) on the vertical axis. Show the number of resampling runs and, where helpful, display the distribution or spread of pairwise scores as well as their mean. Because pairwise comparisons share runs, the \(R(R-1)/2\) scores are dependent; treating them as independent observations to form a conventional standard error would understate the dependence. A descriptive spread or a resampling-based uncertainty analysis that respects the design is more defensible than a naive standard error.
Worked Example: Reading a Stability Profile Without Selecting Its Peak
In a hypothetical analysis, 40 resampling runs are performed at each candidate \(k\). The table reports mean pairwise ARI and the smallest cluster size in the full-data fit.
| Candidate \(k\) | Mean pairwise ARI | Smallest full-data cluster |
|---|---|---|
| 2 | 0.96 | 2 of 60 samples |
| 3 | 0.87 | 12 of 60 samples |
| 4 | 0.82 | 9 of 60 samples |
| 5 | 0.78 | 7 of 60 samples |
The plotted line falls from 0.96 at \(k=2\) to 0.78 at \(k=5\). A rule that simply picks the maximum would select \(k=2\). But at \(k=2\), the full-data partition contains a cluster of only 2 of the 60 samples and a complementary cluster of 58. The high mean ARI says that this very coarse split is reproducible under the stated perturbations; it does not establish that the split represents useful subgroup structure. The appropriate report is that \(k=2\) has the highest measured stability, accompanied by the severe size imbalance, while larger \(k\) values show lower stability. Which solution is useful depends on the scientific objective and other evidence, not on the profile alone.
Metric Choice and What a Peak Can Hide
Different agreement measures emphasize different aspects of partition similarity. The ARI summarizes pairwise co-membership and includes a chance adjustment. Variation of information (VI) is an information-theoretic distance: it is zero for identical partitions, and larger values indicate greater difference. One common normalized form divides VI by \(\log_2(n)\), yielding a value between 0 and 1 for partitions of the same \(n\). State the exact normalization if using it; do not place a VI distance and an ARI on the same vertical scale as if larger always meant more stability.
Worked Example: One Pair, Two Agreement Summaries
Use the \(2\times2\) table from the previous example. Both partitions split eight samples into groups of four, so each partition has entropy \(H=1\) bit. In either direction, the conditional cluster proportions are \(3/4\) and \(1/4\), giving
The same value holds for \(H(P_2\mid P_1)\). Thus
and the normalized VI is \(1.6226/\log_2(8)=1.6226/3=0.5409\). This indicates appreciable disagreement, consistent with the modest ARI of 0.1250. The scales and chance adjustments differ, so the numerical values should not be compared directly. Across \(k\), choose a measure before interpreting the profile and keep its direction clear: higher ARI means greater agreement, whereas lower VI means greater agreement.
A useful comparison is between a single mean profile and supporting summaries. Co-clustering probabilities summarize how often pairs are assigned together across runs and can reveal patterns obscured by one average. Cluster-wise Jaccard summaries can show whether agreement is uneven across clusters, but require a cluster-matching rule and can be sensitive to small groups. VI provides a distance rather than a similarity, which some analysts find easier to interpret as information lost between partitions. None is a universal substitute for the others: a profile should name its metric, and important conclusions should be checked for sensitivity to reasonable alternatives.
Justification, Diagnostics, and Limitations
The mean profile has a simple interpretation because it averages the same kind of comparison at each \(k\). For a fixed \(k\), there are \(R(R-1)/2\) distinct run pairs. If each pair's score is \(m_{rs}\), their arithmetic mean is
This identity justifies the profile summary; it does not make the mean a complete description. Two candidate values of \(k\) can have the same mean but very different distributions of pairwise scores. A narrow distribution suggests more uniform agreement across run pairs than a wide one, although dependence among pairwise scores still matters.
Before interpreting a profile, check that each candidate was evaluated on comparable samples and that the pipeline was refitted as intended in each resample. Inspect the distribution of pairwise scores, cluster-size summaries, the frequency of failed or empty-cluster fits, and whether high means are driven by a small number of run pairs. If the clustering algorithm can return fewer than the requested number of nonempty clusters, report the realized cluster counts rather than assuming they equal \(k\). A flat profile can indicate similar reproducibility across the tested range, but can also arise from a metric that is insensitive to a feature of interest or from insufficiently informative perturbations.
Stability estimates depend on the resampling design and on the analysis choices. Repeatedly perturbing the same samples in nearly identical ways may produce high agreement without showing robustness to meaningful changes. Conversely, aggressive perturbation, noisy measurements, or small sample sizes can lower agreement even when a population-level structure exists. A profile is therefore conditional on the data, pipeline, perturbations, and metric; it is not a probability that a partition is true.
Use the profile as one component of a decision, not as a stand-alone selection algorithm. It is most informative when the candidate range and perturbations are scientifically defensible, the preprocessing and fitting steps are consistent, and the plot is accompanied by enough diagnostics to expose degeneracy. A high score is evidence of reproducibility under the chosen design; it is not evidence of causation, biological importance, or a uniquely correct number of clusters.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- With \(R=10\) runs at a fixed \(k\), how many distinct pairwise agreement scores contribute to the mean stability?
- A profile reports mean ARI 0.90 at \(k=2\) and 0.80 at \(k=4\). What can be concluded, and what cannot be concluded from these values alone?
- For the contingency table in the first worked example, why is the ARI unchanged if the cluster labels in one run are swapped?
- Why is it problematic to compare a profile of ARI values with a profile of untransformed VI values by choosing the larger number at each \(k\)?
Solutions. (1) There are \(\binom{10}{2}=10\times9/2=45\) distinct run pairs. (2) Under the specified resampling and metric, the \(k=2\) partitions have higher average pairwise adjusted agreement. These values alone do not establish that two clusters are scientifically meaningful, nor that \(k=2\) is the correct choice; cluster sizes and other diagnostics are needed. (3) Swapping labels only permutes the contingency table's columns. The sums over cells and column totals in the ARI formula do not change, so the index is invariant to label names. (4) ARI is a similarity for which higher values indicate greater agreement, while VI is a distance for which lower values indicate greater agreement. Their scales and directions differ, so their raw magnitudes cannot be ranked as if they measured the same quantity.