Stability Is Agreement Under a Specified Change
A clustering is not a direct observation of cell types; it is the output of an analysis. Changing the input data slightly, or changing a defensible analysis choice, can alter that output. Clustering stability describes how much the clustering agrees with a reference clustering when the analysis is repeated under a defined perturbation. The word “defined” matters: a stability result has meaning only in relation to what was changed, what was held fixed, and how agreement was measured.
For example, one could hold the cells, representation, graph, and algorithm fixed while comparing two nearby resolution settings. That asks about sensitivity to resolution, not sensitivity to the data or to every possible analysis choice. Conversely, changing several components at once may reveal overall sensitivity, but it does not identify which change caused disagreement.
A partition is a division of cells into non-overlapping clusters. Cluster labels themselves are arbitrary: calling a group “1” in one run and “3” in another does not mean the assignments differ. Agreement measures should therefore compare relationships among cells or otherwise account for label permutations, rather than compare the label numbers directly.
Two Scales of Agreement
A cell-level stability score asks how consistently one cell relates to the other cells across two partitions. For a cell \(i\), compare its same-cluster or different-cluster relationship with every other cell \(j\). Its score is the fraction of these \(n-1\) relationships that are preserved. This is a local summary: two cells can have very different scores in the same comparison.
Here \(n\) is the number of cells common to both partitions, \(P(i)\) denotes the cluster containing cell \(i\), and \(\mathbf{1}\{\cdot\}\) equals 1 when its condition is true and 0 otherwise. A relationship is preserved if the two cells are together in both partitions or apart in both. The cell score is not the probability that the cell belongs to a particular biological type.
A whole-partition stability score summarizes agreement across all cell pairs. A simple label-invariant measure is the Rand index: the proportion of all unordered cell pairs whose together/apart relationship agrees between partitions. The adjusted Rand index (ARI), introduced in the earlier material on chance-corrected partition comparison, adjusts pair agreement for the agreement expected under a chance model. Both are global summaries; they can conceal a small set of unstable cells.
The equality follows because each unordered pair is counted once for each of its two cells in the cell-level scores. Thus the sum of all cell scores counts every agreeing pair twice, and the denominator \(n(n-1)\) is twice \(\binom{n}{2}\). This connects the local and global views, but does not make them interchangeable: the average local score is the global Rand index, while the individual scores show where disagreement is concentrated.
Worked Example: Local and Global Agreement for Eight Cells
Suppose a hypothetical reference partition is \(P_0=\{1,2,3,4\}\mid\{5,6,7,8\}\), and a controlled perturbation gives \(P_1=\{1,2,3,5\}\mid\{4,6,7,8\}\). There are \(\binom{8}{2}=28\) cell pairs. In each partition, 12 pairs are together. Six pairs are together in both: three pairs among cells 1, 2, and 3, and three among cells 6, 7, and 8. Therefore 6 pairs are together only in \(P_0\), 6 are together only in \(P_1\), and \(28-6-6-6=10\) are apart in both.
The whole-partition Rand index is \(R=(6+10)/28=16/28=0.5714\). This says that 57.14% of pairs preserve their together/apart relationship. It does not say that 57.14% of cells retain their cluster labels.
For cell 1, relationships with cells 2 and 3 remain “together,” and relationships with cells 6, 7, and 8 remain “apart.” Its relationship with cell 4 changes from together to apart, and its relationship with cell 5 changes from apart to together. Thus \(S_1=5/7=0.7143\). Cells 2 and 3 have the same score. Cells 4 and 5 each preserve only their relationship with the other one, giving \(1/7=0.1429\). Each of cells 6, 7, and 8 preserves five of its seven relationships, giving \(5/7=0.7143\). The average is \((6(5/7)+2(1/7))/8=4/7=0.5714\), matching the Rand index.
The ARI gives a different global summary. The contingency table has counts \(3,1,1,3\); each partition has cluster sizes 4 and 4. The sum of \(\binom{n_{ij}}{2}\) is 6, each row- or column-margin sum of pair counts is 12, and the total number of pairs is 28. Substitution into the ARI formula gives \[ \mathrm{ARI}=\frac{6-(12\cdot12/28)}{\{(12+12)/2\}-(12\cdot12/28)} =\frac{6-36/7}{12-36/7}=\frac{1}{8}=0.125. \] The Rand index and ARI differ because the ARI corrects for chance-expected pair agreement. Neither number replaces the cell-level scores when the question is which cells are unstable.
Designing a Perturbation Comparison
A defensible comparison begins by naming the target of the sensitivity analysis. Perturbations might represent small changes in an analysis setting, a different defensible representation, or a controlled change to the data. The choice depends on the scientific question. For a narrow diagnostic, change one component at a time and hold the rest fixed. If the aim is to assess robustness to a collection of reasonable choices, define that collection in advance and report what it contains.
The partitions must refer to comparable cells for direct pairwise calculations. The score also depends on the choice of reference partition: comparing every perturbed result with one baseline answers a different question from comparing all pairs of perturbed results. State the reference and the perturbation range rather than reporting “stability” without qualification.
Name the reference partition, the cells to be compared, and the particular analysis component or data change to perturb.
This makes a difference in agreement interpretable as sensitivity to the planned change rather than an inseparable mixture of changes.
Use a partition-level measure for overall agreement and cell-level scores when identifying concentrated disagreement matters.
Show the range or distribution of agreement and describe where the disagreements occur; do not rely only on a single average.
This design also clarifies what stability does not establish. A partition may be stable because the chosen perturbations are mild, because the method repeatedly expresses the same bias, or because the data contain continuous structure that the clustering procedure divides consistently. Stability is a diagnostic of sensitivity, not proof of discrete biological categories or causal structure.
Worked Example: A Cell Can Be Stable While Its Partition Changes
Consider six hypothetical cells. The reference partition is \(P_0=\{1,2,3\}\mid\{4,5,6\}\), and a nearby analysis setting gives \(P_1=\{1,2\}\mid\{3,4,5,6\}\). There are 15 pairs. Four pairs are together in both partitions: pair \(\{1,2\}\) and the three pairs among cells 4, 5, and 6. Two pairs are together only in \(P_0\), namely \(\{1,3\}\) and \(\{2,3\}\). Three are together only in \(P_1\), namely \(\{3,4\},\{3,5\},\{3,6\}\). The remaining six pairs are apart in both.
The Rand index is \((4+6)/15=10/15=0.6667\). Cell 3 has score \(S_3=0/5=0\): its relationships with cells 1 and 2 change from together to apart, while its relationships with cells 4, 5, and 6 change from apart to together. In contrast, cell 1 preserves its relationship with cell 2 and with cells 4, 5, and 6, but changes its relationship with cell 3, so \(S_1=4/5=0.8000\). This comparison shows why a single partition-level score cannot identify which cells drive instability.
The example does not establish that cell 3 is biologically ambiguous. It establishes only that its pairwise clustering relationships are sensitive to this particular change in analysis setting. Other evidence would be needed for a biological interpretation.
Summarizing More Than One Perturbation
When a family of perturbations is used, calculate agreement separately for each member before summarizing. A mean can provide a compact description, but it can conceal a poor result under one important perturbation. A range, median, or table organized by perturbation type can make that sensitivity visible. Do not treat the resulting scores as independent observations or use their spread as a conventional confidence interval without a justified sampling model: the perturbations are design choices, not automatically random samples from a population.
Worked Example: An Average Can Hide a Weak Comparison
Suppose a hypothetical reference partition is compared with five predeclared analysis settings, changing one setting at a time. The resulting partition-level agreement scores are 0.92, 0.88, 0.91, 0.67, and 0.89. Their mean is \[ \bar{R}=(0.92+0.88+0.91+0.67+0.89)/5=4.27/5=0.854. \] The ordered values are 0.67, 0.88, 0.89, 0.91, and 0.92, so the median is 0.89 and the range is 0.67 to 0.92.
Reporting only the mean of 0.854 would obscure that one setting gives considerably lower agreement. The appropriate conclusion is that agreement is high for four of the five specified comparisons but noticeably lower for the setting with score 0.67; the cause and importance of that discrepancy should be examined in context. No universal cutoff makes 0.854 “stable” for every application, and the scores do not quantify the probability that the reference clusters are true.
Diagnostics, Comparisons, and Limits
A stability result is most informative when the analysis makes potential failure visible. Check whether the same cells are included in each partition, whether labels were compared without regard to their names, and whether the perturbation changed only the intended component. Inspect local scores alongside the global measure: a favorable global summary can coexist with a small group of cells whose relationships change substantially. Also examine whether disagreement is concentrated around a boundary or spread broadly, while avoiding a biological explanation that the clustering comparison alone cannot support.
Different summaries answer different questions. The Rand index is easy to interpret as the fraction of agreeing pairs, but it does not adjust for chance agreement. The ARI provides a chance-adjusted whole-partition comparison, but remains a single global value. Cell-level scores localize disagreement, but depend on the set of other cells included and do not by themselves describe whether a cluster as a whole is reproducible. No one score is a complete substitute for specifying the perturbation and examining the pattern of disagreement.
Common failure modes include perturbing several analysis components at once and then attributing the result to one of them; interpreting cluster-number or label changes as disagreement without comparing cell relationships; and treating a high score as validation of biological truth. Another mistake is to report an average over several perturbations without identifying an important low-agreement result. The correction is to define the perturbation family, use label-invariant comparisons, report both the summary and relevant variation, and keep the conclusion limited to robustness under the tested changes.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Two runs assign the same cells to clusters but use different numerical labels. Is this by itself evidence of instability? Why?
- For a cell with 9 of 11 pairwise relationships preserved, calculate its cell-level score.
- Two partitions agree on 18 of 30 cell pairs. Calculate their Rand index and interpret it in context.
- Why can a high whole-partition agreement score fail to identify an unstable cell?
- A researcher changes the representation and clustering resolution simultaneously. What limitation does this create for interpreting the stability comparison?
Solutions. (1) No. Cluster labels are arbitrary; compare cell relationships or use a label-invariant measure. (2) \(S_i=9/11=0.8182\), so the cell preserves about 81.82% of its pairwise together/apart relationships in that comparison. (3) \(R=18/30=0.6000\); 60.00% of the unordered cell pairs have the same together/apart status in both partitions. (4) A global score averages over all pairs, so disagreement involving a small subset of cells can be diluted by many agreeing pairs; inspect cell-level scores as well. (5) The observed disagreement cannot be attributed to either change separately, because both components changed at once. A controlled follow-up would vary one component while holding the other fixed.