Why Separate Two Sources of Variation?
A clustering algorithm can return different assignments for two distinct reasons. First, it may be stochastic: different random initializations or random choices can produce different partitions even when every cell and every analysis setting is unchanged. Second, the result may be sensitive to the data supplied to the algorithm: changing the sampled cells, features, or measured values can alter the partition. These are different questions about reliability, so changing the seed and the data at the same time does not reveal which source caused a difference.
The clean starting point is to hold the data and analysis pipeline fixed, vary only the random seed, and measure assignment agreement. This estimates the algorithm’s repeatability conditional on that particular dataset and pipeline. Then introduce defined data perturbations and compare the resulting partitions, preferably while also repeating seeds. The result is not a universal score of “truth”; it is a decomposition by experimental design of two possible sources of observed assignment variation.
What Must Be Held Fixed?
Let \(D\) denote the fixed cell-by-feature data supplied to the analysis. Let \(\theta\) collect the non-random analysis choices, such as the feature set, normalization and dimensionality-reduction settings, distance or graph construction, clustering algorithm, resolution, and requested cluster number. Write the partition produced with random seed \(r\) as \(C(D,\theta,r)\). A seed comparison uses the same \(D\) and \(\theta\) in every run. If another setting changes, the comparison no longer isolates seed effects.
In practice, “same pipeline” requires more than recording a seed. Keep software versions and relevant computational settings fixed, and check whether parallel operations or hardware-dependent routines introduce additional nondeterminism. If repeated runs with the same seed differ, the nominal seed is not controlling all randomness; that issue should be diagnosed before interpreting differences across seeds.
A partition comparison must not depend on the names assigned to clusters. Cluster labels such as 1 and 2 are arbitrary: swapping them does not change the partition. The adjusted Rand index (ARI), introduced in the earlier material on chance-corrected agreement, is a label-invariant measure. For completeness, its calculation is stated here. For two partitions of the same \(n\ge 2\) cells, let \(n_{ij}\) be the number of cells in cluster \(i\) of the first partition and cluster \(j\) of the second. Let \(a_i\) and \(b_j\) be the corresponding cluster sizes, and let \(M=\binom{n}{2}\) be the number of possible cell pairs. Then, when the denominator is nonzero,
If the denominator is zero, define ARI as 1; for \(n\ge2\), this occurs only when both partitions consist entirely of singleton clusters or both consist of a single cluster. The numerator corrects the observed number of cell pairs co-clustered in both partitions by its expectation under a fixed-margin chance model; the denominator scales that excess agreement. An ARI of 1 indicates identical partitions, an expected chance-level value is 0, and negative values can occur. It is an agreement summary, not a probability that the partitions are correct.
Worked Example: Comparing Two Seed Runs
Suppose six hypothetical cells are clustered twice on exactly the same data and with the same settings. Run A gives clusters \(\{1,2,3\},\{4\},\{5\},\{6\}\). Run B gives \(\{1,2\},\{3,4\},\{5,6\}\). Each partition contains three within-cluster pairs: A has \(\{1,2\},\{1,3\},\{2,3\}\), while B has \(\{1,2\},\{3,4\},\{5,6\}\). Of the \(\binom{6}{2}=15\) possible cell pairs, only \(\{1,2\}\) is together in both partitions.
For the contingency table, the nonzero intersections are two cells in \(\{1,2\}\) with A’s first cluster, and one cell in each of the intersections \((\{1,2,3\},\{3,4\})\), \((\{4\},\{3,4\})\), \((\{5\},\{5,6\})\), and \((\{6\},\{5,6\})\). Thus the sum of \(\binom{n_{ij}}{2}\) is 1. The cluster-size pair totals are 3 for A and 3 for B. Substituting into the ARI formula gives
The two seed runs have low, but positive, chance-adjusted agreement on these six cells. Because only the seed changed, this difference is evidence of algorithmic variation conditional on this data and pipeline. It does not tell us whether either partition reflects biologically meaningful cell populations.
A Repeated-Run Design
A single pair of runs can be misleading: it may happen to show unusually high or low agreement. Use several distinct seeds and summarize the distribution of pairwise ARIs. With \(R\) runs, there are \(\binom{R}{2}\) pairwise comparisons. Report the number of runs, the range or a suitable center such as the median, and the actual analysis settings. These pairwise values share runs and are therefore dependent; they are useful descriptive summaries, not \(\binom{R}{2}\) independent replicates for a conventional standard error.
To study data perturbation, define the perturbation before running the analysis. Examples include cell subsampling, feature perturbation, or a specified change to the measured data. The earlier material on cell subsampling and feature perturbations treats those choices as distinct stability probes; they should not be blended into a single vaguely defined “perturbed” condition. If a perturbation changes which cells are present, compare assignments only for cells shared by the runs and state that restriction. A small overlap can make the comparison unstable or unrepresentative.
A useful design crosses data conditions and seeds: run multiple seeds on the original data and multiple seeds on each perturbed dataset. Within one fixed data condition, variation across seeds describes seed effects. Comparing conditions after averaging or summarizing over seeds describes sensitivity to the chosen perturbation. If the seed effect differs substantially between data conditions, there is an interaction: algorithmic randomness itself depends on the input data.
Record the input cells and features, preprocessing, clustering method, tuning parameters, software, and computational settings.
Run the unchanged pipeline repeatedly. Calculate label-invariant agreement for each pair of runs on the same cells.
Change only the specified input component, then repeat the clustering across seeds under that condition.
Report within-condition seed agreement separately from cross-condition agreement, and note any dependence of seed variability on the data condition.
Worked Example: Summarizing Seed Agreement
Suppose four hypothetical seed runs on the same fixed data produce six pairwise ARIs of \(0.91, 0.88, 0.94, 0.86, 0.90,\) and \(0.89\). The mean is
The observed range is \(0.86\) to \(0.94\). These hypothetical results indicate high agreement across seeds for this data and pipeline, with some variation among runs. They do not imply that 89.67% of cells are “correctly” assigned: ARI is a pair-based, chance-adjusted partition comparison. Nor should the six values be treated as six independent experimental replicates, because each of the four runs contributes to three comparisons.
Comparing Seed Effects With Data Effects
The comparison is clearest when both sources are assessed using the same agreement measure and the same population of cells where possible. Consider a hypothetical baseline partition A from the previous example. A second seed on unchanged data gives partition B, with ARI \(0.1667\) against A. Now suppose a defined data perturbation, followed by clustering, gives partition C: \(\{1,2,3\},\{4,5\},\{6\}\). Against A, the shared cluster \(\{1,2,3\}\) contributes three co-clustered pairs. The pair totals are 3 for A and 4 for C, and \(M=15\), so
In this particular hypothetical comparison, the changed seed gives less agreement with A than the specified data perturbation does. The correct conclusion is limited: these observed partitions differ by these amounts. It is not evidence that seed effects are generally larger than data effects. That conclusion requires multiple seeds and, for a broader claim about perturbations, multiple perturbation replicates and a clearly stated perturbation scheme.
A compact comparison makes the design logic explicit:
| Comparison | What changes? | What it describes |
|---|---|---|
| Same data, different seeds | Random seed only | Algorithmic repeatability conditional on the fixed input and pipeline |
| Different data conditions, same settings | Specified input perturbation | Sensitivity to that perturbation, averaged or summarized across seed runs |
| Different data and different settings | More than one source | A mixed comparison that cannot isolate either source |
Diagnostics, Failure Modes, and Limits
Low seed agreement can flag sensitivity to initialization, optimization, or stochastic graph construction. First verify that the data and settings really were fixed. Check the same-seed reproducibility, inspect convergence or termination messages, and record cluster counts and cluster-size distributions alongside the ARI. Agreement scores alone may conceal a relevant change, such as a small cluster disappearing, or may be low because of unstable borderline cells while major groups remain similar.
High seed agreement is also conditional evidence, not proof of robustness. Every seed run may reproduce the same artifact of the chosen preprocessing or feature set. Conversely, a data perturbation can produce low agreement because it was extreme or because it removed the cells that define a rare group. State what was perturbed and how many cells were available for comparison.
- Changing several things at once: If the seed, resolution, and feature set all change, the resulting ARI cannot isolate random-seed effects.
- Comparing arbitrary cluster names: Label 1 in one run need not correspond to label 1 in another. Use a label-invariant partition comparison rather than counting matching labels directly.
- Overinterpreting a mean: A high average can hide one poor run or a cluster-specific failure. Report a spread and inspect individual runs.
- Treating pairwise scores as independent: Comparisons reuse the same runs. Summarize them descriptively or use a design that respects their dependence.
- Calling stability validity: A reproducible partition may still be an unsuitable representation of the biology. Stability addresses repeatability under specified conditions, not whether clusters are true natural kinds.
Seed-only repeats are usually the simplest diagnostic and should precede claims based on perturbations. A factorial design is more informative when computational resources permit it, because it shows whether seed variability changes across data conditions. However, no design removes the need to define a meaningful perturbation, choose an appropriate agreement measure, and interpret results in the context of the analysis goal.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- A researcher changes the random seed and the number of selected genes between two runs. Can the ARI difference be attributed to the seed alone? Why?
- Three runs on the same data have pairwise ARIs \(0.80, 0.90,\) and \(1.00\). Calculate their mean and state what it summarizes.
- Why is it not valid to treat all pairwise ARIs from a large set of runs as independent observations?
- All seed runs agree closely, but a cell-subsampling perturbation gives low agreement. What can and cannot be concluded?
Solutions. 1. No. Both the seed and feature set changed, so the comparison mixes algorithmic randomness with a change in input representation. 2. The mean is \((0.80+0.90+1.00)/3=0.900\); it summarizes average pairwise agreement for these three fixed-data runs, not the fraction of correct assignments. 3. Pairwise comparisons reuse runs, so their values are dependent. 4. The clustering is repeatable across seeds conditional on the original data, but is sensitive to that particular cell-subsampling procedure; neither result establishes whether the partition is biologically true.