Reproducibility Is Not Biological Truth
A clustering algorithm can repeatedly return the same partition when cells are subsampled, features are perturbed, or random seeds are changed. That reproducibility is useful: it suggests the assignments are not highly sensitive to those particular changes. But it does not establish that the partition corresponds to distinct biological populations. A stable division can reflect a technical artifact, a chosen representation, or a convenient cut through continuous variation.
The key question is therefore not simply, “Is this clustering stable?” It is, “Stable under which perturbations, and what feature of the data makes it stable?” Stability is conditional on choices such as the measured features, preprocessing, distance, clustering algorithm, and number of clusters. The earlier material on perturbation-based stability provides ways to measure reproducibility; here the focus is what such reproducibility can and cannot establish.
For example, let \(C_i^{(b)}\) be the cluster assignment of cell \(i\) in run \(b\). A pairwise summary is the proportion of runs in which two cells are assigned to the same cluster, among runs in which both cells are present:
Here \(B\) is the number of runs and \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. A value near 1 indicates that the pair is usually grouped together under these perturbations; a value near 0 indicates that it is usually separated. Neither value says why the cells are grouped, whether the grouping is biologically meaningful, or whether a different representation would produce it.
Worked Examples: Stable Partitions With Other Explanations
Worked Example: A Batch Effect Produces Perfect Stability
Hypothetical data. Suppose eight cells are all from one biological population, but four were processed in batch A and four in batch B. A measured technical feature \(x\) has values:
| Batch | Values of \(x\) |
|---|---|
| A | 0.0, 0.1, −0.1, 0.0 |
| B | 8.0, 8.1, 7.9, 8.0 |
Assume a two-cluster \(k\)-means analysis on \(x\), with ordinary squared Euclidean distance. The batch means are 0 and 8. Assigning each batch to its own cluster gives within-cluster sum of squares \(0^2+0.1^2+(-0.1)^2+0^2 + 0^2+0.1^2+(-0.1)^2+0^2=0.04\). The overall mean is 4, so putting all cells into one group would give a sum of squared deviations of \(128.04\). The large separation relative to within-batch variation makes the batch division an exceptionally compelling two-cluster solution for this feature.
Now suppose repeated subsamples contain cells from both batches and small feature perturbations do not erase the separation. In the hypothetical runs, every pair from the same batch is co-assigned whenever both cells are present, while every cross-batch pair is separated. Thus \(\hat{p}_{ij}=1\) for same-batch pairs and \(\hat{p}_{ij}=0\) for cross-batch pairs. The result is maximally stable by this pairwise measure.
Yet the stated data-generating explanation is that the cells belong to one biological population and the measured difference is technical. The analysis has robustly recovered batch structure, not demonstrated two cell types. A useful follow-up is to check whether assignments align with processing batch and whether the separation persists in analyses that appropriately address that technical variation. Simply repeating the same pipeline on similar data does not supply independent evidence against the batch explanation.
Worked Example: A Stable Cut Through a Continuous Gradient
Hypothetical data. Consider eight cells ordered by a single continuous state score: \(x=-3.5,-2.5,-1.5,-0.5,0.5,1.5,2.5,3.5\). The spacing is uniform; there is no unusually large gap or empty interval around zero. A two-means analysis nevertheless divides the values into the four lower and four upper observations.
The mean of the lower group is \((-3.5-2.5-1.5-0.5)/4=-2\), and the mean of the upper group is \(2\). The within-cluster sum of squares is \(5+5=10\): in each group, the squared deviations from its mean are \(2.25,0.25,0.25,2.25\), which sum to 5. The division gives a useful two-group approximation, but the equal spacing offers no evidence of a natural boundary between two populations.
Small perturbations of each value by at most 0.05 preserve the order and leave the central division strongly favored over a substantially unbalanced split. If repeated runs use such perturbations and the same \(k=2\), the identities on either side can remain highly reproducible. That stability reflects a robust way of dividing an ordered gradient under the chosen model and number of clusters. It does not turn the gradient into two discrete biological states.
This distinction matters when describing the result. “Two reproducible groups under a two-means representation” is supported by the analysis. “Two discrete cell types” requires additional evidence, such as a defensible biological boundary and corroboration that is not merely the same score or clustering procedure reused.
Worked Example: Stability Depends on Which Features Are Included
Hypothetical data. Suppose eight cells have a technical quality score \(q\) and a biological feature \(g\). Four cells have \(q=0\), and four have \(q=10\); within each set, the values of \(g\) are 0, 1, 0, and 1. Assume the large difference in \(q\) represents a technical artifact, and \(g\) does not distinguish the two sets.
In an unscaled two-feature Euclidean analysis, the squared distance between cells from opposite \(q\) groups is at least \(10^2=100\), before adding any contribution from \(g\). Within a \(q\) group, the squared distance between cells with different \(g\) values is 1. Consequently, \(q\) strongly governs the partition. If the analysis repeatedly retains \(q\), small perturbations to \(g\) are unlikely to change the assignments, and the clustering can appear stable.
But this stability is conditional on including \(q\) at its current scale. If a justified analysis excludes or appropriately handles that technical score, the evidence for the same division may disappear. The check is not to remove features until a preferred answer appears; it is to investigate whether the feature driving the partition is scientifically appropriate and whether conclusions persist under defensible representations. Stability across changes that leave the dominant driver untouched cannot rule out that driver as an artifact.
Why Stability Cannot Identify the Explanation
The limitation can be stated as an identification problem. Let the observed feature \(X\) be generated by a latent binary variable \(Z\) plus small noise. A clustering procedure can recover \(Z\) almost perfectly and do so consistently. But \(Z\) might represent a genuine cell state, a processing batch, or a technical quality category. If the observed values have the same distribution under two explanations, a stability statistic calculated from those values has the same distribution too. It cannot distinguish explanations that the observed data do not distinguish.
In a simple illustration, suppose \(X=aZ+\varepsilon\), where \(a\) is a large separation and \(\varepsilon\) is small noise. The partition by \(Z\) will resist many small perturbations because the separation \(a\) remains large relative to the noise. That mathematical fact is compatible with either a biological cause of \(Z\) or a systematic technical cause. Stability describes the persistence of the partition under the chosen analysis; interpretation needs evidence about what generated the separation.
This also explains why many repeated runs do not automatically solve the problem. If every run uses the same biased representation and preserves the same confounded feature, the runs can repeat the same systematic error. Increasing \(B\) can make an estimate of reproducibility more precise, but it cannot make the perturbation scheme test explanations it never varies.
Interpreting Stability With Other Evidence
A sound interpretation asks what is varied and what is held fixed. Subsampling asks whether assignments depend on particular cells; feature perturbations ask whether they depend on particular measured variables; preprocessing changes ask whether they depend on a transformation or scaling choice. These checks address different sources of sensitivity. Passing one is not a substitute for passing the others, and none alone establishes biological validity.
| Evidence or check | What it can support | What it cannot establish alone |
|---|---|---|
| High co-assignment across cell subsamples | Assignments are not driven only by a few sampled cells under that resampling scheme | That the partition is a set of discrete biological populations |
| Robustness to feature perturbations | The result does not depend entirely on the particular features varied | That unvaried or systematically biased features are harmless |
| Robustness to preprocessing choices | The result is not highly sensitive to the specific choices examined | That those choices or the common underlying representation are biologically appropriate |
| Alignment with technical variables | A possible technical explanation that deserves investigation | That technical variables necessarily caused the partition |
| Independent biological or external evidence | Corroboration beyond reproducibility within the clustering procedure | A universal guarantee that every assignment is correct |
Evidence should be interpreted in context. A partition that is stable and replicated across independent donors, while not being explained by batch or quality variables, is more persuasive than one that is stable only within a single processing batch. Likewise, a continuous trajectory may be reproducibly divided at a chosen resolution; the appropriate description can remain a partition of a continuum rather than a claim of discrete cell types.
No single stability threshold makes this judgment automatic. A high value is informative only together with the perturbation design, the representation, the clustering target, and plausible alternative explanations. Conversely, low stability warns that assignments are sensitive under the tested conditions, but does not by itself prove that no meaningful structure exists.
Common Pitfall and Practical Reasoning
Correction: The result supports reproducibility under the resampling scheme. A batch effect or other systematic feature can also reproduce across resamples, and a clustering method can stably discretize a continuous gradient. State what was perturbed, investigate what drives the partition, and seek evidence that can distinguish biological structure from alternatives.
A practical interpretation separates three claims. First, describe the observed partition and the evidence that it is reproducible. Second, specify the perturbations and analysis choices to which that conclusion is conditional. Third, assess whether technical structure, continuous variation, or representation choices offer plausible explanations. Only then decide how strongly to describe the clusters biologically.
Identify the perturbations, features, preprocessing, and clustering settings across which assignments persisted.
Check whether cluster membership tracks batch, donor, quality measures, or one dominant feature.
Ask whether the data support separated groups or whether the partition may be a cut through continuous variation.
Use appropriate independent biological or external evidence before interpreting reproducibility as evidence for distinct biological populations.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- If cells from the same batch are always co-assigned across the tested subsamples, what does that establish, and what does it leave unresolved?
- In the continuous-score example, why does a stable two-cluster solution not demonstrate a natural boundary?
- A cluster is stable when a technical quality feature is retained, but changes when that feature is excluded in a scientifically justified analysis. What should be reported?
- Why does increasing the number of resampling runs not, by itself, eliminate a systematic artifact?
Solutions. (1) It establishes reproducibility of those pairwise assignments under the specified subsampling scheme; it leaves unresolved whether batch, biology, or another feature explains them. (2) The values are evenly spaced, with no natural gap, so the method’s reproducible division can be a convenient cut through continuous variation. (3) Report that stability was conditional on including the feature, describe the sensitivity, and investigate the technical explanation; do not present stability as evidence of distinct biological populations. (4) More runs can estimate reproducibility more precisely, but if each run preserves the same artifact and does not test its explanation, the systematic effect can remain stable across all runs.