A Worked Stability Analysis for Two Cell Populations

Why a Two-Population Example Is Useful

A clustering method can return two groups even when the underlying cell populations overlap substantially. Conversely, populations that are well separated in a chosen representation can yield a convincing validity score and highly reproducible assignments. A controlled two-population simulation helps separate these situations: we set the population separation and sample size, then examine what an internal validity score and a resampling stability measure reveal.

The goal is not to identify a universal score threshold. A silhouette or stability value depends on the geometry, distance, clustering procedure, and resampling design. The useful question is how each measure responds as conditions change, and what conclusions remain justified.

This model makes assumptions that are intentionally simple: two equally frequent populations, equal within-population variances, and a one-dimensional Gaussian representation. Real single-cell data can violate all three assumptions, for example through unequal population sizes, non-Gaussian shapes, donor effects, or a continuous cell state. Here the simplified setting lets us vary overlap and sample size separately.

Overlap and Sample Size Are Different

Even if the population parameters were known, some cells would be difficult to classify when the distributions overlap. With equal prior probabilities and equal variances in this model, the optimal classification boundary is zero, halfway between the two means. The probability of assigning a cell to the wrong population is the same for each population:

where \(\Phi\) is the standard normal cumulative distribution function. The same error probability applies to population 2 by symmetry. This is a population-level limit for classification under the stated model; it is not a guarantee about the result of a particular clustering algorithm.

Sample size does affect how precisely the population means are estimated. If the numbers of observed cells are \(n_1\) and \(n_2\), and the model assumptions hold, the difference between their sample means has standard error \(SE=\sigma\sqrt{1/n_1+1/n_2}\). More cells reduce this uncertainty. They do not, by themselves, make two overlapping population distributions less overlapping.

Validity: What the Silhouette Measures

For a fixed partition and a chosen distance, the silhouette summarizes whether each cell is closer, on average, to its assigned cluster than to its nearest alternative cluster. For cell \(i\), let \(a(i)\) be the mean distance from \(i\) to the other cells in its assigned cluster. For each other cluster, calculate the mean distance from \(i\) to that cluster; let \(b(i)\) be the smallest such mean. For a cell in a singleton cluster, the silhouette is conventionally set to zero.

In the two-population simulation, a well-separated pair of groups will often have larger average silhouettes than a heavily overlapping pair, provided the chosen distance represents the simulated geometry reasonably. But the score is a property of the observed representation and partition. It does not establish that the groups are discrete cell types, nor that the chosen number of clusters is biologically correct.

Stability: Would the Assignments Recur?

Stability asks whether similar clustering assignments are recovered when the data are perturbed and the analysis is repeated. To compare runs, cluster labels must first be aligned because labels such as “cluster 1” and “cluster 2” are arbitrary. A useful design here is to repeatedly subsample cells, rerun the clustering procedure with the same specified preprocessing and settings, and compare assignments on a common set of cells. The comparison should account for cells absent from a subsample rather than silently treating them as assigned.

For a simple assignment-agreement summary, suppose each of \(R\) runs assigns every cell in a fixed reference set. After aligning labels, calculate the fraction of reference cells assigned to the same cluster as in a designated reference run, separately for each run. The mean of these fractions is an easy-to-read stability summary. Other established comparisons, such as adjusted Rand index or variation of information, use different summaries of partition agreement; their values and interpretations should not be interchanged.

Reading Validity and Stability Together

The following hypothetical results illustrate patterns that can arise when the two simulation factors are varied. The scores are illustrative, not universal expected values. Assume that the same distance, clustering algorithm, and resampling procedure are used throughout. Stability is summarized by mean matched assignment agreement on a common reference set.

The last two rows isolate the key distinction: sample size can improve the reproducibility of a fitted partition without materially improving the population separation measured by silhouette. That pattern is not guaranteed. Stability can remain low if the clustering algorithm is sensitive to initialization, if preprocessing changes across runs, or if the overlap region contains many cells whose assignments are intrinsically ambiguous.

Assumptions, Diagnostics, and Alternatives

The Gaussian model is useful for explanation, but conclusions drawn from its numerical overlap calculation depend on equal variances, equal prior probabilities, and the one-dimensional feature. The silhouette calculation requires a defined distance and a partition with meaningful within- and between-cluster distances. Resampling stability requires a perturbation design that reflects the intended source of uncertainty and a comparison that handles cluster-label permutations and missing cells correctly.

Useful diagnostics include plotting the simulated distributions and their overlap, checking whether cell-level silhouettes cluster near zero or include many negative values, and inspecting which cells change assignments across runs. A high overall silhouette can conceal a small, poorly separated subset; a high mean stability can conceal a rare cluster that repeatedly disappears. Also check whether conclusions change under reasonable preprocessing or feature perturbations. If the result depends strongly on a particular representation or random seed, report that sensitivity rather than presenting a single score as definitive.

The main alternatives answer related but distinct questions. The silhouette evaluates within-partition geometry for a chosen distance. Resampling agreement evaluates recurrence under a specified perturbation. In a simulation, comparison with the known generating labels can directly evaluate recovery of the simulated populations, but such labels are unavailable in ordinary data and recovery of a simulation’s labels does not establish biological validity. A plot can reveal shape or overlap that a scalar score hides, but visual separation is not a substitute for a defined quantitative comparison.

A common mistake is to reason, “The stability is high, so these must be true cell populations.” The correction is that stability establishes reproducibility only under the procedure and perturbations used. A method can repeatedly recover a partition imposed by a strong technical gradient or by a modeling choice. The earlier material on stable clusters and truth is directly relevant: reproducibility is valuable evidence about robustness, but it is not proof of biological meaning.