Troubleshooting Unstable Cluster Assignments

From an instability score to a diagnosis

A summary can show that a clustering changes across runs without showing why it changes. A cell might sit between two populations, belong to a small group that is inconsistently recovered, or change assignment when preprocessing changes. Alternatively, the same input may yield different results because the clustering algorithm is stochastic. These explanations call for different checks; a single overall stability score cannot distinguish them.

The aim here is to use cell-level assignment frequencies and cluster-level overlap together, while varying one source of perturbation at a time. The resulting patterns are diagnostic evidence, not proof that a cluster is a biologically true population. Earlier material on matching clusters across resampled runs established why cluster labels must be aligned before comparing them: labels such as “cluster 1” are arbitrary and may switch between runs.

Including omission is essential when runs do not all contain the same cells: the matched-cluster frequencies alone then need not sum to one and cannot be used as a probability distribution in the entropy formula. If the question is specifically about assignment conditional on inclusion, use \(r_{ik}=n_{ik}/m_i\) for \(m_i>0\) and calculate \(H_i^{\mathrm{incl}}=-\sum_{k:r_{ik}>0}r_{ik}\log(r_{ik})\). This conditional entropy describes assignments among runs in which the cell appears; report the omission frequency separately. If \(m_i=0\), conditional assignment entropy is undefined.

Entropy is in natural-log units when \(\log\) denotes the natural logarithm. Zero entropy means all observed outcomes are the same; higher entropy means outcomes are more evenly distributed. Its maximum depends on the number of possible outcomes. For example, with two cluster outcomes and no omissions, the maximum is \(\log(2)\), attained when the two frequencies are both one-half. Entropy is not a probability that a cell is misclassified.

Why entropy responds to mixed assignments

The formula is the average surprisal of the outcomes: an outcome with frequency \(q\) contributes surprisal \(-\log(q)\), weighted by its frequency \(q\). Thus \(H=-\sum q\log(q)\). For \(L\) possible outcomes, the uniform distribution has entropy \(\log(L)\), and no distribution over those outcomes has larger entropy. One justification uses the nonnegativity of Kullback–Leibler divergence from \(q\) to the uniform distribution \(u_j=1/L\):

This bound explains why raw entropy values should not be compared casually when the number of possible outcomes differs. Entropy also does not identify the cause of variation: a balanced split between two clusters and a mixture involving omissions can both yield uncertainty, but they indicate different problems.

Cluster overlap reveals group-level failures

Cell frequencies show which cells move, but a small group may be unstable even when most of its members do not have high entropy individually. Cluster overlap supplies a group-level view. For two runs, restrict attention to the cells present in both runs. If \(C\) and \(D\) are the sets assigned to a matched cluster in those runs, their Jaccard overlap is \(J(C,D)=|C\cap D|/|C\cup D|\). It ranges from 0, for disjoint sets, to 1, for identical sets. Restricting to shared cells avoids treating a cell absent from one run as though it had been assigned to a different cluster. Record coverage or omissions separately.

A matched cluster with low overlap may have split across multiple clusters or merged with another group. A single best match can conceal either pattern, so inspect the overlap with the next-best cluster too. Matching is useful for comparing labels, but it does not make a one-to-one match a complete description of a split or merge. For small groups, even a few changed cells can greatly alter the overlap; interpret the value alongside the cluster’s size and the cell-level frequencies.

Separate the perturbations

A useful diagnostic design changes one factor at a time, or crosses factors so their contributions can be compared. To assess stochastic variation, hold the cells and preprocessing fixed while changing random seeds. To assess preprocessing sensitivity, compare pipelines while holding seeds or other algorithm settings as comparable as possible. If data resampling is part of the design, distinguish cells omitted by a resample from cells assigned to a different cluster. Frequencies describe the perturbation scheme actually used; they are not automatically population probabilities.

Consider a crossed design with two preprocessing pipelines, each run with three seeds. For a particular cell, suppose the cell is assigned to cluster A in all three runs under pipeline P1 and cluster B in all three runs under pipeline P2. Pooling all six runs gives \(q_{iA}=3/6=0.5\) and \(q_{iB}=3/6=0.5\), hence \(H_i=0.6931\). But within each pipeline the cell has zero entropy. The pooled instability is therefore associated with the preprocessing choice, not with seed variation within either pipeline. This is why pooled assignment frequencies should be read alongside stratified frequencies.

Cluster-overlap summaries can make the same distinction at the group level: compare runs within a pipeline and then compare the matched groups between pipelines. High within-pipeline overlap but low between-pipeline overlap points to preprocessing sensitivity. Low overlap among different seeds under the same pipeline points to seed sensitivity. If both are low, both factors may matter, or the clustering may respond to their interaction.

Diagnostic workflow and failure modes

A frequent mistake is to divide assignment counts by the number of runs that included a cell, then interpret the resulting entropy as unconditional stability. That calculation is valid only as entropy conditional on inclusion. The correction is to include omission as an outcome for unconditional entropy, or to label the result explicitly as conditional and report omission separately.

Other warning signs include interpreting a low Jaccard overlap without checking cluster size, treating matched labels as evidence that groups are biologically equivalent, and reading assignment entropy as a calibrated probability of error. If many cells are omitted, compare omission rates across runs and groups; if the included cells differ systematically, conditional frequencies may describe a selected subset. Finally, repeated algorithm runs are not independent biological replicates. Stability supports reproducibility under specified perturbations, not biological truth or causal interpretation.

Alternative summaries answer different questions. Co-association measures how often pairs of cells cluster together and can reveal shared structure without requiring a single cell-to-cluster frequency display. Variation of information compares whole partitions, while cluster overlap focuses on corresponding groups. Neither replaces assignment frequencies for locating individual unstable cells. Use summaries that match the diagnostic question and examine disagreements rather than expecting all measures to rank runs identically.