Diagnosing Rare Cluster Stability

Why Rare Clusters Need Their Own Stability Check

A cluster containing only a small fraction of the cells can look stable or unstable for reasons that a single overall partition score obscures. In a resample, it may have no members available to recover; it may be available but fail to reappear; or it may reappear with only partial overlap or a substantially different size. These are distinct outcomes. Treating them all as “unstable” loses information, while counting only successful resamples can make a fragile rare group look reliably recovered.

The focus here is on diagnosing a candidate rare cluster across repeated resampled runs. The resampling design should be appropriate to the study; donor-stratified resampling, for example, preserves the intended donor structure. For each run, the cluster-matching rule and any criterion for calling a corresponding cluster detected should be held fixed. The earlier material on matching clusters across resampled runs provides the matching framework; the summaries below make explicit how to handle small-cluster availability, overlap, and size.

A matching rule should not be adjusted after inspecting the results merely to make the rare cluster look more reproducible. If detection requires a minimum overlap or other admissibility criterion, specify it in advance and report it. In some matching schemes, a corresponding cluster can be identified even when \(m_r=0\), for example by using information other than direct overlap. In a scheme that relies solely on overlap with reference members, detection cannot be assessed in that run. That distinction matters when reporting an all-run detection rate.

Availability, Detection, and Recovery Are Different

Let \(A\) be the number of runs with \(m_r>0\). Among these available runs, let \(D\) count runs in which the prespecified rule detects a corresponding cluster. When \(A>0\), \(D/A\) estimates the fraction of available runs with a detected corresponding cluster; when \(A=0\), conditional detection is unestimable from these runs. It does not count resamples in which the reference cluster had no members to recover.

The joint fraction of all runs with both available reference members and a detected corresponding cluster is \(D/B\). It is not generally the unconditional detection fraction: a corresponding cluster might also be detected in a run with \(m_r=0\), if the matching procedure can assess it. For an unconditional detection fraction, count detected corresponding clusters across all \(B\) runs, including assessable runs with \(m_r=0\), and divide by \(B\). If detection is not assessable in some runs, state the number of assessable runs and use that denominator instead of implying a result for all \(B\).

This identity follows because \(D/B=(A/B)(D/A)\): the joint fraction is the available-run fraction multiplied by the detection fraction among available runs. Thus, a low joint fraction can reflect infrequent availability, low conditional detection, or both. Reporting the two components helps distinguish those explanations.

Measure Overlap Rather Than Detection Alone

A detected cluster need not reproduce the reference membership well. For an available run with a detected matched cluster, define recall as the fraction of available reference members assigned to that matched cluster, and precision as the fraction of the matched cluster’s cells that are available reference members. The Jaccard index compares the intersection with the union. These measures describe different errors: recall is reduced by reference members left out, while precision is reduced by additional cells included.

The Jaccard denominator is the size of the union \(A_r\cup M_r\), since \(|A_r\cup M_r|=|A_r|+|M_r|-|A_r\cap M_r|\). Thus, \(J_r=1\) for identical sets and \(J_r=0\) for disjoint sets. If no corresponding cluster is detected, overlap is not defined from a matched set; report non-detection separately rather than silently dropping the run. A summary may additionally assign zero recovery to such available runs, but that convention must be stated.

For a rare cluster, the denominator in each run may be very small. If only two reference members are available, recall can change by 0.5 when one cell changes assignment. It is therefore useful to report the available-member counts alongside overlap summaries, and to examine the distribution across runs rather than relying only on a mean. Do not average per-run fractions without noting that runs with different \(m_r\) contribute equally to an unweighted mean; a pooled fraction weights runs according to their available cell counts and answers a different question.

Summarize Size Variation on a Comparable Scale

Among runs with a detected corresponding cluster, its size can vary. Raw cluster counts are comparable only when the number of sampled cells is the same across runs. When sampling fractions differ, compare the matched-cluster proportion \(|M_r|/|S_r|\), or another scale standardized to the resampling design, rather than interpreting raw counts alone. The reference proportion \(|C|/N\) gives a useful baseline, but a resampled cluster proportion need not equal it exactly, even under reproducible structure.

For \(K\) detected matched clusters with sizes \(q_1,\ldots,q_K\), where \(K\geq1\), the sample mean size is \(\bar{x}=\sum_{i=1}^{K}q_i/K\), and, when \(K\geq2\), the sample standard deviation is \(s=\sqrt{\sum_{i=1}^{K}(q_i-\bar{x})^2/(K-1)}\). If fewer than two clusters are detected, report the sample standard deviation and coefficient of variation as undefined; when \(K=0\), the mean is also undefined. When \(\bar{x}>0\), the coefficient of variation \(s/\bar{x}\) expresses the standard deviation relative to the mean. It is unitless and can help compare relative size variability across clusters with different average sizes. It does not correct unequal sampling fractions, nor does it make a small set of runs precise.

Diagnostics, Alternatives, and Limits

A useful rare-cluster assessment presents availability, conditional detection, overlap, and size together. Check whether the available-member counts are often zero or just one or two; whether a single run determines the mean overlap; whether detection depends on a matching threshold; and whether sampling fractions differ. Also inspect the distribution of run-level values. A mean can conceal a mixture of nearly perfect recoveries and complete failures.

Several common alternatives answer related but not identical questions. The adjusted Rand index compares whole partitions and adjusts pair agreement for chance, but a good global value can coexist with failure to recover a small cluster. Variation of information summarizes partition differences, while co-association summaries describe how often pairs of cells are grouped together. These global or pair-based summaries can complement rare-cluster-specific recovery and overlap; they do not replace reporting the cluster’s availability and denominators.

Rare-cluster diagnostics have important limits. A cluster that recurs under a chosen perturbation is reproducible under that perturbation, not necessarily a distinct biological population. Conversely, low recovery can result from too few sampled members, unstable boundaries, an unsuitable matching rule, or genuine sensitivity to the analysis choices. Resampling observations from the same study does not create independent biological replication, and repeated runs are not independent new experiments. Interpret stability conditional on the data, preprocessing, clustering procedure, and resampling design.

A Practical Reporting Sequence