What a Stability Estimate Can—and Cannot—Show
A subgroup stability analysis asks whether a discovered membership pattern is reproduced when the data or analysis is perturbed. That question is useful only when each comparison is made on a consistent basis. A high score can be an artifact of mismatched preprocessing, arbitrary label names, an inappropriate resampling unit, or selective reporting. Even a correctly calculated, high stability score describes reproducibility under the chosen perturbation; it does not establish that a subgroup is a biologically real disease entity.
Let \(D\) denote the observed data and let \(A\) denote the specified analysis pipeline, including its preprocessing, feature selection, tuning, and clustering steps. A perturbation, indexed by \(b\), produces a fitted partition and subgroup memberships. Stability summarizes how similar those memberships are across perturbations, under a defined comparison rule. The interpretation therefore depends on what was perturbed, what was held fixed, which units were resampled, and how groups were matched.
For example, if \(S_0\) is a reference subgroup and \(S_b\) is its matched counterpart in perturbation \(b\), their Jaccard similarity is
When a replicate contains only some of the reference participants, compare memberships on the participants available for that replicate: restrict \(S_0\) to those participants before calculating the intersection and union. This avoids treating a participant who was not sampled as a disagreement. The set-based formula treats each participant as present or absent; if bootstrap duplicates are retained as multiplicities instead, that convention must be defined and used consistently.
Errors That Change the Comparison
A stability score is meaningful only if the two memberships being compared represent the same analysis target. The following assumptions are practical checks, not formal guarantees of good clustering: the target population and subgroup definition are specified; the compared samples have identifiable participant-level membership; the matching method is appropriate; and all preprocessing and fitting choices follow the stated perturbation design. A violation can change the score without reflecting genuine reproducibility.
Inconsistent preprocessing
The same nominal clustering algorithm can yield different groups when scaling, missing-data handling, batch adjustment, feature selection, or dimension reduction changes. Conversely, using information from the full data in some replicates but not others can make replicate results incomparable or introduce leakage. If the intended question concerns the reproducibility of an end-to-end analysis, data-adaptive preprocessing and tuning should generally be refitted within each resample. If the intended question instead concerns the stability of clustering under a frozen transformation, that transformation can be held fixed—but the resulting claim is narrower and should say so. A prespecified external calibration may also appropriately remain fixed.
A diagnostic is to compare the full sequence of transformations across reference and replicate fits: input features, missingness rules, scaling parameters, selected features, and tuning decisions. If they differ, determine whether the differences are part of the planned perturbation or accidental inconsistencies. Do not silently compare scores from pipelines that answer different questions.
Worked Example: A Scale Change Alters a Nearest-Center Assignment
Suppose two features are measured in incompatible numerical scales, and a point \(x=(2,0.9)\) is compared with cluster centers \(C_A=(0,0)\) and \(C_B=(10,1)\). First consider ordinary Euclidean distance on the raw values.
The distances are
On the raw scale, the point is closer to \(C_A\). Now suppose the first feature is divided by 10 before distances are calculated, while the second is unchanged. The transformed point is \((0.2,0.9)\), and the transformed centers are \((0,0)\) and \((1,1)\). Then
Now the point is closer to \(C_B\). This arithmetic does not show that either scaling is automatically correct. It shows that comparing memberships from inconsistent scaling rules can report differences caused by preprocessing rather than by the intended perturbation. The feature units and scaling rule must be justified and applied according to the analysis design.
Unaligned cluster labels
Cluster labels such as “1” and “2” are arbitrary identifiers. A replicate that swaps those identifiers has not changed the partition. Comparing the digits directly—such as counting cases where the reference label equals the replicate label—can therefore produce a misleadingly low agreement. Label-invariant partition measures, including variation of information, avoid this particular problem. When assessing a particular subgroup with a cluster-wise measure such as Jaccard similarity, first match the replicate cluster to the reference subgroup using an explicitly stated rule, such as the greatest overlap among eligible clusters.
The match itself should be checked. If the largest overlap is small or two candidate matches are nearly tied, the replicate may not contain a clear counterpart. Forcing a match in that situation can disguise ambiguity. State the matching rule before examining which rule produces the most favorable result.
Worked Example: Reversing Labels Does Not Reverse the Partition
In a reference partition of eight participants, subgroup A contains \(\{1,2,3,4\}\) and subgroup B contains \(\{5,6,7,8\}\). A replicate assigns label 1 to \(\{5,6,7,8\}\) and label 2 to \(\{1,2,3,4\}\). If the numerical labels are compared directly, all eight labels differ, so the apparent exact-label agreement is \(0/8=0\).
For subgroup A, the replicate cluster with the greatest overlap is label 2, containing exactly \(\{1,2,3,4\}\). Thus
The Jaccard similarity is 1, indicating identical membership on these participants. The discrepancy came entirely from arbitrary label names. For a larger or imperfectly reproduced partition, use a stated matching procedure and report cases where the match is ambiguous rather than treating label equality as meaningful.
Resampling the wrong unit
The resampling unit should reflect the independent sampling unit and the target question. In multi-omics data, one participant may contribute several assays, repeated visits, or technical replicates. Resampling assay rows independently can split information from the same participant across parts of an analysis, treating dependent measurements as independent. The resulting stability can be spuriously reassuring. Resample participant-level units when the target is reproducibility across participants, keeping that participant’s linked omics measurements together. If the target instead concerns variation across sites or cohorts, the resampling design must represent that level of variation.
A useful diagnostic is to write down exactly what one row represents at each stage, then check whether all records from the same independent unit remain together. Also check whether the perturbation matches the intended inference: resampling participants within one cohort does not, by itself, assess reproducibility across new cohorts or platforms.
Worked Example: Repeated Assays Are Not Independent Participants
Imagine five participants, each with two assay rows. Suppose an analyst takes an 80% subsample of the ten rows independently. For any particular participant, the probability that neither row is selected is \(0.2\times0.2=0.04\). Therefore, the probability that at least one row from that participant is selected is
If the question concerns reproducibility across participants, a participant-level 80% subsample instead includes a given participant with probability \(0.80\), with both assay rows kept together when included. The row-level design includes at least one measurement from a participant with probability 0.96, not 0.80, and can separate that participant’s linked measurements. It therefore represents a different perturbation and can make repeat measurements look like additional independent information.
The correction is to define the participant as the resampling unit and carry all linked assays together, unless a different unit is justified by the scientific target. The calculation does not quantify the exact bias in a stability score; it demonstrates why row-level sampling changes the inclusion structure.
Errors in What Gets Reported
Selective reporting
Reporting only the most stable subgroup, the most favorable metric, or the cluster count that looks best after inspecting many results can overstate reproducibility. This is particularly concerning if the reported choice was made using the same resampling results that are then presented as confirmation. The issue is not that summaries or selected targets are forbidden; it is that readers need to know the selection process and see enough of the results to understand it.
A basic diagnostic is to compare the stated analysis plan with the reported analyses. Check whether cluster counts, subgroups, perturbations, metrics, or thresholds were omitted after results were seen. A transparent summary should give the set of evaluated choices and the rule used to emphasize particular results. Avoid turning an exploratory maximum into a general claim about all discovered subgroups.
Worked Example: Reporting Only the Largest Stability Score
Suppose four prespecified subgroups have estimated stability scores \(0.91\), \(0.78\), \(0.61\), and \(0.58\). Their mean is
If the analyst reports only “stability was 0.91,” readers could reasonably—but incorrectly—take that value to summarize the four subgroups. It is the maximum, not the mean, and the other three estimates are appreciably lower. A defensible account identifies the 0.91 estimate as the largest of four, shows the other values, and explains any prespecified reason for focusing on that subgroup. If it was selected because its score was highest, that selection should be explicit; the maximum alone is not evidence that all four subgroups are stable.
Why a Stable Subgroup Is Not Necessarily a Biological Subgroup
A stable result can arise because the same technical or statistical structure is repeatedly present in the data. For example, a batch effect may consistently separate samples, and a strong technical signal can be reproduced across resamples. Stability then supports the claim that the pipeline repeatedly finds a pattern under the tested perturbations; it does not establish that the pattern represents a biological mechanism or clinically meaningful subtype. This distinction is especially important when a result is stable under resampling but the perturbations do not vary the source of concern, such as cohort, platform, or batch.
The earlier material on why stable clusters can be biologically meaningless emphasizes that reproducibility and biological interpretation are different claims. Here the practical error is treating the stability statistic as if it answered both questions. Interpret the statistic in its measured units and scope: for instance, a Jaccard score describes overlap of memberships, not the probability that a subgroup is biologically true. Biological interpretation requires evidence suited to that separate claim.
A Compact Diagnostic Workflow
State which subgroup, participants, perturbations, and membership comparison the stability estimate concerns.
Check that every adaptive transformation is refitted or held fixed according to the same prespecified design, and that differences are intentional.
Use a label-invariant measure or an explicit matching rule; inspect ambiguous or weak matches rather than forcing them.
Keep dependent records together and confirm that the resampling scheme reflects the intended source of variation.
Check for unreported choices and describe stability as conditional reproducibility, not as proof of biological truth.
These checks do not replace the stability measures introduced earlier, such as cluster-wise Jaccard stability, variation of information, consensus clustering, or end-to-end pipeline assessment. They protect the interpretation of those measures. A different resampling design can answer a different question, so there is no universally best perturbation independent of the target.
| Approach | What it can assess | Common caution |
|---|---|---|
| Participant-level subsampling or bootstrap | Sensitivity to changes in the sampled participants, under the specified design | Does not automatically assess new cohorts, platforms, or biological validity |
| Alternative resampling units, such as site-level perturbations | Sensitivity to variation at the chosen higher level | Requires enough independent units and a target that justifies that level |
| Label-invariant partition comparison | Agreement between partitions without relying on arbitrary label names | A whole-partition score may conceal instability in a particular small subgroup |
| Cluster-wise matching and Jaccard comparison | Overlap for a specified subgroup and its matched counterpart | Matching rules and ambiguous counterparts affect interpretation |
Common Mistake and Correction
A second frequent error is to assume that matching cluster numbers is sufficient. Since label names have no intrinsic meaning, use an explicitly defined matching step or a label-invariant comparison. If different reasonable matching choices produce substantially different conclusions, report that sensitivity instead of presenting one match as uniquely correct.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A replicate partition has the same members as the reference partition but swaps labels 1 and 2. Why can direct label agreement be misleading, and what should be done instead?
- A participant has three linked omics assays. The analysis resamples assay rows independently while claiming to measure stability across participants. Identify the problem and give the correction.
- Four subgroup scores are 0.80, 0.76, 0.55, and 0.49. What is their mean, and why would reporting only 0.80 as “the stability” be incomplete?
- A subgroup is stable when participants are resampled, but samples are always processed in the same batch. What does that stability result support, and what does it not establish?
Solutions. (1) Label names are arbitrary, so a label swap can make exact-label agreement appear poor despite identical membership. Match clusters using a stated rule, or use a label-invariant partition measure. (2) Assay rows from one participant are dependent and should not be treated as separate participants. Resample participants and keep their linked assays together, unless another unit is justified by the target. (3) The mean is \((0.80+0.76+0.55+0.49)/4=2.60/4=0.65\). The value 0.80 is the maximum, not a summary of all four groups; report the other scores and explain any selection. (4) The result supports reproducibility under participant resampling with that batch structure held fixed. It does not show robustness to batch variation or prove that the subgroup is biologically meaningful.