Why Reporting Choices Matter
A statement such as “the subtypes were stable” is not interpretable on its own. Stability depends on what was resampled, which parts of the analysis were repeated, how partitions were compared, and whether the summary describes the whole partition or a particular subgroup. A report should make those choices visible before presenting a numerical result. Then a reader can tell whether the evidence concerns sensitivity to sampling, algorithmic randomness, or a narrower source of variation.
In this tutorial, a reference partition is the grouping obtained from the full analysis data, and a resampled partition is the grouping obtained from one resampled data set. Stability summaries describe agreement between these partitions under a specified procedure. They do not, by themselves, establish that the groups are biologically meaningful or will recur in a new population.
Specify the Design Before Giving the Number
The resampling design determines what a stability estimate can mean. State the resampling unit (usually the participant, not an individual omics measurement), whether sampling was with or without replacement, the resample size or fraction, the number of repetitions, and any stratification by cohort, batch, or other design variable. Also report how incomplete profiles, repeated measurements, and duplicated sampled participants were handled. If the design is stratified, give the strata and how many observations were drawn from each.
A second essential choice is the extent of refitting. An end-to-end assessment ordinarily repeats the steps whose sampling sensitivity is being evaluated: for example, data-dependent filtering, scaling, feature selection, dimension reduction, and clustering. If a step is held fixed, say so and explain why. Fitting preprocessing parameters once on the full data and then reusing them in every resample answers a different question from estimating those parameters anew. Neither design is universally correct; the report must make the target perturbation clear.
For a pipeline with random initialization, distinguish sampling variation from algorithmic variation. One approach fixes the initialization rule or random seed within each resampled fit, so changes primarily reflect the resampled data. Another repeats random starts within each resample, thereby including algorithmic variability. State which approach was used, including how the final fit was chosen if several starts were run. Give software and relevant version information, and report the random seed when it supports exact reproduction.
Worked Example: Writing a Reproducible Resampling Description
Suppose a hypothetical multi-omics analysis contains 240 participants from three cohorts. The investigators want to describe sensitivity to participant sampling while preserving each cohort's representation. They draw 80% of participants without replacement within each cohort and repeat the procedure 200 times.
For each repetition, they refit the filtering thresholds, scaling parameters, selected-feature set, dimension reduction, and clustering. They use the same rule for random initialization in each fit and compare each resampled partition with the full-data reference partition on participants included in that resample. These details distinguish this procedure from one that resamples individual molecular measurements or freezes feature selection.
A suitable report is: “We performed 200 cohort-stratified subsamples, drawing 80% of participants without replacement within each cohort. In every subsample, we refit filtering, scaling, feature selection, dimension reduction, and clustering using the prespecified initialization rule. We compared resampled assignments with the full-data reference assignments among participants present in each subsample.”
The description identifies the unit, scheme, fraction, repetitions, refitted steps, and comparison population. If the investigators used multiple random starts, they would also need to report the number of starts and how a fit was selected; “prespecified initialization rule” alone would not be enough to reproduce that choice.
Define the Comparison Measure and the Cluster-Level Result
Cluster labels are arbitrary: a group called “1” in one fit may correspond to group “3” in another. Therefore, raw label agreement is not a valid comparison unless labels have first been aligned. A report should give the alignment rule and calculate agreement only over the participants eligible for that comparison.
For a reference cluster \(C_k\) and a resampled cluster \(D_l\), restricted to the participants included in both partitions, the Jaccard similarity is the size of their intersection divided by the size of their union. A common cluster-level procedure matches each reference cluster to the resampled cluster with the largest Jaccard similarity, then records that similarity for the reference cluster. State whether matching is one-to-one; if so, specify the assignment rule. This matters when two reference clusters both have their largest overlap with the same resampled cluster.
It is often informative to report cluster-level precision and recall as well as Jaccard similarity. For a matched pair, precision is \(|C_k\cap D_l|/|D_l|\), and recall is \(|C_k\cap D_l|/|C_k|\). Precision describes the fraction of the resampled cluster that overlaps the reference cluster; recall describes the fraction of the reference cluster recovered in the resampled cluster. Their denominators differ, so they help explain whether an overlap reflects a relatively pure but incomplete recovery or a broad cluster containing many additional participants.
Worked Example: Calculating Cluster-Level Overlap
In a hypothetical comparison, 20 participants are present in both partitions. A reference cluster contains 10 of them, and its best-matching resampled cluster contains 9. Seven participants belong to both clusters. Assume this is the selected match under the stated matching rule.
The union contains \(10+9-7=12\) participants. Thus the Jaccard similarity is $$ J=\frac{7}{10+9-7}=\frac{7}{12}=0.5833. $$
The matched-pair precision is \(7/9=0.7778\), and recall is \(7/10=0.7000\). In context, 70.00% of the reference cluster was recovered in its matched resampled cluster, while 77.78% of that resampled cluster came from the reference cluster. These statements describe overlap among the 20 shared participants; they do not describe the ten reference members who might have been absent from the resample.
An overall partition measure can complement these cluster-level results. The adjusted Rand index (ARI), for example, compares whether pairs of participants are grouped together or apart in two partitions and adjusts the agreement for that expected under random assignment with the same cluster sizes. It ranges from \(-1\) to 1; 1 indicates identical partitions, and values near 0 indicate agreement around the chance-adjusted baseline. Report the ARI definition or implementation, the evaluated participant set, and how missing or repeated identities were treated. An overall score can conceal a poorly recovered small cluster, so it should not replace cluster-level summaries.
Summarize Variation Across Resamples
A stability result should show the distribution of the chosen measure across repetitions, not only its most favorable value. For an overall measure, report a center such as the median or mean and a spread such as the interquartile range or selected quantiles. For cluster-level results, show the corresponding distribution separately for each reference cluster, along with cluster sizes. If useful, report the fraction of resamples in which a cluster had no acceptable match under a prespecified rule. Define that rule rather than selecting a threshold after seeing the results.
Quantiles of resampling results are descriptive summaries of variation under the stated resampling design. They are not automatically confidence intervals for a population stability parameter. A confidence interval requires a defined target quantity and a justified inferential procedure. In particular, the variation across overlapping subsamples is not equivalent to independent repeated studies.
Worked Example: Reporting Resampling Variation
Suppose 100 hypothetical resamples produce Jaccard similarities for one reference cluster with a median of 0.76, a 25th percentile of 0.62, and a 75th percentile of 0.87. Fourteen of the 100 values are below a prespecified threshold of 0.50.
A concise report is: “Across 100 resamples, the cluster's median matched Jaccard similarity was 0.760 (interquartile range 0.620–0.870); 14% of resamples had similarity below the prespecified 0.50 threshold.” The percentage follows from \(14/100=0.14\), or 14%.
This report communicates both typical recovery and its lower tail. The interval from 0.620 to 0.870 is the interquartile range of resampling outcomes, not a 95% confidence interval for the cluster's true stability. Interpretation remains conditional on the resampling fraction, refitting choices, and comparison rule.
Choose Measures That Answer Different Questions
| Summary | What it emphasizes | Important reporting caution |
|---|---|---|
| Matched-cluster Jaccard similarity | Overlap of one reference subgroup with its best-matching resampled subgroup | State the matching rule and shared-participant population; small clusters can have volatile values. |
| Precision and recall | Purity of the matched subgroup and completeness of recovery, respectively | They are directional summaries and can differ substantially for the same match. |
| Adjusted Rand index | Agreement of pairwise grouping decisions across the full partition | An overall score may obscure instability in a small or rare subgroup. |
| Cluster-size distribution | Changes in subgroup prevalence or splitting and merging | Similar sizes do not guarantee that the same participants are grouped together. |
No single measure captures every aspect of reproducibility. Jaccard similarity gives an interpretable cluster-level overlap; ARI gives a chance-adjusted partition-level comparison; precision and recall distinguish two directions of cluster recovery. A report can use more than one, but should avoid presenting a long list of measures without explaining which question each answers.
Limitations, Diagnostics, and Common Pitfalls
Stability is conditional on the data representation, sampling scheme, analysis pipeline, and comparison procedure. A stable result may still reflect batch structure or other systematic artifacts rather than meaningful biology. Conversely, a small subgroup can appear unstable because few of its members are available in a given subsample. Report relevant cluster sizes and design constraints so readers can judge these possibilities; do not treat a stability score as a test of biological validity.
Potential warning signs include a wide resampling distribution, a large gap between overall and cluster-level summaries, repeated splitting or merging, and strong dependence on whether preprocessing or feature selection was refit. Also inspect whether some resamples exclude most members of a rare cluster: a low or undefined overlap in such a resample may reflect limited representation as well as changing assignments. Report how undefined comparisons were handled and how many occurred.
- Wrong: “The ARI was high, so every subtype was reproducible.” Correction: An overall ARI can hide a weakly recovered subgroup; report cluster-level results too.
- Wrong: “The 25th–75th percentile range is a confidence interval.” Correction: It is a descriptive range of resampling outcomes unless a separate inferential justification supports a confidence-interval interpretation.
- Wrong: “Cluster 1 matched cluster 1 in every fit.” Correction: Labels are arbitrary; report the identity-based matching procedure.
- Wrong: “We assessed stability by resampling,” with no further detail. Correction: Name the resampling unit and scheme, refitting choices, number of repetitions, comparison population, and summaries.
A useful alternative to one headline score is a compact table showing, for each reference cluster, its size, median matched overlap, spread across resamples, and match-failure rate, alongside an overall partition measure. This is more informative than either a single global score or a single unusually favorable resample. The limitation is that these measures still summarize only the perturbations actually examined; they do not substitute for independent evidence.
A Reporting Sequence
A clear report can follow this order: define the target partition and perturbation; describe the resampling and all refitted or fixed pipeline steps; specify the matching rule and comparison measures; present overall and cluster-level distributions with denominators; and close with limitations that bound the interpretation. Keep the distinction between “stable under this resampling design” and broader claims about reproducibility explicit.
Name the reference partition, the resampled fits, and the participants included in each comparison.
Give the sampling unit, scheme, fraction, repetitions, stratification, and treatment of randomness.
List the steps refit in each resample and the steps held fixed.
Report an overall measure, cluster-level recovery, cluster sizes, and resampling spread.
Describe what the results support under this design, and name what they cannot establish.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A study reports “median Jaccard = 0.81” without the resampling fraction or whether feature selection was refit. Name two details needed to interpret the result.
- A reference cluster has 12 members, its matched resampled cluster has 10, and they share 8 participants. Calculate Jaccard similarity, precision, and recall.
- Why should a high overall ARI not be treated as evidence that every subgroup is stable?
- A report labels the 5th and 95th percentiles of resampled Jaccard values a “90% confidence interval.” What is the problem with that wording?
Solutions. (1) Report the resampling scheme and fraction, and state whether feature selection was refit; also identify the comparison population and repetitions. (2) The union is \(12+10-8=14\), so Jaccard is \(8/14=0.5714\); precision is \(8/10=0.8000\); recall is \(8/12=0.6667\). (3) ARI summarizes agreement across the whole partition, so strong agreement for large groups can mask poor recovery of a small subgroup. (4) Those quantiles describe the resampling distribution under the chosen design. They are not automatically a confidence interval for a population quantity; use that label only with a defined estimand and justified inferential procedure.