Stability Is Not Evidence of Discreteness
A clustering procedure can repeatedly divide the same samples in nearly the same way even when the biology varies continuously. A strong expression gradient, a batch offset, or a fixed choice of cluster count can each create reproducible boundaries. Run-to-run agreement answers whether a partition recurs under specified perturbations; it does not, by itself, establish that nature contains discrete biological subtypes.
A useful next question is comparative: Would a pipeline with no discrete subtype structure produce stability this high, if it retained other features of these data? A null or structure-preserving reference analysis makes this question explicit. It compares the observed stability with stability from data generated under a model that excludes the discrete-subtype explanation but preserves relevant non-subtype structure.
The null hypothesis is not simply “there are no patterns.” For example, a null model may allow strong, smoothly varying biological signals and batch differences. It asks whether discrete subtypes are needed to explain the observed stability, conditional on the structures the model retains. A failure to reject that null does not prove that subtypes are absent; it means the chosen reference model can produce stability at least as large as the observed value.
Specify the Comparison Before Interpreting It
Let \(D\) denote the observed multi-omics data and let \(A\) denote the complete clustering pipeline, including preprocessing, feature selection, the rule for choosing the number of clusters, and resampling. Define \(T(D)\) as a prespecified summary of stability from that pipeline. One choice is the mean adjusted Rand index (ARI) across the relevant pairs of resampled partitions. The ARI measures agreement adjusted for agreement expected by chance; its value is near 1 for highly agreeing partitions and near 0 for agreement near its chance baseline.
A reference analysis draws datasets \(D^{(1)},\ldots,D^{(B)}\) under a specified null model and calculates \(T(D^{(b)})\) by repeating the pipeline for each dataset. When the question is whether observed stability is unusually high, the reference p-value is the upper-tail proportion, with a correction for finite Monte Carlo sampling:
Here \(I\{\cdot\}\) equals 1 when the condition is true and 0 otherwise. If the observed statistic and the \(B\) simulated statistics are exchangeable under the null, their rank gives a valid Monte Carlo test; the added 1 avoids reporting a p-value of zero. This interpretation depends on the null model being appropriate and on the same analysis procedure being applied to the observed and reference data. A small reference p-value indicates that the null rarely produces stability this high. It does not establish that the clusters are biologically meaningful.
Specify which structures may remain, such as a smooth gradient or batch offsets, and what discrete structure is excluded.
Use a generative model or a structure-preserving rearrangement that retains the stated features without inserting discrete subtypes.
Apply the same preprocessing, selection, clustering, and resampling choices to each reference dataset.
Calculate the reference tail area, then check that the reference reproduces the data features it was intended to preserve.
A key design choice is what counts as “the full pipeline.” If the observed cluster count was selected after inspecting stability across several values, the reference analysis should repeat that selection rule; otherwise, the comparison ignores a source of selection. Similarly, feature selection or tuning based on the observed labels or outcomes must be repeated appropriately under the reference procedure. Holding these steps fixed is defensible only when they were fixed independently of the observed clustering question.
Worked Reference Analyses
Worked Example: A Stable Partition Along a Smooth Gradient
Suppose 120 hypothetical samples have multi-omics measurements that vary smoothly with a continuous inflammatory score. Investigators apply a fixed pipeline that selects two clusters. Across repeated subsamples, the mean pairwise ARI is \(T(D)=0.78\). They want to know whether a continuous gradient alone could generate this stability.
They fit a reference model with smoothly varying feature means along the score and residual variation, but no discrete mixture component. The model retains the observed score distribution and the estimated cross-feature covariance. For each of \(B=999\) reference datasets, they repeat the same subsampling and clustering pipeline. In 962 reference datasets, the resulting stability is at least 0.78.
The Monte Carlo reference p-value is
The observed stability is not unusually high relative to this continuous-gradient reference. Thus, a reproducible two-way split is compatible with discretizing the smooth signal; this analysis does not support interpreting reproducibility alone as evidence for two discrete biological subtypes. The conclusion depends on the reference adequately representing the gradient and covariance. A poor fit to either would weaken the comparison.
Worked Example: Stable Clusters That Track Batch
In a hypothetical data set of 80 samples processed in two laboratory batches, a clustering pipeline produces partitions with mean pairwise ARI \(T(D)=0.92\). The cluster labels are also strongly aligned with batch. To assess whether batch structure can account for the stability, the team specifies a null with batch-specific feature offsets and within-batch unimodal residuals, but no discrete biological subtypes.
They generate \(B=499\) reference datasets while keeping the batch assignments and sample counts fixed. The pipeline is repeated, including the same resampling scheme. In 82 reference datasets, stability is at least 0.92. Therefore,
This stability is not exceptional under the batch-preserving null. The result is consistent with batch structure contributing to a stable partition, but it does not prove that batch is the only explanation. The analysis should also check whether the reference reproduces the observed within-batch distributions and whether the cluster-batch alignment is as strong in the reference as in the data. If those diagnostics fail, the reference p-value may not answer the intended question.
Worked Example: Stability From a Forced Cluster Count
Consider 60 hypothetical samples whose three standardized measurements follow a single unimodal distribution with no discrete groups. Analysts nevertheless specify \(k=3\) for \(k\)-means. The observed mean pairwise ARI across the chosen resamples is \(T(D)=0.84\). They generate 199 reference datasets from a fitted single-component unimodal model and apply the same \(k=3\) pipeline. In 39 reference datasets, the stability is at least 0.84.
The stability does not exceed what this single-component reference often produces when the algorithm is required to return three clusters. The partition may be a reproducible geometric division imposed by the algorithm, rather than evidence of three natural groups. This example also shows why cluster-count selection matters: a fixed value of \(k\) defines what the algorithm must return, not what the biology contains.
Why the Reference Model Needs Diagnostics
A reference analysis is informative only to the extent that its null model represents the non-subtype structure relevant to the question. Preserving too little can make the observed stability look artificially unusual; preserving too much can reproduce the very separation being tested. The aim is not to make reference data look identical to the observed data in every respect. It is to state clearly which structures are treated as nuisance or continuous variation and which discrete claim is under examination.
- Check retained structure. Compare the observed and reference marginal distributions, covariance patterns, smooth relationships with continuous covariates, and batch-specific distributions when those features are meant to be preserved.
- Check what was removed. Confirm that the reference generator does not encode the observed cluster labels through group-specific means or another hidden label-dependent feature.
- Check pipeline symmetry. Use the same transformations, feature filtering, tuning, cluster-count selection, and resampling rules for the observed and reference data, except where the null requires a clearly stated change.
- Check sensitivity. Repeat the comparison under plausible alternative reference assumptions. A conclusion that changes substantially with small, defensible modeling choices is uncertain.
Different reference approaches answer different questions. A parametric reference can preserve modeled gradients, covariance, and batch offsets, but may be sensitive to distributional assumptions. A permutation or residual-rearrangement approach can avoid some distributional assumptions, but is valid only when the rearranged units are exchangeable under the null; unrestricted permutation can destroy batch or covariate structure that should be retained. A naive random partition or a null that preserves no structure is often an unhelpful comparator because it asks whether real structured data are more stable than structureless labels, not whether discrete subtypes are needed.
The reference p-value is not a probability that the null model is true, nor a measure of biological importance. A small value can arise from a misspecified reference, an unaccounted search across many analyses, or an unusually large data set; a large value can reflect limited power or a reference that is too permissive. Stability is one component of evidence. Biological interpretation also requires checking whether the partition captures a meaningful and coherent signal, rather than treating its reproducibility as proof of a categorical ontology.
Common Pitfall: Treating a High ARI as a Subtype Test
Correction: A high ARI establishes repeatability under the specified perturbations, not discreteness or biological meaning. Compare the same stability statistic with an appropriate null reference that retains plausible gradients, batch structure, and algorithmic constraints, and interpret the result in light of reference-model diagnostics.
Even a very small reference p-value supports only the limited claim that the chosen null model rarely generates stability as high as observed. It does not rule out other non-subtype explanations, and it does not establish that the clusters correspond to distinct biological states. Conversely, stability that is common under a credible continuous or batch-preserving reference is a direct warning against presenting repeatability alone as evidence for discrete subtypes.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A pipeline has observed stability 0.70. In 49 of 499 reference datasets, stability is at least 0.70. Calculate the Monte Carlo reference p-value.
- Why is a null model that destroys all batch structure unsuitable when batch effects are a plausible source of stability?
- A small reference p-value is obtained under a smooth-gradient null. State what it supports and one conclusion it does not support.
- If the observed cluster count was chosen by searching across several values, what should be done in each reference analysis?
Solutions. (1) \(\hat{p}_{\mathrm{ref}}=(1+49)/(499+1)=50/500=0.100\). (2) It fails to preserve a relevant alternative explanation, so observed stability could appear unusually high simply because the reference omits batch structure. (3) It supports that stability is unusually high relative to that specified gradient model, if the model is adequate; it does not prove that the clusters are biologically meaningful or exclude other non-subtype explanations. (4) Repeat the same cluster-count search and selection rule on every reference dataset so the comparison accounts for the selection process.