Why Assess the Entire Pipeline?
A clustering result depends on more than the final clustering algorithm. In multi-omics data, it may also depend on how measurements are transformed, which features are selected, how tuning parameters are chosen, and how the clustering routine is initialized. A partition can look reproducible if the last step is repeated while all earlier choices are held fixed, yet change substantially when the full analysis is repeated on perturbed data.
End-to-end stability asks a broader question: if the analysis were carried out again on a perturbed sample, following the same analysis rules, how similar would the discovered subgroups be? This is a reproducibility assessment for a specified pipeline and perturbation scheme. It is not, by itself, evidence that the subgroups are biologically meaningful or that they will reproduce in a new population.
Specify the Pipeline Before Repeating It
For a resampled data set \(D_b\), a pipeline might estimate preprocessing parameters, transform the omics measurements, select features, tune clustering settings, and then fit a clustering procedure. Write this schematically as
where \(\hat\eta_b\) contains preprocessing quantities learned from \(D_b\), \(\hat\gamma_b\) represents the selected features, \(\hat\phi_b\) represents any tuning choices, and \(u_b\) represents algorithmic randomness such as initialization. The exact notation is less important than the operational rule: anything estimated from the data must be estimated again using that resample.
For example, if features are standardized using sample means and standard deviations, those quantities are recalculated within each resample. If features are selected by a data-dependent score, the score and selected set are recalculated. If a tuning rule selects a parameter by an internal criterion, that rule is run again using only the resample’s training data. The same rule must be used each time, even though its selected features or tuning values may differ.
Before starting, state the resampling unit and scheme, the pipeline steps to be refit, the tuning rule, the clustering settings, and how partitions will be compared. The perturbation scheme should reflect the sampling structure relevant to the question. The sampled units should be exchangeable under that scheme; if the data have dependencies or a complex sampling design, an ordinary subject-level resample may not represent plausible variation. A resample also needs enough observations for the specified preprocessing, selection, tuning, and clustering operations to be defined.
A Practical End-to-End Assessment
For each resample, run the entire prespecified pipeline and retain both the final partition and intermediate outputs. When two runs contain different subjects, compare their partitions on subjects appearing in both runs, after matching their identities. A label such as “cluster 1” has no intrinsic meaning across runs, so use a partition-comparison measure that is invariant to label permutations, such as the adjusted Rand index (ARI). If a method instead requires labels to be aligned, perform alignment before using label-specific summaries.
An overlap comparison has a limitation: the set of shared subjects can vary across pairs of runs. Record the overlap size and inspect whether it is large enough for the comparison to be informative. Another option is to define a common evaluation set and use each fitted pipeline to assign its members to clusters, but that requires a prespecified, valid assignment rule. Not every clustering method naturally provides one. In either design, do not silently compare labels for different subjects as though they were the same observations.
Record the resampling unit, every data-dependent step, the tuning rule, and how run-to-run partitions will be compared.
Within each resample, refit preprocessing, repeat feature selection, retune according to the same rule, and refit clustering.
Compare partitions on a common set of identified subjects, and retain the comparison-set size and any relevant tuning or selection outputs.
Report the distribution of partition-agreement measures and examine which pipeline steps vary, rather than relying on one average alone.
Worked Examples
Worked Example: Agreement After the Full Pipeline Is Refit
Hypothetical data. Two resamples are run through the same complete pipeline. Their shared evaluation set has eight subjects. The first run assigns four subjects to each of two clusters; the second also has two clusters. On the shared subjects, their contingency table is:
| Run 2, cluster 1 | Run 2, cluster 2 | |
|---|---|---|
| Run 1, cluster 1 | 3 | 1 |
| Run 1, cluster 2 | 1 | 3 |
The ARI compares the observed number of subject pairs placed together in both partitions with the amount expected under a chance-adjusted reference based on the cluster sizes. For contingency counts \(n_{ij}\), row totals \(a_i\), column totals \(b_j\), and \(n\) shared subjects, its formula is
Here, \(\sum_{ij}\binom{n_{ij}}{2}=\binom{3}{2}+\binom{1}{2}+\binom{1}{2}+\binom{3}{2}=6\). Each run has row or column totals \(4,4\), so both corresponding sums equal \(\binom{4}{2}+\binom{4}{2}=12\), while \(\binom{8}{2}=28\). Therefore,
The two end-to-end runs show weak agreement beyond the chance-adjusted reference, despite having the same cluster sizes. This single comparison is not a general stability estimate: it concerns only these runs and these eight shared subjects. A broader assessment summarizes many prespecified resamples and considers how much the overlap sets vary.
Worked Example: Tracking Feature Selection Across Pipeline Runs
Hypothetical data. A pipeline is rerun on ten resamples. In each run, feature selection is repeated from the start. Feature A is selected in 8 runs, feature B in 5, and feature C in 2. The selection frequency for a feature is its number of selections divided by the number of runs:
The corresponding frequencies are \(5/10=0.50\) for B and \(2/10=0.20\) for C. These values describe how often each feature entered the fitted pipeline under this particular perturbation scheme. They do not establish that A is a causal driver or that B and C are irrelevant. If the selected features differ while partitions agree, the pipeline may have interchangeable feature choices; if both feature choices and partitions vary, the instability may enter earlier than the final clustering step.
Worked Example: Comparing Frozen and Refit Pipelines
Hypothetical data. Four matched perturbations are used for two assessments. In the end-to-end assessment, preprocessing, selection, tuning, and clustering are all refit; the four ARIs are \(0.82, 0.61, 0.74,\) and \(0.53\). In a comparison assessment, preprocessing, selection, and tuning are frozen at their original values, and only clustering is refit; the matched ARIs are \(0.90, 0.84, 0.88,\) and \(0.80\).
The mean end-to-end ARI is
The mean for the frozen-pipeline assessment is
In this hypothetical comparison, agreement is lower when upstream steps are allowed to vary. The \(0.180\) difference between these means is a descriptive contrast for these four matched perturbations, not a test of a population difference. The frozen assessment is conditional on the original preprocessing, selection, and tuning choices; it cannot substitute for the end-to-end result when the target is reproducibility of the whole pipeline.
Interpret the Summary, Not Just the Average
A mean or median ARI compresses a collection of comparisons into one number. Report the spread and range as well, and note how many resamples and subject-pair comparisons contributed. A high average can conceal a small number of poorly agreeing runs; a low average may be driven by a subset of perturbations or by very small overlap sets. Co-clustering summaries and cluster-wise measures can add detail about which subjects or subgroups vary, but they answer related, not identical, questions.
Intermediate outputs help locate where variation enters. Record selected-feature sets, chosen tuning values, preprocessing estimates, cluster sizes, and any failures to fit the pipeline. These are diagnostics of the specified pipeline, not independent tests that can replace partition comparisons. For example, stable selected features do not guarantee stable partitions, and stable partitions do not imply that every selected feature is reproducible.
If algorithm initialization is random, decide whether to hold random seeds fixed or vary them. Holding seeds fixed can make a comparison more focused on sample perturbation, but does not measure sensitivity to initialization. Varying seeds measures a mixture of sampling and algorithmic variation. State which design was used; do not interpret the resulting stability as isolating one source unless the design does so.
Common Mistakes, Alternatives, and Limits
- Refitting only the clustering step. This estimates stability conditional on frozen upstream choices, not end-to-end stability. Refit each data-dependent step that belongs to the specified pipeline.
- Selecting features before resampling. The held-out or resampled observations have then influenced the feature set. Repeat feature selection within each resample, using only the data permitted by the design.
- Comparing labels directly. Cluster labels are arbitrary across runs. Use a label-invariant partition measure such as ARI, or align labels before a label-specific comparison.
- Reporting only one summary. A single mean omits spread, overlap size, and run-level failures. Report enough diagnostics to show whether the summary is representative.
- Calling stability validation. Repeated perturbations of one data source assess reproducibility under those perturbations; they do not by themselves establish performance or replication in an independent cohort.
An alternative is to assess only a fixed final model, which is computationally cheaper and useful when the question is specifically about conditional clustering variability. End-to-end reruns are more demanding but answer the broader pipeline question. Independent validation addresses a different target and cannot be inferred from resampling stability. No stability score establishes that a partition is clinically useful, biologically interpretable, or correct; those claims require evidence suited to them.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A pipeline standardizes features once using the full data set, then resamples subjects and repeats clustering. Is this an end-to-end assessment? Why or why not?
- Two runs have different cluster labels but identical subject groupings. What should a label-invariant partition comparison indicate?
- In 12 full pipeline runs, a feature is selected 9 times. Calculate its selection frequency and state what that frequency does—and does not—mean.
- Why should the number of shared subjects be reported when ARIs are computed on pairwise run overlaps?
Solutions. (1) No. Standardization was estimated using the full data and then frozen; the scaling parameters must be refit within each resample if they are part of the pipeline. (2) The comparison should indicate perfect partition agreement, because renaming clusters does not change membership. (3) The frequency is \(9/12=0.75\), or 75%. It describes selection under the specified pipeline and perturbation scheme, not biological importance or causality. (4) The ARI is calculated on the shared subjects, so different or small overlap sets change the comparison’s basis and may make it less informative.