A Worked Stability Analysis With Multi-Omics Data

From a Stability Question to a Resampling Analysis

A subgroup solution is a product of choices: which subjects are included, how each omics block is processed and combined, and how the clustering algorithm is run. A stability analysis asks a specific question: under a defined perturbation of the data, how much does the specified analysis pipeline change its subgroup assignments? The answer concerns robustness to that perturbation; it does not by itself establish that the subgroups are biologically meaningful or will reproduce in an independent cohort.

The earlier material on end-to-end stability established why the full analysis pipeline, rather than only the final clustering step, is the relevant object to perturb. Here we carry that idea through from a written protocol to subgroup-level summaries. The example uses hypothetical matched subjects, each with measurements from two omics blocks. A subject is the resampling unit: both of that subject’s omics profiles must be included or omitted together.

Before calculating any summary, specify what is held fixed and what is allowed to vary. A useful protocol states the target subjects, the resampling unit and scheme, the number of clusters, preprocessing and integration steps, clustering settings, the label-alignment rule, and the summaries to report. Resampling the two omics blocks independently would break the matched-subject structure and could create combinations of measurements that were never observed together.

Worked Example: Specifying the Resampling Protocol

Aligning Labels Before Comparing Partitions

Cluster names have no intrinsic meaning. A resampled cluster called X may contain the subjects assigned to reference cluster A, while resampled label Y corresponds to reference cluster C. Comparing the printed labels directly would count such a match as disagreement. When both solutions have the same fixed number \(K\) of clusters, align them by finding the one-to-one assignment that maximizes overlap among common subjects.

The justification is direct. Every common subject contributes one count to exactly one cell of the overlap table. Under an alignment \(\pi\), the number of agreements is \(\sum_a M_{a,\pi(a)}\); therefore, the number of disagreements is \(n_{\mathrm{common}}-\sum_a M_{a,\pi(a)}\), where \(n_{\mathrm{common}}\) is the number of common subjects. Maximizing overlap is thus equivalent to minimizing label disagreements among those subjects.

Worked Example: Finding the Maximum-Overlap Alignment

This calculation assumes both partitions use the same fixed \(K\) and that a one-to-one correspondence is meaningful for the comparison. If the number of clusters differs, forcing a one-to-one alignment can hide splits or merges. In that setting, use summaries designed for partitions with differing cluster counts, such as variation of information, or report the split-and-merge structure directly.

Summarizing Stability at the Subgroup and Subject Levels

Alignment makes label-based summaries interpretable, but no single number describes every aspect of stability. Two useful summaries answer different questions. For a reference subgroup \(G_a\), compare it within a resample to its aligned cluster \(H_{\pi(a)}\), restricting \(G_a\) to subjects actually sampled. Their Jaccard similarity is the size of their intersection divided by the size of their union. A value of 1 means exact agreement among the sampled subjects; lower values indicate more membership differences.

Here \(S_r\) is the set of subjects in resample \(r\), and \(\pi_r\) is its label alignment. The denominator includes sampled reference members and sampled members assigned to the aligned cluster, so both omissions from the reference subgroup and additions to it reduce the Jaccard value. Report the distribution across resamples, not just its mean: a mean can conceal a subgroup that is usually stable but occasionally changes sharply.

A subject-level retention rate gives another view. For reference subject \(i\) in cluster \(a\), count the repetitions in which \(i\) was included and assigned, after alignment, to \(a\). Divide by the number of repetitions in which \(i\) was included. This denominator matters: an omitted subject supplies no evidence about its resampled membership. Pairwise co-clustering is also useful: for a pair, divide the number of repetitions in which both subjects were included and assigned to the same cluster by the number of repetitions in which both were included. It describes reproducibility of that pair’s joint assignment under the resampling scheme.

Worked Example: Interpreting Subgroup and Subject Summaries

Diagnostics, Alternatives, and Limitations

A stability summary is most useful when checked against the design and the underlying resampled partitions. Inspect subgroup sizes and overlap tables, the distribution of Jaccard values, and the inclusion denominators for subject-level rates. A very high overall agreement can be driven by a large subgroup while a smaller subgroup changes substantially. Conversely, a low score for a rare subgroup may reflect limited information about that subgroup; it should not be hidden by an overall average. Review unusually low-overlap repetitions for changes in preprocessing, convergence, or cluster sizes, and check whether the label assignment has near-tied alternatives.

The resampling scheme determines what perturbation the results address. Subsampling without replacement, as in the example, avoids duplicate subjects within a fit and directly evaluates reduced sample size. A bootstrap samples with replacement and can represent sampling variation differently, but duplicated subjects affect the fitting procedure and need deliberate handling when comparing memberships. Neither scheme is universally best; the choice should match the stability question and be stated explicitly.

Consensus co-clustering offers a complementary summary: for each pair, it estimates how often the pair is assigned together among repetitions in which both are observed. It avoids arbitrary cluster names and can reveal stable local relationships, but does not by itself provide a single aligned subgroup correspondence. Label-aligned Jaccard scores are easier to interpret for fixed-\(K\) subgroup retention, while partition-wide measures such as variation of information summarize differences between whole partitions. These summaries answer related, not interchangeable, questions.

Stability is conditional on the chosen subjects, pipeline, resampling scheme, and analysis settings. Stable clustering can still reflect batch structure or other non-biological variation, and a resampled reference comparison is not independent validation. A small number of repetitions gives a noisy summary; subgroup sizes, the distribution of scores, and uncertainty in the estimated summaries all matter. There is no universal Jaccard threshold that establishes a valid subtype.

• Mistake: “The resampled cluster is called B, so it matches reference cluster B.” Cluster names are arbitrary. Correction: align labels from overlap before using them in subgroup-specific summaries.
• Mistake: “A subject omitted from a resample counts as having lost its subgroup.” Omission is not a changed assignment. Correction: use only repetitions in which the subject was included as the denominator for its retention rate.
• Mistake: “High stability proves biological validity.” The repetitions test sensitivity to a chosen perturbation, not whether a subgroup is biologically meaningful. Correction: interpret stability alongside substantive evidence and independent evaluation.