What Does It Mean for a Subgroup to Be Stable?
A discovered omics subgroup is an assignment of samples to categories produced by an analysis procedure. But an assignment from one dataset does not show whether the same samples would receive the same assignments if the data or analysis conditions changed slightly. Subgroup stability describes that reproducibility, always relative to a specified way of perturbing the analysis.
The key qualification is “specified.” A subgroup may be stable when a few observations are varied but unstable when the feature set changes. Stability is therefore not an intrinsic label such as “good” or “bad” attached to a subtype. It is a property of assignments under a defined perturbation scheme, analysis pipeline, and set of samples being compared.
Possible perturbations include changing which observations are used to develop the subgroups, varying the available features, or adding measurement variation. These choices ask different questions; a result for one does not automatically answer the others. The later material on stability assessment considers particular perturbation designs. Here the goal is to define the target clearly before choosing a design or summary.
There are also two related but distinct objects to assess. One is the reproducibility of a particular sample’s subgroup assignment. The other is the reproducibility of the partition as a whole: whether the same samples tend to remain together or apart. If subgroup names such as A and B are arbitrary, comparing the literal labels across runs can be misleading. A label-wise comparison requires a defensible correspondence between subgroups; without one, the comparison should focus on the grouping relationships rather than the names.
A Basic Reproducibility Summary
Suppose a reference analysis assigns sample \(i\) to subgroup \(g_i\). Apply the procedure to \(B\) perturbed versions of the analysis, and let \(g_i^{(b)}\) be the assignment for that sample in perturbation \(b\). When the subgroup labels have a meaningful correspondence across runs, a simple subject-level summary is the proportion of perturbations that reproduce its reference assignment:
Here \(I(\cdot)\) equals 1 when the condition is true and 0 otherwise. The value \(\hat{s}_i=1\) means that all compared perturbations reproduced sample \(i\)’s reference assignment; \(\hat{s}_i=0\) means none did. An overall summary can average these values over the samples being compared. It is a compact description, not a complete account: the average can hide a subgroup or a subset of samples with poor reproducibility.
This calculation estimates reproducibility only for the chosen perturbation scheme. If perturbations are independent draws from \(Q\), and the reference assignment is held fixed, the indicators in the sum have a common success probability \(s_i\). Their average has expectation \(s_i\) and variance \(s_i(1-s_i)/B\):
The expectation follows because each indicator has expectation \(s_i\); independence makes the variance of their sum equal to the sum of their variances. This justification depends on independent perturbation draws and a fixed reference. If runs are dependent, or if the reference is itself changed, the displayed variance expression does not apply as written. In all cases, \(\hat{s}_i\) describes observed agreement across the runs; it is not a probability that the subgroup is biologically real.
Worked Example: Reproducibility for Five Samples
In a hypothetical analysis, five samples receive reference labels A, A, A, B, and B. Four perturbed analyses produce the assignments below. Assume that A and B have been given a defensible correspondence across runs.
| Sample | Reference | Run 1 | Run 2 | Run 3 | Run 4 | Matches |
|---|---|---|---|---|---|---|
| 1 | A | A | A | A | A | 4 of 4 |
| 2 | A | A | A | B | A | 3 of 4 |
| 3 | A | B | A | A | A | 3 of 4 |
| 4 | B | B | B | B | A | 3 of 4 |
| 5 | B | B | A | B | B | 3 of 4 |
For sample 3, for example, three of the four perturbed runs reproduce its reference assignment A, so \(\hat{s}_3=3/4=0.75\). Across all five samples, there are \(4+3+3+3+3=16\) reproduced assignments out of \(5\times4=20\). The average agreement is therefore \(16/20=0.80\). Under this particular perturbation scheme, the average sample-level reproducibility is 0.80, with sample 1 more reproducible than the other four. This does not establish that the subgroups are well separated or biologically meaningful.
Stability Is Not Separation, Fit, or Validity
Several properties of a subgroup solution can be desirable, but they answer different questions. Keeping them separate prevents a common error: treating a single attractive feature of the analysis as proof of all the others.
| Property | Question it answers | What it does not establish |
|---|---|---|
| Stability | Do assignments reproduce under the specified perturbations? | That groups are far apart or biologically important |
| Separation | How distinct are the groups in measured features? | That assignments will reproduce after perturbation |
| Fit | How well does a chosen model or objective represent the analyzed data? | That the solution is reproducible or externally relevant |
| Biological or clinical validity | Does the subgroup correspond to meaningful biology or relevant outcomes? | That the discovered assignments are stable in new analyses |
Separation concerns differences between groups in the observed feature space. For example, a clustering procedure may produce groups with overlapping distributions for an assay feature, even if it returns almost the same assignments under repeated perturbations. Conversely, groups that look distinct in one dataset can have unstable boundaries if small changes shift which samples fall on either side. Stability alone cannot tell which situation holds.
Worked Example: Stable Assignments With Weak Separation
Suppose a hypothetical analysis assigns ten samples to A and ten to B. Across ten perturbations, all twenty samples retain their reference labels in every run. Thus, each sample has \(\hat{s}_i=10/10=1\), and the overall reproducibility is \(20\times10/(20\times10)=1.00\).
Now consider one assay feature whose mean is 0.10 units in A and 0.20 units in B, with a pooled within-group standard deviation of 1.00 unit. The difference in means is \(0.20-0.10=0.10\) units, or \(0.10/1.00=0.10\) pooled standard deviations. That feature shows little separation on this scale, despite perfectly reproduced assignments in the stated perturbations. Other features or the assignment rule may account for the stable partition, but the stability calculation itself says nothing about how distinct the groups are on this feature.
The conclusion is limited: assignments were reproducible under the ten specified perturbations, while this particular feature showed a small standardized mean difference. It would be incorrect to infer strong separation from the stability value alone.
Fit is also distinct. A fit criterion is computed on data according to a model or optimization objective; stability compares assignments across changed analyses. Even when the same objective is used, a solution with a better value on the original data need not yield more reproducible memberships. Fit values are comparable only when the objective and its evaluation conditions are comparable.
Worked Example: Better Fit Does Not Guarantee More Stability
Two hypothetical solutions are evaluated using the same within-group sum-of-squares objective on the same original dataset. Solution A has a value of 18, while solution B has a value of 20. Because smaller values indicate a better fit for this objective, A fits the original data better by 2 squared feature units.
In ten perturbed analyses, however, solution A retains the reference assignments for 6 of 10 tracked samples, giving \(6/10=0.60\) agreement. Solution B retains assignments for 9 of 10, giving \(9/10=0.90\). Thus A has the better original-data objective value, while B has higher observed assignment reproducibility under the specified perturbations. Neither number can substitute for the other, and this comparison does not show which solution is biologically preferable.
Biological or clinical validity requires evidence about whether subgroups correspond to relevant processes, measurements, or outcomes. That evidence may include external measurements or independent observations, interpreted in their scientific context. Repeatedly recovering the same partition is useful evidence about reproducibility, but it is not independent confirmation of biological meaning. Nor does an association between a subgroup and an outcome by itself establish that subgroup membership caused the outcome.
Worked Example: Stable Subgroups Without Evidence of Outcome Stratification
In a hypothetical cohort of 20 samples, the assignment procedure reproduces the same A and B labels for all samples in every compared perturbation. The assignments are therefore stable under that scheme. A separately recorded binary response occurs in 4 of 10 samples in A and 4 of 10 samples in B.
The response proportions are \(4/10=0.40\) in A and \(4/10=0.40\) in B, a difference of \(0.40-0.40=0.00\), or 0 percentage points. These data provide no observed difference in response proportions between the groups. The subgroup assignments may be reproducible, but the example gives no evidence that they stratify this response. It also cannot settle whether the subgroups have other biological relevance.
Assumptions, Diagnostics, and Scope
A stability statement is interpretable only when its comparison is well defined. State which samples are being compared, what is perturbed, and what parts of the analysis remain fixed. For a label-wise summary such as \(\hat{s}_i\), also state how subgroup labels are treated across runs. When subgroup names are arbitrary, a change from A to B may be a naming change rather than a change in the partition. A numerical label-wise score without a meaningful correspondence can therefore misrepresent reproducibility.
Check the result at more than one level. An overall average can appear high if most samples have reproducible assignments while a small subgroup is unstable. Inspect summaries by subgroup and by sample, and note how many perturbations and comparable samples contributed. If the set of samples differs between runs, the comparison applies only to samples that can be meaningfully compared; it does not describe assignment reproducibility for samples absent from a run.
The perturbation scheme defines the scope of the conclusion. A high value under one mild perturbation is not evidence of robustness to every plausible change in data collection or analysis. A very broad claim such as “the subtype is stable” should therefore be replaced by a precise statement about the samples, assignment procedure, and perturbations examined. The stability summary is descriptive unless an appropriate inferential framework has also been specified.
Common Mistakes
- Equating stability with separation. “The assignments barely changed, so the groups must be far apart” is not justified. Reproducibility concerns assignment changes; separation concerns distances or distributional differences in features.
- Equating fit with stability. “The model has the best objective value, so its subgroups are reliable” confuses performance on the analyzed data with reproducibility under perturbation.
- Treating stability as biological validation. Reproducing a pattern does not show that it corresponds to a meaningful mechanism or predicts a relevant outcome.
- Ignoring the perturbation definition. A stability result without a stated perturbation scheme has no clear scope. Different perturbations may test different kinds of reproducibility.
- Comparing arbitrary labels literally. A permutation of subgroup names can make identical partitions appear different. Establish a meaningful correspondence or compare grouping relationships instead.
Check Your Understanding
Answer each question, then compare with the solutions below.
- For one sample, 12 of 15 perturbed runs reproduce its reference subgroup. Calculate its empirical reproducibility.
- Why does a high overall stability average not guarantee that every subgroup is stable?
- A solution has a lower fit-objective value than another, but fewer assignments are reproduced under perturbation. Which solution is more stable under that perturbation, and why?
- If labels A and B are arbitrary, why might a direct comparison of label names be misleading?
Solutions. (1) The reproducibility is \(12/15=0.80\), or 80% of the compared runs. (2) An average can be dominated by samples or groups with high reproducibility; a smaller subgroup may have low reproducibility while the overall average remains high, so subgroup-specific results should also be examined. (3) The solution with more reproduced assignments is more stable under the stated perturbation. The lower objective value describes fit on the evaluated data, not assignment reproducibility. (4) Swapping the names A and B can leave the partition unchanged. A literal name comparison could count identical groupings as disagreements unless labels have a meaningful correspondence.