A Diagnostic Workflow For Reproducible Subtypes

Why Stability Needs a Workflow

A subtype can look convincing under one summary and fragile under another. Two analyses may produce similar overall partitions while disagreeing about a small, clinically important subgroup; a cluster may recur often but contain different samples each time; or its membership may persist while its defining features change with preprocessing. A reproducibility judgment therefore needs to combine evidence about the partition, the particular subtype, its members, its features, and the analysis choices.

The goal here is a disciplined decision process, not a universal stability cutoff. The earlier material on end-to-end stability, individual subgroups, uncertain membership, influential samples, and subtype-defining features supplies the relevant diagnostics. The workflow makes their roles explicit: define what must reproduce, evaluate complementary evidence, investigate disagreements, and limit the final claim to what the analysis actually assessed. Stability under internal perturbations is not, by itself, evidence that a subtype reproduces in an independent population.

Specify the Target and the Evidence Before Looking

Start by naming the object whose stability matters. It may be the full \(K\)-group partition, one focal subtype, a set of samples assigned to that subtype, or a feature pattern used to characterize it. These are related but not interchangeable targets. For example, an overall partition score can conceal instability in a small subgroup. State the cohort or analysis data, the reference fit, the resampling or sensitivity designs, and the intended use of the subtype. Do not choose a favorable perturbation or metric after seeing the results.

Let \(z_i^{(0)}\) denote the reference assignment of sample \(i\), and \(z_i^{(b)}\) its assignment in perturbed fit \(b\), when sample \(i\) is present in that fit. Cluster labels are arbitrary: label 1 in one fit need not correspond to label 1 in another. Comparisons therefore require label-invariant partition measures or an explicit matching rule for a focal cluster. For a cluster-specific comparison, document how clusters are matched and what minimum overlap or similarity qualifies as a match. A fit with no credible counterpart should be recorded as a non-match, not silently omitted.

A defensible workflow also checks that the perturbations preserve the question being asked. Resampling should respect relevant design features such as cohort or batch when the sampling design calls for it. Sensitivity settings should be plausible alternatives, not deliberately extreme choices used to make the result fail. The preprocessing and feature universe must be comparable enough for the intended comparison. If a perturbation changes the target itself, explain that change rather than treating its output as a direct replicate of the original target.

Read the Evidence as Complementary Diagnostics

Partition agreement summarizes how similar two complete partitions are. One common label-invariant measure is the adjusted Rand index (ARI). Given a contingency table with cell counts \(n_{ij}\), row totals \(a_i\), column totals \(b_j\), and \(N\) samples, define \(S=\sum_{ij}\binom{n_{ij}}{2}\), \(R=\sum_i\binom{a_i}{2}\), \(C=\sum_j\binom{b_j}{2}\), and \(T=\binom{N}{2}\). Then

The adjustment subtracts the expected pair agreement under a generalized chance model with the table margins fixed. ARI is 1 for identical partitions and is near 0 for agreement at the chance-adjusted baseline; negative values are possible. It is a whole-partition summary, not a direct measure of whether a particular small subtype recurs.

Cluster recurrence asks whether the focal subtype has a counterpart across perturbed fits. For reference member set \(A\) and matched set \(B\), the Jaccard similarity is \(|A\cap B|/|A\cup B|\). Recurrence can be summarized as the proportion of eligible fits in which the subtype is matched under a prespecified matching rule. Report the similarity distribution as well as the recurrence proportion: a simple count of matches discards how closely members agree.

Sample uncertainty concerns whether individual members retain their assignments. For a sample appearing in \(B_i\) perturbed fits, its assignment frequency to the focal subtype is the number of those fits assigning it to that subtype divided by \(B_i\), after label alignment. Frequencies near 0 or 1 indicate consistent exclusion or inclusion in the specified perturbations; intermediate frequencies flag uncertain membership. These are empirical perturbation frequencies, not automatically calibrated probabilities that a sample truly belongs to a biological class.

Feature consistency asks whether the same features support the subtype across fits. Relevant summaries can include selection frequency and, for signed effects or contrasts, the frequency with which the direction agrees with the reference. A feature that appears often but changes direction is not consistent evidence for a stable directional signature. The feature set, direction convention, and comparison scale should be specified before interpreting these summaries.

Finally, sensitivity checks examine whether conclusions survive plausible changes in the analysis, such as preprocessing choices, feature filtering, or algorithm settings. Keep these results distinct from resampling results: the first probes dependence on analysis choices, while the second probes dependence on sampled observations under its resampling design.

A Decision Workflow With Explicit Gates

Use the evidence in sequence, and keep both favorable and unfavorable findings visible. A strong result on one dimension should not automatically compensate for a failure on another dimension that is essential to the scientific claim.

Worked Decisions

Justification, Diagnostics, and Limits

The workflow is deliberately not a weighted average of stability scores. The reason is that its components answer different questions. A global partition statistic aggregates pair relationships across the full sample, whereas recurrence concerns a chosen cluster. Membership frequencies concern particular samples, and feature consistency concerns the subtype’s characterization. A favorable value on one scale cannot logically establish a favorable result on another.

This separation also gives a practical diagnostic: when evidence disagrees, locate the level of disagreement before summarizing. High global agreement with poor focal recurrence can arise when other clusters are stable but the focal group is not. High recurrence with a few intermediate assignment frequencies suggests a stable core with an uncertain boundary. Stable membership with changing feature directions suggests that the group’s identity is less secure than its partition label. Sensitivity failures indicate dependence on analysis choices even if resampling results look favorable.

Interpretation depends on the design. Resampling should reflect the sampling structure and the intended source of uncertainty; treating dependent observations as independent can make recurrence appear more precise than warranted. Cluster matching rules must be meaningful for the target, and results can depend on those rules. Feature consistency can also be misleading if the feature set or direction convention was chosen after inspecting the same results. State these choices, show the relevant distributions or counts, and avoid presenting a cutoff as universally valid.

A stability workflow has further limits. It measures robustness to the perturbations actually examined, not to every conceivable data collection or modeling change. A highly stable partition can still lack biological meaning, and internal stability evidence does not establish transport to a new cohort. The appropriate output is therefore a bounded statement: what persisted, under which checks, which samples or features were uncertain, and what was not evaluated.

Common Mistakes

• Using one score as a verdict. A high global agreement statistic does not establish recurrence of every individual subtype. Report global and focal evidence separately.
• Dropping non-matches. Removing fits without a counterpart inflates the apparent recurrence among remaining fits. Define eligibility in advance and report unmatched fits.
• Calling resampling frequencies probabilities of truth. They describe assignments across the specified perturbations, not calibrated biological membership probabilities.
• Ignoring a sensitivity failure because resampling looked strong. Resampling and analysis-choice checks probe different sources of variation. Report both and narrow the claim if an essential result changes.
• Claiming external reproducibility from internal stability. Internal perturbations assess robustness within the analyzed data and design. They do not establish performance in independent data.