Replicating Molecular Subtypes in an Independent Cohort

What Does It Mean for a Molecular Subtype to Replicate?

A subtype discovered by clustering is a partition of a particular cohort, not automatically a universal category. A second cohort may show similar molecular structure while giving its clusters different numeric labels; alternatively, matching labels may conceal different patterns because the same clustering procedure was applied to different assays or populations. Replication therefore requires specifying what is being replicated and how it will be evaluated.

Two complementary questions are useful. First, can a subtype definition learned in the discovery cohort be applied to independent samples, with its preprocessing and assignment rule held fixed? Second, if the validation cohort is clustered independently, does its partition agree with the transferred assignments or with the discovery structure? The first assesses transport of a particular definition; the second asks whether a similar grouping reappears without forcing the discovery labels onto the new data. Neither question alone establishes that subtypes are distinct biological entities.

Two Validation Strategies and Their Assumptions

In frozen assignment validation, the discovery analysis produces a rule, such as subtype centroids in a specified feature space. Validation samples are processed using the prespecified compatible transformation and assigned without refitting the centroids or choosing features based on validation outcomes. This tests whether the discovery rule is usable in the new cohort. It does not by itself show that the validation cohort would have yielded the same groups under independent clustering.

In independent rediscovery, the validation cohort is clustered separately using an analysis plan fixed in advance. The resulting partition is compared with transferred subtype assignments or with another independently derived partition. Cluster numbers are arbitrary: cluster 1 in one analysis need not correspond to cluster 1 in another. Comparisons must therefore be invariant to relabeling, or explicitly match labels using a prespecified method. A label-invariant statistic such as the adjusted Rand index (ARI) is often useful.

Both strategies require independent subjects and a meaningful correspondence between measured features. Preprocessing must not use validation outcomes or labels in ways that leak information into the rule. A transferred assignment also assumes that the validation measurements can be put on the discovery scale, or that the rule explicitly accounts for platform differences. When assay panels differ, comparisons should be restricted to a justified common feature set or use a prespecified mapping; missing or noncomparable features cannot simply be treated as equivalent measurements.

Cohort and platform effects are especially important. If assay platform is nearly identical to cohort membership, a pattern attributed to subtype may instead separate technical batches or populations. The earlier material on cohort and platform confounding is relevant: statistical adjustment cannot reliably separate effects that the design does not distinguish. Validation is strongest when subtypes and technical conditions are represented across cohorts, rather than perfectly aligned with them.

Applying a Frozen Definition

Consider a discovery analysis that standardizes two molecular features using discovery-cohort means and standard deviations, then defines two subtype centroids. For a validation sample, apply those same discovery means and standard deviations and calculate its distance to each centroid. Assign the sample to the nearest centroid only if this rule was specified before examining validation results. If features, scaling, or centroids are refit using validation labels or selected to improve agreement, the result is no longer an independent test of the frozen rule.

Comparing Partitions With the Adjusted Rand Index

A contingency table cross-classifies the same samples by two partitions. Let \(n_{ij}\) be the number assigned to group \(i\) in the first partition and group \(j\) in the second. Let \(a_i=\sum_j n_{ij}\), \(b_j=\sum_i n_{ij}\), and \(n=\sum_{ij}n_{ij}\). The quantity \(\sum_{ij}\binom{n_{ij}}{2}\) counts sample pairs placed together by both partitions; \(\sum_i\binom{a_i}{2}\) and \(\sum_j\binom{b_j}{2}\) count pairs placed together by each partition.

The adjustment subtracts the expected number of jointly grouped pairs under a chance model that fixes the two partitions’ group sizes and randomly allocates memberships. Thus the ARI compares observed pair agreement with chance agreement conditional on those margins. Its value is 1 for identical partitions, is near 0 on average under that chance model, and can be negative when agreement is less than expected. It is not a probability, and there is no universal cutoff that proves replication.

The justification follows by defining \(A=\sum_{ij}\binom{n_{ij}}{2}\), \(B=\sum_i\binom{a_i}{2}\), and \(C=\sum_j\binom{b_j}{2}\). Under random allocation conditional on the margins, a pair together in the first partition has probability \(C/\binom{n}{2}\) of also being together in the second, so \(E(A)=BC/\binom{n}{2}\). Subtracting this expected agreement gives the numerator. The denominator is the average of the two partitions’ total within-group pair counts, less that same expected agreement; hence identical partitions attain 1 when the denominator is nonzero. If both partitions contain only one group, the denominator is zero and the ARI is undefined, not evidence of perfect subtype recovery.

Interpreting Concordance and Diagnosing Failure

A second, easier-to-read summary is the proportion of samples whose labels agree after matching the cluster names. For two partitions, matching can be chosen to maximize the diagonal total, but this choice makes the summary optimistic on the data used to select the match. In contrast, ARI is label-invariant and adjusts for chance agreement given group sizes. It is often useful to report both, making clear how labels were matched and not presenting raw agreement as a chance-adjusted result.

Diagnostics should examine more than a single agreement statistic. Check subtype frequencies, missingness, feature distributions, and whether transferred samples fall far outside discovery-cohort ranges. Inspect whether discordance is concentrated by site, assay platform, ancestry or other population structure, or clinical composition. Test sensitivity to reasonable preprocessing choices and to the common-feature restriction when panels differ. If the apparent concordance disappears after addressing an obvious technical separation, that weakens the replication claim.

For uncertainty, resample independent subjects rather than individual feature measurements, and recompute the complete comparison for each bootstrap sample. A bootstrap interval describes sampling variability under the resampling design; it does not account for all sources of model-selection or platform uncertainty. If subtypes were discovered and many specifications tried before one was selected, a narrow interval computed only after selection can be misleading. Keep the validation analysis prespecified and report the number and nature of the attempted comparisons.

Common Pitfalls, Alternatives, and Limits

• Assuming the same numeric label means the same subtype. Cluster labels are arbitrary. Use a label-invariant measure or state and justify the matching rule.
• Refitting until validation agreement looks good. This turns validation into model development. Freeze features, transformations, and assignment rules before evaluating the independent cohort.
• Equating transfer with rediscovery. Applying a frozen classifier tests transport of that classifier; independent clustering tests whether a similar partition emerges. They answer related but distinct questions.
• Treating technical harmonization as biological replication. A common scale is necessary for some comparisons but cannot remove perfect cohort-platform confounding or make nonoverlapping measurements interchangeable.

ARI is a strong general-purpose partition comparison, but alternatives may be useful. Adjusted mutual information is another chance-adjusted, label-invariant comparison. Matched-label agreement is intuitive when subtype correspondence is meaningful, while per-subtype sensitivity and positive predictive value can reveal poor recovery of a small subtype that a global statistic obscures. No single summary substitutes for inspecting the contingency table and subgroup-specific performance. Conclusions also depend on cohort size, subtype prevalence, assay coverage, and the stability of the discovery definition. Replication of a partition is evidence about reproducibility of structure, not causal evidence or a complete account of subtype biology.

The central practical discipline is to define replication before looking at validation results: specify whether the target is transport of a frozen assignment rule, independent recovery of a partition, or both. Then use a comparison appropriate to that target, document assay and cohort constraints, and interpret agreement as evidence with limits rather than as a binary certificate of biological truth.