Diagnosing Stability Of Individual Subgroups

Why Track Subgroups Individually?

An overall stability score can summarize agreement between two clusterings, but it may hide what happens to a particular subgroup. A solution could reproduce most of its clusters while repeatedly losing one clinically relevant subgroup. Conversely, one subgroup could persist well even when other clusters make the overall partition look unstable. Individual tracking asks a more specific question: does this reference subgroup reappear with much the same membership across resamples, and if not, how does its membership change?

This tutorial uses membership overlap to trace reference subgroups across resampled clusterings. It focuses on four patterns: persistence, fragmentation into multiple clusters, merging with another subgroup, and disappearance. The resulting diagnostics complement the whole-partition summaries covered in the earlier material on adjusted Rand index, variation of information, and consensus clustering.

Reference Groups, Resampled Groups, and Comparable Membership

Let the full-data clustering define reference subgroups \(A_1,\ldots,A_K\). In resample \(r\), let the clustering produce groups \(B_{r1},\ldots,B_{rK_r}\). The number of groups \(K_r\) need not equal \(K\). Cluster labels are arbitrary, so a label such as “cluster 2” is not itself evidence that the same subgroup has been recovered.

Membership must be compared for subjects whose labels are available in both the reference solution and the resampled solution. Let \(M_r\) be that common evaluation set for resample \(r\). For example, \(M_r\) can be the resampled subjects that have reference labels, or a fixed set of subjects assigned to each refitted clustering. When resampling creates repeated copies of a subject, the comparison should respect subject identity rather than count duplicate copies as distinct people.

Reference coverage answers, “What fraction of the evaluable members of \(A_a\) were placed in \(B_{rb}\)?” Precision answers, “What fraction of the evaluable members of \(B_{rb}\) came from \(A_a\)?” Jaccard overlap is the intersection divided by the union, so it penalizes both missed reference members and extra members in the resampled group. All three quantities range from 0 to 1, with larger values indicating greater overlap under their respective definitions.

For a given pair, write \(n_A=|A_a\cap M_r|\), \(n_B=|B_{rb}\cap M_r|\), and \(I=I_{rab}\). Since the union contains the \(n_A\) reference members and the \(n_B\) resampled members, with the \(I\) shared members counted twice in that sum, its size is \(n_A+n_B-I\). Therefore,

This identity also shows why high recall alone does not guarantee a close match: a resampled cluster can contain most of the reference subgroup and many additional subjects, reducing Jaccard overlap and precision.

Matching and Recurrence

A simple trace for each reference subgroup selects its best-overlapping resampled group in each eligible run. Eligibility requires at least one evaluable member of the reference subgroup and at least one evaluable member in a candidate resampled group. The best Jaccard match in run \(r\) is the group \(B_{rb}\) with the largest \(J_{rab}\). This is useful for summarizing a run, but it should not replace inspection of the full overlap pattern: choosing only one best match can conceal fragmentation.

A recurrence rule records how often a reference subgroup has a match that is strong enough by prespecified criteria. For illustration, suppose the criteria are Jaccard overlap at least 0.60 and reference coverage at least 0.70. If \(R_a\) is the number of eligible resamples for \(A_a\), define:

where \(\mathcal{E}_a\) is the set of eligible resamples, \(R_a=|\mathcal{E}_a|\), and \(I\{\cdot\}\) is 1 when its condition is true and 0 otherwise. The same resampled group \(b\) must satisfy both thresholds. The recurrence is an empirical proportion under the chosen resampling procedure; it is not, without further assumptions, a probability that the subgroup exists in a population.

The numerical thresholds are examples, not universal standards. Choose and report thresholds in advance, examine the overlap values as well as the resulting pass/fail count, and show how conclusions change under reasonable alternatives. When a subgroup has no evaluable members in a run, that run is not an eligible recurrence trial for that subgroup; report how many runs were eligible rather than silently treating the missing comparison as a failure.

Reading the Shape of Instability

The pattern of overlaps across a row or column of a run-specific overlap table helps distinguish different failures. A reference subgroup persists when the same reference group repeatedly has one strong match, with high Jaccard overlap and coverage, and without substantial overlap from unrelated reference groups. It fragments when members of one reference group are divided among multiple resampled groups. It merges when a resampled group contains substantial membership from two or more reference groups. It disappears when no resampled group has meaningful overlap with the reference group.

For fragmentation, inspect how the evaluable members of \(A_a\) are distributed across all \(B_{rb}\), not just the best match. For merging, inspect which reference groups contribute members to each \(B_{rb}\). A useful diagnostic is to report both the largest overlap and the next-largest overlaps, with their coverage or precision. Any rule declaring a split or merge should specify what counts as “substantial”; no single cutoff is intrinsic to the data.

Assumptions, Diagnostics, and Alternatives

These diagnostics require comparable subject identities and a defensible way to obtain labels for the common evaluation set. The clustering and preprocessing must also represent the resampling analysis of interest. Earlier material on refitting preprocessing within resamples addresses one source of leakage; here the key requirement is that the comparison not use incompatible or selectively chosen evaluation subjects. If different runs have different common sets, report their sizes and membership rules. Changes in the evaluation set can change overlap values even when the cluster assignments look similar.

The recurrence fraction is conditional on the resampling design, preprocessing, clustering method, and any rule used to assign subjects to fitted clusters. Bootstrap resamples, subsamples, and train-test perturbations probe different perturbations; their recurrence values should not be compared as if they estimated an identical quantity. The same caution applies when \(K_r\) varies across runs. A subgroup may not have a single match because the fitted solution has split or merged it, not merely because its label changed.

Useful diagnostics include the distribution of best Jaccard overlaps, coverage and precision alongside Jaccard, eligible-run counts, the number of competing matches, and the full overlap matrix for runs with low recurrence. Check that each overlap is computed from the stated common set, and that a reported high coverage is not paired with very low precision because the matched resampled group is much larger. Review whether failures cluster in particular batches or cohorts only if the resampling design and available data support that comparison; do not infer a cause from the overlap score alone.

Individual-subgroup tracking answers a different question from whole-partition measures. Adjusted Rand index and variation of information summarize agreement across pairs of complete partitions; they can be useful when no particular subgroup is the target, but they may obscure subgroup-specific failures. Co-clustering summaries aggregate repeated pairwise co-membership and can reveal stable relationships without requiring one-to-one labels, but do not by themselves label a particular reference group as persistent, split, or merged. Jaccard overlap is direct and interpretable for a named subgroup, but depends on the chosen matching and evaluation set. Using these views together is often more informative than selecting only one score.

The main limitation is that stability is not biological validity. A subgroup can recur because the data, preprocessing, or algorithm consistently impose the same structure, without that structure representing a meaningful clinical subtype. Recurrence also depends on the cutoff chosen: a result near a threshold should not be presented as categorically different from one just below it. Report the underlying values and pattern, not only a label such as “stable.”

Common Mistake