Bootstrap Assessment Of Subgroup Stability

Why Bootstrap Subgroup Stability Needs Care

A cluster solution can look clear in the observed data and still depend strongly on which subjects happened to be measured. A subject-level bootstrap assesses this dependence by repeatedly drawing subjects with replacement, reclustering each resampled data set, and recording what happens to the discovered groups. The central bookkeeping issue is that a bootstrap replicate contains repeated copies of some subjects and omits others. Copies are not new independent people, and an omitted subject cannot be treated as if its subgroup membership had been observed in that replicate.

The earlier material on choosing a resampling unit establishes why resampling must follow the independent observational unit. Here, assume that the \(n\) subjects are the appropriate units. Each subject has one row of integrated omics features for the clustering analysis. The aim is to assess stability to subject sampling, not to compare different numbers of clusters or feature sets.

What the Bootstrap Resamples

The standard subject bootstrap assumes that subjects are independent draws from a population of interest and that the observed subjects are representative enough for resampling to approximate sampling variation. Subjects should be exchangeable for this purpose: no subject should have a different resampling probability unless the design explicitly calls for one. If several omics records belong to the same person, resample the person as a unit and keep that person’s records together.

To isolate subject-sampling variability, keep the clustering specification fixed across replicates: for example, the chosen number of clusters, distance, and feature set. Random initialization is a separate source of variation. Use a controlled initialization strategy, such as a fixed seed or multiple starts with a consistent selection rule, and report that choice. Otherwise, a change in assignments can reflect either a changed sample or a changed algorithmic start.

The bootstrap draws \(n\) times, but generally does not include \(n\) distinct subjects. For a particular subject, the probability of being omitted is the probability that none of the \(n\) draws selects that subject.

As a result, a typical replicate has about \(1-(1-1/n)^n\) of the original subjects represented. This is not a defect in the procedure; it is part of what the bootstrap resampling scheme produces. It does mean that group recurrence should report how much of the original group was available to be assessed.

Representing Membership Without Double-Counting Copies

Let the reference clustering on the original data define a subgroup \(G\), containing \(|G|\) distinct subjects. In bootstrap replicate \(b\), let \(m_{ibk}\) be the number of copies of subject \(i\) assigned to replicate cluster \(k\). For an included subject, the fraction of its copies assigned to that cluster is \(q_{ibk}=m_{ibk}/W_{ib}\). If \(W_{ib}=0\), set \(q_{ibk}=0\) for every \(k\). Thus, for an included subject, its \(q\) values across replicate clusters sum to one. This rule ensures that a subject drawn four times does not contribute four times as much to a membership summary as a subject drawn once. If duplicate copies are split across clusters, the subject’s contribution is divided across those clusters rather than assigned arbitrarily to just one.

For each reference subgroup and replicate cluster, define a reference-normalized recurrence score. It measures how much of the original subgroup is represented in that replicate cluster, counting omitted members as not represented.

For a summary of the best-matching replicate cluster, use \(S_{gb}=\max_k S_{gbk}\). This is a descriptive score, not a universal stability measure. Report it alongside coverage, the fraction of reference-group subjects that appeared at least once: \(\sum_{i\in G}I(W_{ib}>0)/|G|\). Coverage separates a weak match caused by many omitted subjects from a weak match among subjects that were actually sampled.

A complementary purity calculation asks what fraction of a candidate replicate cluster’s membership contribution came from the reference subgroup. For cluster \(k\), this is \(\sum_{i\in G}q_{ibk}/\sum_{i=1}^{n}q_{ibk}\), when the denominator is positive. A high recurrence score alone can occur when a replicate cluster includes many subjects outside \(G\); purity helps reveal that. These summaries require matching a reference group to a replicate cluster. For this tutorial, the match is the cluster with the largest recurrence score for that reference group. If there is a tie, report the tie or apply a prespecified tie rule.

Summarizing Recurrence Across Replicates

Repeat the draw-and-recluster process for \(B\) replicates. For each reference subgroup, retain the replicate-level score, coverage, and purity rather than keeping only a single average. A mean score describes typical recurrence, while a median and a distribution or range can show whether that mean hides a mixture of good and poor matches.

A recurrence frequency can also be reported if a decision rule is useful. Before examining results, specify minimum acceptable values for the recurrence score, coverage, and, if relevant, purity. In replicate \(b\), define an indicator \(I_b\) equal to one when all prespecified criteria are met and zero otherwise. The estimated recurrence frequency is the proportion of replicates meeting the rule.

This frequency is conditional on the chosen clustering specification, reference groups, matching rule, and thresholds. A threshold is a practical convention, not a natural boundary between “stable” and “unstable.” It is good practice to show the underlying score distribution as well as any threshold-based summary.

Implementation, Diagnostics, and Alternatives

A practical analysis should retain subject identifiers and multiplicities throughout resampling. For each replicate, save the draw, unique subject count, cluster assignment of every sampled copy, and the recurrence summaries. Then check that the number of drawn rows equals \(n\), multiplicities sum to \(n\), and every included subject’s membership fractions sum to one. These checks catch coding errors that otherwise distort group recurrence.

Several warning signs indicate that the result needs caution. A reference subgroup may have low coverage in many replicates, making its recurrence difficult to assess. A clustering algorithm may fail to converge or produce empty groups in some replicates. Duplicate rows may have a strong influence on a distance-based fit; for some algorithms, repeated rows act like increased weights, while for others their consequences are less simple. If the method cannot accept repeated observations sensibly, an implementation that uses multiplicity weights is appropriate only when it reproduces the intended fitting objective.

Bootstrap stability is not the same as stability to algorithmic initialization. Repeating the clustering several times on the unchanged data diagnoses sensitivity to starts, whereas bootstrapping changes which subjects are represented. These can be studied separately and, if desired, jointly with a clearly defined design. Alternatives such as subsampling without replacement or repeated train-test perturbations change which data are analyzed and answer related but not identical questions; they should not be interpreted as interchangeable bootstrap replicates.

The bootstrap also has limits. It assesses sampling variability under the empirical distribution of the observed subjects; it does not establish that a subgroup is a true biological class, validate its clinical utility, or remove bias from a nonrepresentative sample. With very small or rare groups, many replicates may omit most members, and apparent instability can partly reflect limited information. Conversely, strong recurrence under a fixed feature set and fixed preprocessing does not test whether the result survives changes to those choices.

Common Mistake