Why the Resampling Unit Matters
A resampling analysis perturbs the observed data to ask whether discovered subgroups persist under plausible changes in the sample. That question only has a clear interpretation if the resampling unit is chosen carefully. A row in an assay file is not necessarily an independent observation: it might be one technical replicate, one visit, or one omics modality from a participant whose other records are also present.
The appropriate unit depends on the sampling design and on the population to which the stability claim is intended to apply. If the target is variation across comparable subjects, resampling subjects is usually the starting point. If subjects are sampled in related families or at shared sites, the independent units may instead be families or sites. Treating dependent records as independent can make the apparent sample much larger than the information it contains and can distort the apparent stability of a subgroup solution.
Match the Unit to the Design and Target
Start by identifying which records belong to the same biological or sampling unit. Measurements from several omics platforms on one person are paired observations, not several people. Technical replicates are repeated measurements of material from the same biological sample. Repeated visits from one person share that person’s characteristics and history. Family members may share inherited and environmental influences, while patients at one site may share recruitment and measurement conditions.
Then state the target of the stability assessment. A claim about how a subgroup solution would vary across new subjects from the same collection of sites is different from a claim about how it would vary across new sites. The first can call for subject-level perturbation that preserves site structure; the second requires information at the site level. Resampling cannot create independent sites that were never observed.
- Independent subjects: resample whole subjects, keeping that subject’s modalities and other linked records together.
- Related subjects: when dependence within families is material and families are independently sampled, resample families as units.
- Multiple sites: decide whether sites are fixed parts of the study or sampled representatives of a wider site population. The target determines whether the site itself must be perturbed.
- Repeated records: keep a person’s visits or technical replicates together when the subject is the independent unit. Do not let individual records stand in for independent people.
These choices do not imply that every study must use one universal unit. Designs can be multilevel: subjects may be grouped within families and families recruited at sites. A suitable perturbation must preserve the grouping relevant to the target. If the target is variation among subjects within the observed sites, it may be appropriate to perturb subjects while retaining site membership. If the target is variation across sites, sites are the higher-level units, and the number of independent sites constrains what can be learned.
Worked Example: Technical Replicates Are Not New Subjects
Suppose a hypothetical study has 12 subjects, each measured in three technical replicates for the same assay. The file contains \(12 \times 3 = 36\) records, but there are 12 biological subjects. Assume, for illustration, that the within-subject correlation between replicate measurements is \(\rho=0.80\). For equal cluster size \(m\), a standard design-effect approximation for an average is \(1+(m-1)\rho\), giving an effective sample size of \(n/[1+(m-1)\rho]\).
Here \(n=36\) records and \(m=3\) replicates per subject. Thus:
This approximation is for the variance of an average under the stated equal-cluster and common-correlation assumptions; it is not a direct stability score for a clustering algorithm. It nevertheless shows why 36 correlated records do not provide the information of 36 independent subjects. For a subject-level stability question, sample the 12 subjects as units and carry each selected subject’s three replicates together. Resampling the 36 replicates separately could give one subject disproportionate weight and would treat repeated measurements as if they supplied independent biological variation.
A Short Justification: Correlation Changes the Information
The design-effect approximation follows directly from the variance of an average. Suppose there are \(K\) independent clusters, each containing \(m\) observations. Each observation has variance \(\sigma^2\), and any two observations in the same cluster have correlation \(\rho\). The total number of observations is \(n=Km\). The variance of their overall mean is the variance of the sum divided by \(n^2\). Each cluster contributes \(m\) variances and \(m(m-1)\) ordered covariance terms, so:
If the \(n=Km\) observations were independent, the variance would instead be \(\sigma^2/(Km)\). The multiplier \(1+(m-1)\rho\) is therefore the variance inflation from within-cluster correlation in this simplified setting. Equating the correlated-data variance to \(\sigma^2/n_{\text{eff}}\) gives \(n_{\text{eff}}\approx n/[1+(m-1)\rho]\). The derivation explains why counting records alone can exaggerate information. It does not establish that every stability statistic has this exact variance or that this formula should be used to correct a stability score.
Worked Example: Related Subjects Sampled in Families
Consider a hypothetical study of 10 independently recruited families, with two siblings measured in each family: 20 subjects in total. Suppose the scientific target is whether subgroup patterns persist across new families. The family, not the sibling, is then the relevant independent unit if dependence between siblings is part of the data-generating structure.
For a simple illustration of the dependence penalty, take equal cluster size \(m=2\) and within-family correlation \(\rho=0.60\). The design effect is:
This is an approximate effective sample size for a mean-type quantity under the stated correlation model, not a replacement count for a subgroup stability statistic. It does show that 20 siblings do not provide the same independent information as 20 unrelated subjects. To perturb the sample for a new-family target, select families as whole units and retain both siblings whenever a family is selected. Resampling 20 siblings independently would break the family-level dependence the target requires the analysis to preserve.
If the target were instead explicitly the distribution of subjects in these particular families, subject-level perturbation might be relevant, but it would not support the same claim about new families. The target must be stated before choosing the unit.
Worked Example: Distinguishing Subject and Site Targets
Suppose a hypothetical consortium has five sites, each contributing 10 subjects, for 50 subjects total. If the question concerns new subjects recruited under the same five-site setting, resampling subjects while preserving the observed site composition may address that target. If the question concerns generalization to new sites, the site is the higher-level unit to perturb.
With only five sites, a site-level resampling scheme that draws five sites with replacement has an expected number of distinct sites equal to:
Thus, even though each resampled data set contains five site draws, it contains about 3.36 distinct sites on average under this scheme. The site-level information is limited; repeatedly reusing these five sites cannot make the evidence equivalent to observing many independent sites. A result may be sensitive to one or two influential sites, and uncertainty about generalization across sites may remain substantial.
Preserve the Whole Biological Unit
In multi-omics studies, a subject often has several linked measurements: one row per modality, several assay runs, and possibly several visits. When the subject is the sampling unit, select the subject and carry all records needed for that subject together. This preserves pairing across modalities and prevents the resampling procedure from making a biological subject appear partly present and partly absent.
Worked Example: Paired Modalities and Repeated Visits
Imagine 12 hypothetical subjects, each with three visits and three omics modalities measured at each visit. The data contain \(12 \times 3 \times 3=108\) subject-visit-modality records. For a subject-level stability question, there are 12 independent units, not 108. A resampled subject contributes all \(3 \times 3=9\) of that subject’s records; a subject not selected contributes none.
The arithmetic is \(12\) subjects \(\times 9\) linked records per subject \(=108\) records. Resampling individual records would allow, for example, a subject’s transcriptomic record at one visit to appear without that subject’s corresponding measurements in other modalities. That breaks the observed pairing and may produce a perturbation that does not represent the intended sampling process. If the scientific target is instead variability over visits within people, the design and unit need to reflect that question; visits are still not independent people.
Diagnostics, Alternatives, and Limitations
Before interpreting stability, check that the resampling unit matches the data’s identity structure. Confirm the number of unique subjects, families, and sites rather than relying on a row count. Verify that all linked records for a selected unit travel together. Check whether a small number of families or sites dominate the sample, and whether conclusions change when influential units are omitted in a sensitivity analysis. These checks diagnose the design and dependence structure; they do not by themselves prove that the units are independent.
A useful comparison is between the target, the unit, and the main risk of a mismatch:
| Target of the stability claim | Candidate unit | Risk if mismatched |
|---|---|---|
| Variation across subjects in the study setting | Subject, retaining linked modalities and records | Resampling records separately breaks within-subject pairing |
| Variation across newly sampled families | Family, retaining family members together | Subject-level resampling ignores within-family dependence |
| Variation across newly sampled sites | Site, with its sampled subjects kept together | Subject-only perturbation cannot represent between-site variation |
| Technical measurement repeatability for existing samples | Technical replicate, only if that is the stated target | Confusing repeatability of an assay with biological subgroup stability |
Alternatives are not interchangeable fixes. Resampling subjects may be suitable for a subject-level target but not for a claim about new sites. Resampling families may preserve dependence but can leave very few independent units. Resampling technical replicates can be appropriate for a question about measurement variation, but it answers a different question from stability across biological subjects. When the design has several dependence levels, the unit should preserve the levels relevant to the target, and the analysis should acknowledge how few independent units are available.
A common mistake is to argue, “There are 300 assay rows, so a resample of 300 rows represents 300 independent opportunities for the subtype solution to change.” That reasoning confuses recorded measurements with independent sampling units. If those rows come from 30 subjects with repeated visits and multiple modalities, there are at most 30 subject-level units for a subject target. Keeping rows independent may over-weight subjects with more records, break pairing, or understate uncertainty. The correction is to map records to their sampling units first, state the target population, and resample the appropriate units as wholes.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A study has 24 subjects, each measured on four omics platforms. For stability across subjects, how many units are there, and what should be kept together?
- There are 18 sibling pairs from independently recruited families. The target is stability across new families. Which unit is appropriate, and why?
- A study has 36 subjects in three sites and aims to describe stability for patients in those same sites. Does this target alone require resampling sites as independent new sites?
- In the equal-cluster design-effect approximation, calculate the effective sample size for 40 records arranged in clusters of size \(m=4\), with within-cluster correlation \(\rho=0.50\). Interpret the result cautiously.
Solutions. (1) There are 24 subject-level units, not \(24 \times 4=96\) independent units. Keep each subject’s four linked platform measurements together. (2) Resample families as whole units, because the target is new families and siblings within a family may be dependent; selecting siblings independently would not preserve that structure. (3) No. A target limited to the same three sites does not by itself make the sites sampled representatives of a wider site population. Subject-level perturbation that retains the site setting may be relevant; a new-site claim would require site-level information. (4) The design effect is \(1+(4-1)(0.50)=2.50\), so \(n_{\text{eff}}\approx40/2.50=16.00\). This is an approximate effective sample size for a mean-type quantity under the stated assumptions, not a direct correction to a subgroup-stability statistic.