Why the Resampling Scheme Changes the Question
A feature-selection frequency summarizes how often a feature is selected when the data are perturbed and the model is refitted. In high-dimensional genomic analyses, that summary depends not only on the selection algorithm, but also on how the perturbation is created. A bootstrap sample has the original sample size but includes repeated observations; a subsample contains fewer distinct observations and no repeats within a replicate. Their frequencies therefore describe different forms of sensitivity.
The comparison is meaningful only after the fitting procedure is specified. That procedure may include preprocessing, penalty selection, and a rule for declaring a feature selected, such as a nonzero penalized coefficient. If any of those steps vary across resamples, they are part of the procedure whose stability is being summarized. For instance, repeating cross-validation within each resample measures the stability of the combined tuning-and-fitting procedure, while holding the penalty fixed measures stability conditional on that penalty choice.
What Bootstrap and Subsampling Replicates Contain
In an ordinary nonparametric bootstrap, each replicate draws \(n\) sampling units independently with replacement from the \(n\) observed units. Equivalently, it assigns each observed unit a random multiplicity \(W_i\), where the multiplicities sum to \(n\). A unit drawn several times contributes several times to the fit; a unit not drawn contributes nothing in that replicate. The repeated copies are not new people, samples, or independent measurements. They are weights placed on the original observations as a way to perturb the empirical data distribution.
In subsampling without replacement, a replicate chooses \(m\) distinct units from the \(n\) observed units, usually with \(m<n\). No selected unit is duplicated within that replicate. Across independently generated replicates, however, the same original unit can appear in many different replicates. The chosen size \(m\) is part of the definition of the stability analysis: smaller \(m\) generally produces a more substantial data perturbation.
Both schemes fit the same feature-selection procedure to altered versions of the observed data. But the perturbations are not interchangeable. The bootstrap retains nominal size \(n\) while changing unit weights; subsampling changes the number of distinct units used in each fit. Consequently, bootstrap frequencies answer a question about sensitivity to bootstrap reweighting, whereas subsampling frequencies answer a question about sensitivity to repeatedly using smaller, distinct subsets.
Worked Example: Reading a Bootstrap Sample as Weights
Suppose a hypothetical dataset has five sampling units, labelled A through E. One bootstrap replicate of size five draws A, A, B, C, E. Its multiplicities are \(W_A=2\), \(W_B=1\), \(W_C=1\), \(W_D=0\), and \(W_E=1\). There are four distinct observed units in this replicate, even though its nominal size is five.
For a penalized least-squares fit based on mean squared error, the data-fit term for this replicate can be written as $$ \frac{1}{5}\sum_{i=1}^{5} W_i (y_i-x_i^\mathsf{T}\beta)^2. $$ With the stated multiplicities, this is $$ \frac{1}{5}\left\{2(y_A-x_A^\mathsf{T}\beta)^2+(y_B-x_B^\mathsf{T}\beta)^2+(y_C-x_C^\mathsf{T}\beta)^2+(y_E-x_E^\mathsf{T}\beta)^2\right\}. $$ Unit A contributes twice to the loss, while unit D contributes nothing. The two appearances of A do not add a second independent subject or a second independent genomic profile; they alter the weight of A in this particular fit.
If the feature-selection procedure chooses feature \(j\) in this replicate, it contributes \(1\) to that feature's bootstrap selection count. If it does not, it contributes \(0\). Repeating this process estimates how often the specified fitting procedure selects \(j\) under bootstrap reweighting of these five observed units.
For a standard bootstrap of size \(n\), the probability that a particular observed unit is absent is \((1-1/n)^n\). Thus its probability of appearing at least once is \(1-(1-1/n)^n\), which approaches about \(0.632\) as \(n\) grows. The expected number of distinct observed units in the replicate is therefore $$ E(U)=n\left[1-\left(1-\frac{1}{n}\right)^n\right]. $$ This result explains why a bootstrap sample of size \(n\) typically contains substantially fewer than \(n\) distinct units. It does not mean that the bootstrap is simply a subsample of that expected size: bootstrap multiplicities can exceed one, whereas a without-replacement subsample gives every included unit multiplicity one.
Computing and Comparing Frequencies
Worked Example: Two Schemes Give Different Frequencies
Consider a hypothetical analysis with three features, G1, G2, and G3, and \(B=10\) fitted replicates under each scheme. Suppose the numbers of selections are as follows.
| Feature | Bootstrap selections | Bootstrap frequency | Subsample selections | Subsample frequency |
|---|---|---|---|---|
| G1 | 7 of 10 | 0.70 | 5 of 10 | 0.50 |
| G2 | 4 of 10 | 0.40 | 7 of 10 | 0.70 |
| G3 | 1 of 10 | 0.10 | 2 of 10 | 0.20 |
For G1, the bootstrap estimate is \(7/10=0.70\), and the subsampling estimate is \(5/10=0.50\). For G2 the corresponding estimates are \(4/10=0.40\) and \(7/10=0.70\). These arithmetic summaries show that the procedure selected G1 more often under the bootstrap perturbations and G2 more often under the chosen subsampling perturbations.
The difference does not establish that G1 is more biologically important under the bootstrap, or that G2 is more important under subsampling. The frequencies are conditional on different perturbation schemes. To interpret a difference, one would check that both analyses used the same observed units, selection definition, preprocessing policy, and tuning rule, and would report the subsample size \(m\). Even with those elements aligned, different frequencies can be a genuine consequence of the schemes perturbing the data differently.
A fair comparison also requires attention to how the penalty is handled. If a penalty parameter is fixed at the same numerical value in all fits, a smaller subsample can have a different effective degree of penalization depending on how the loss is scaled. If the penalty is retuned within each replicate, tuning variability becomes part of the comparison. Apply the chosen sample-size scaling convention consistently, and state whether tuning is fixed or repeated; a raw numerical penalty value does not by itself guarantee comparable fits.
Worked Example: Separating Monte Carlo Error From the Frequency
Suppose 100 independently generated bootstrap replicates select a hypothetical feature in 60 fits. Its estimated bootstrap selection frequency is $$ \hat{\pi}_j=\frac{60}{100}=0.60. $$ Conditional on the observed data and the specified procedure, the replicate indicators are independent Bernoulli variables with a common selection probability under the bootstrap scheme. The estimated Monte Carlo standard error of the frequency is $$ SE_{\mathrm{MC}}=\sqrt{\frac{\hat{\pi}_j(1-\hat{\pi}_j)}{B}} =\sqrt{\frac{0.60(0.40)}{100}} =\sqrt{0.0024} \approx 0.0490. $$
This standard error describes the simulation noise from using a finite number of bootstrap replicates to estimate the bootstrap frequency. It does not include uncertainty from having observed only one dataset, and it is not a standard error for a biological effect or a population-level inclusion probability. Increasing \(B\) reduces Monte Carlo error, but cannot remove the dependence of the frequency on the observed data, resampling scheme, or fitting procedure.
For comparison, if 60 of 100 independently drawn subsamples select the feature, the estimated subsampling frequency is also \(0.60\), and the same plug-in Monte Carlo standard error is about \(0.0490\). Equal numerical estimates do not make the two quantities identical: they refer to different resampling mechanisms and, for subsampling, require a stated \(m\).
Assumptions, Diagnostics, and Failure Modes
The calculations above presume that the sampling units are the units that can reasonably be resampled independently. In a genomic study, these are often individuals, not genes, probes, or repeated measurements from the same individual. If units are clustered or related, independently resampling individual rows can break the dependence structure. The resampling design must respect the way units were sampled; otherwise the frequencies describe perturbations that do not correspond to the data-generating design.
The resampling scheme also assumes that the observed units are suitable representatives for the empirical distribution being perturbed. Neither bootstrap nor subsampling repairs biased recruitment, unmeasured confounding, batch effects, or a mismatch between the study sample and the intended target population. A high frequency is evidence of stability under the specified perturbations, not a cure for design problems.
In practice, diagnose the analysis by checking the following:
- Verify the replicate construction. Confirm that bootstrap replicates contain \(n\) draws with replacement and that each subsample contains exactly \(m\) distinct units. Inspect multiplicity summaries for bootstrap fits.
- Check the whole pipeline. Any data-dependent preprocessing, such as scaling or filtering, should be handled according to a prespecified rule and, when appropriate, repeated within each replicate. Reusing transformations estimated from all observations can make resampled fits depend on information outside their replicate.
- Track fit failures and selection sizes. Record convergence failures, the number of selected features, and any changes in the tuning rule. A frequency based on silently discarded difficult replicates can be misleading.
- Increase \(B\) when Monte Carlo noise is material. Large changes in the frequency after adding replicates indicate that the initial estimate was too noisy for the intended interpretation.
- Report the scheme and its settings. State bootstrap or subsampling, \(n\) or \(m\), the number of replicates, the selection rule, and how tuning and preprocessing were handled.
A further limitation is that selection is a discontinuous operation: a small change in a fitted coefficient can move it from zero to nonzero and change the selection indicator. Frequencies summarize that instability but do not identify its cause. They can also be affected by correlated predictors, for which alternative features may be selected in different replicates. The details of that issue require a separate analysis; a feature-level frequency alone should not be read as a definitive ranking of biological importance.
Choosing the Perturbation for the Stability Question
Neither scheme is universally preferable. Use bootstrap resampling when the desired diagnostic is sensitivity to random reweighting of the observed units while keeping the nominal replicate size at \(n\). Use subsampling when the question concerns selection after fitting to smaller sets of distinct units, and choose \(m\) to represent the perturbation of interest. Because \(m\) changes the amount of information per fit, it must be reported and considered when interpreting the resulting frequencies.
| Choice | What changes within a replicate? | Interpretation of the frequency | Main caution |
|---|---|---|---|
| Bootstrap | Unit weights change through sampling with replacement; some units repeat and some are absent. | Stability under empirical-distribution reweighting at nominal size \(n\). | Repeated copies are not new independent observations; multiplicities can strongly weight individual units. |
| Subsampling | A fixed number \(m\) of distinct units is used without replacement. | Stability across smaller, distinct subsets of the observed data. | Results depend on \(m\) and on how the fitting procedure handles sample size and penalty scaling. |
For either approach, a frequency is a descriptive measure of reproducibility under a defined perturbation. It is not automatically a probability of a true association, a false-selection rate, or a guarantee that the selected features will generalize to new data. Those claims require additional assumptions and procedures beyond calculating resampling frequencies.
Common Mistakes
- “A bootstrap replicate has \(n\) independent observations because it has \(n\) rows.” The correction is that the rows are draws from only \(n\) original units; duplicates repeat original information and change its weight.
- “Bootstrap and subsampling frequencies estimate the same quantity.” They estimate frequencies under different perturbations. A direct comparison must name the scheme and the subsample size.
- “A feature selected 80% of the time has an 80% probability of being truly associated.” The frequency is conditional on the data, algorithm, and perturbation mechanism. It does not assign a probability to biological truth.
- “The same numerical penalty gives comparable selection across all sample sizes.” Loss scaling and tuning choices can change the effective regularization. State and apply the sample-size convention consistently.
- “More replicates fix an unstable analysis.” More replicates reduce Monte Carlo error in the estimated frequency; they do not eliminate instability or bias caused by the data or modeling pipeline.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A bootstrap replicate draws \(n\) units with replacement. Why can it contain fewer than \(n\) distinct units, and what do duplicate rows mean?
- A feature is selected in 36 of 60 subsamples. Calculate its selection frequency and state two details needed to interpret it.
- Two analyses report a frequency of 0.65, one from bootstrap samples and one from subsamples. Are they estimating the same resampling quantity? Explain.
- What does the Monte Carlo standard error of a resampling frequency measure, and what uncertainty does it not measure?
Solutions. (1) Sampling with replacement can draw the same original unit more than once and omit others. A duplicate contributes additional weight from that original observation, not a new independent unit. (2) The frequency is \(36/60=0.60\). Interpretation requires, at minimum, the resampling scheme and subsample size \(m\); the selection rule and tuning policy should also be stated. (3) No. The numerical values match, but bootstrap reweighting and subsampling without replacement define different perturbations. (4) It measures finite-replicate simulation noise in estimating the frequency conditional on the observed data and procedure. It does not measure uncertainty from observing only one dataset or uncertainty about biological truth.