Why Resampling Alone Does Not Undo Selection
A bootstrap creates new data sets by resampling observed data. That can help quantify sampling variability, but it does not automatically remove the dependence created when the same expression measurements were used both to discover clusters and to test genes as markers. If the bootstrap keeps the discovered partition fixed, it repeats the marker calculation conditional on labels that were themselves chosen using the data. The selection step has not disappeared; it has been frozen into the analysis.
The distinction is between resampling a statistic after discovery and resampling the whole procedure that produced the statistic. The first can describe variability around the selected result while retaining its selection bias. The second can reflect selection variability, but is valid for a particular inferential target only if its resampling distribution represents the relevant sampling process or null hypothesis.
“Fixed” describes the resampling procedure; it does not mean the labels were fixed independently of the data in the original analysis. That distinction is central. A fixed-partition bootstrap may be appropriate for a narrowly defined conditional description, but it does not by itself make an ordinary marker-test p-value valid after data-adaptive clustering.
A Short Derivation of Selection Bias
Consider two candidate marker statistics \(Z_1\) and \(Z_2\). For illustration, assume they are independent standard normal variables under a global null: neither candidate has a true effect, and each statistic has mean zero and variance one. If the reported marker is the one with the larger statistic, the selected value is \(\max(Z_1,Z_2)\), not a statistic chosen in advance.
The identity \(\max(a,b)=(a+b+|a-b|)/2\) gives a direct calculation. Since \(Z_1+Z_2\) has expectation zero, while \(Z_1-Z_2\sim N(0,2)\), the expected absolute difference is \(2/\sqrt{\pi}\). Therefore:
Thus, even though both candidate statistics have null expectation zero, the selected statistic has positive expectation. The precise amount of selection bias depends on the number and dependence of candidates and on the selection rule; the calculation is a simple demonstration, not a universal correction factor.
Now suppose a bootstrap fixes the selected gene and the selected partition. Its replicates fluctuate around the observed selected statistic. They do not recreate the unselected alternatives that could have won, so they do not reproduce the comparison that generated the winner. A full-pipeline procedure can include that comparison, but its replicates must also represent the right null or sampling distribution.
Worked Example: A Fixed-Partition Bootstrap Retains a Selected Contrast
Suppose a hypothetical analysis tests two candidate genes on the same two clusters and selects the larger observed difference in mean expression. For gene \(g_1\), the four observed values are \(1,2,3,4\) in cluster A and \(2,3,5,7\) in cluster B. For gene \(g_2\), the observed difference is \(1.2\) expression units. The observed difference for \(g_1\), using B minus A, is:
Because \(1.75>1.2\), the procedure selects \(g_1\). Now hold the observed cluster labels fixed and resample four values with replacement separately within each cluster. The expected bootstrap mean within cluster A is \(2.50\), and the expected bootstrap mean within cluster B is \(4.25\): resampling from each observed group preserves that group’s empirical mean in expectation. Consequently:
For example, one possible bootstrap draw is A: \(1,1,3,4\), with mean \(2.25\), and B: \(2,3,5,7\), with mean \(4.25\). Its contrast is \(4.25-2.25=2.00\). Such variation describes resampling around the observed groups; it does not make \(g_1\) an independently chosen gene. In particular, a standard test that treats \(g_1\) and the partition as prespecified has not accounted for the rule that chose the larger of the two observed differences.
This example isolates the issue by resampling within the fixed groups. Other fixed-partition bootstrap designs may resample cells or samples differently, but if they still hold the data-derived partition and selected feature fixed, they still do not rerun the selection that created the marker claim.
What Rerunning the Pipeline Changes
A full-pipeline resampling replicate starts with a resampled or simulated expression data set, then repeats the specified operations: preprocessing, feature selection, clustering, and marker-statistic calculation. If gene selection is part of the original rule, the replicate must repeat it too. This lets the statistic vary because the clusters and selected marker may vary. It is more faithful to a data-adaptive analysis than freezing the observed partition.
But “rerun everything” is not a complete validity argument. A nonparametric bootstrap sampled from the observed data generally approximates repeated sampling from an empirical distribution that retains the observed signal and associations. It can be useful for assessing variability or selection behavior under that fitted distribution. It is not automatically a null distribution for testing “no marker effect.” A null test needs replicates generated under the null being tested, while preserving the relevant dependence and design features.
Worked Example: Full-Pipeline Null Calibration
Suppose a hypothetical study defines its test statistic as the largest absolute standardized marker contrast found after clustering: \(T_{\mathrm{obs}}=\max_g |Z_g|=2.40\). The analysts specify a null-generating model in which the tested genes have no cluster-associated effects, while retaining the modeled library-size variation and gene-to-gene dependence. They simulate 999 data sets under that null and rerun the complete discovery and marker-selection procedure on each one. In 23 replicates, the resulting maximum statistic is at least \(2.40\).
With the common plus-one correction, the estimated Monte Carlo p-value is:
Under the stated null-generating model and assuming the simulated pipeline matches the analysis, this estimates the probability under that null of obtaining a maximum statistic at least as large as \(2.40\). It does not mean there is a 2.4% probability that the null is true. The plus-one calculation avoids reporting a zero p-value merely because none of a finite number of simulations exceeded the observed statistic.
The result depends on the null model, not just on rerunning the algorithm. If the simulated data omit important dependence, batch structure, or other features that affect clustering or marker statistics, the estimated tail probability may be miscalibrated. Likewise, if the observed analysis selected a cluster count or filtered genes but the simulations do not repeat those steps, the simulated statistic does not represent the full selection rule.
Selection Frequencies Are Not Marker P-Values
Worked Example: Interpreting Full-Pipeline Bootstrap Selection
In a separate hypothetical analysis, analysts resample the observed data 200 times, rerun clustering and marker selection each time, and record which gene is selected as the leading marker. Gene \(g_1\) is selected in 72 replicates. Its selection frequency is:
The defensible description is that \(g_1\) was selected in 36% of these bootstrap replicates under this particular resampling scheme. This can indicate that the selected feature is sensitive to resampling. It is not a 0.36 p-value, a probability that \(g_1\) is a true marker, or a guarantee of 36% confidence in its biological interpretation. The replicates were drawn from the observed data, not necessarily from a null model.
This calculation also requires a consistent definition of “the same” marker across replicates. If clusters change, labels such as “cluster 1” may refer to different groups in different replicates. A comparison should use a prespecified, meaningful matching rule or a label-invariant statistic; otherwise, the recorded frequency may not represent a coherent event.
Assumptions, Diagnostics, and Alternatives
Before using resampling for inference, specify the target: uncertainty in a selected contrast, calibration of a test under a null, or sensitivity of the discovery result to perturbing the data. Then specify the unit that is independently sampled. In single-cell studies, cells from the same donor are not generally independent biological replicates. A cell-level bootstrap may describe variation among sampled cells, but it does not automatically support population-level claims about donors. When donors are the independent units for the target, the resampling design must respect that structure.
For a full-pipeline null bootstrap, the key conditions are that the null data-generating mechanism is appropriate for the hypothesis and design; the independent sampling units are resampled or simulated appropriately; and the complete, prespecified analysis rule is repeated. The statistic must be defined in a comparable way across replicates. These conditions are substantive assumptions, not technical details that can be repaired by increasing the number of replicates.
Useful checks include comparing simulated and observed distributions of relevant summaries, confirming that every selection step is rerun, and checking whether the chosen statistic is defined in replicates where clustering changes or a candidate cluster is absent. Monte Carlo precision should also be assessed: with \(B\) independent simulations, the approximate Monte Carlo standard error of an estimated tail proportion \(\hat{p}\) is \(\sqrt{\hat{p}(1-\hat{p})/B}\). A small simulation count can make a tail estimate noisy even when the model is sound.
The main alternatives answer different problems. Sample splitting separates discovery from testing by reserving data for inference, at a cost in power and with a need to assign held-out cells appropriately. Selective inference targets inference conditional on the selection event, but requires a tractable account of that event and a suitable model. Permutation tests can provide null calibration when the exchangeability conditions for the relevant units and labels hold; their validity is not established by merely permuting something in the data. Independent validation assesses reproducibility in new data, provided the validation analysis is specified without reusing the discovery result to choose the test.
No bootstrap method guarantees unbiased effect estimates, calibrated p-values, or valid confidence intervals merely because it produces many replicates. Selection can make the distribution irregular, particularly when small data changes alter clusters or the winning marker. A full-pipeline bootstrap is an improvement over freezing selection when the goal is to represent the pipeline’s variability, but inferential validity still depends on the resampling target, assumptions, and statistic.
Common Mistakes
- Wrong reasoning: “I bootstrapped the cells, so double dipping is removed.” Correction: If labels and selected genes stay fixed, the selection rule is still conditioned upon rather than repeated.
- Wrong reasoning: “I reran clustering, so the bootstrap p-value is valid.” Correction: Rerunning the pipeline addresses one source of omission; the resampling distribution must still represent the null hypothesis and sampling design.
- Wrong reasoning: “A gene selected in many bootstrap replicates has a small p-value.” Correction: Selection frequency measures recurrence under a specified resampling scheme, not a null tail probability.
- Wrong reasoning: “Thousands of bootstrap replicates compensate for a misspecified model.” Correction: More replicates reduce Monte Carlo error; they do not correct a resampling scheme that targets the wrong distribution.
Check Your Understanding
For each question, decide what the resampling result can support; solutions follow each prompt.
- A fixed-label bootstrap gives a mean contrast of 1.4 expression units across replicates, close to the observed contrast. Does this show that the marker test is calibrated after clustering?
Solution: No. It shows that the bootstrap distribution is centered near the observed contrast under that fixed-partition scheme. It does not account for the rule that selected the partition or marker. - A full-pipeline bootstrap resamples observed data and selects a gene in 80% of replicates. Is 0.20 a p-value for the gene’s null hypothesis?
Solution: No. The 80% is a selection frequency under the chosen resampling distribution. A p-value requires a distribution generated under the null hypothesis. - Under a specified null simulation, 9 of 499 replicates have a statistic at least as large as observed. Calculate the plus-one estimate.
Solution: \(\hat{p}=(1+9)/(1+499)=10/500=0.020\). Its interpretation depends on the null model and full-pipeline assumptions being appropriate. - Why can cell-level resampling be unsuitable for a population-level claim when cells come from multiple donors?
Solution: Cells within a donor are not generally independent biological replicates. Treating them as independent can misrepresent the sampling variation across donors; the resampling units must match the target population and design.