Bootstrap and Resampling Corrections for Selection

Why Resampling Must Reproduce Selection

An association estimate reported because it passed a significance threshold is not an ordinary draw from its sampling distribution: it is a draw that also satisfied a selection rule. Even when the estimate is unbiased before selection, conditioning on that rule can make reported effects too large in magnitude. A bootstrap correction aims to estimate this inflation and subtract it. Its central requirement is therefore not merely to generate plausible data, but to reproduce the process that made an estimate eligible to be reported.

The correction depends on the selection rule, the model used to generate bootstrap data, and the statistic being corrected. A threshold on a one-sided test, a threshold on an absolute test statistic, and selection of the largest signal among many variants are different procedures. They generally produce different selection biases.

In the simple summary-statistic model used below, \(\hat{\beta}\mid\beta\sim N(\beta,SE^2)\), where the standard error is treated as known. This is an approximation; more complex analyses may require a model for the original genotypes and outcomes, or for a vector of correlated association estimates. The assumptions of the chosen bootstrap model must be plausible for the data and the analysis pipeline.

A Normal-Model Derivation of Selection Inflation

Suppose selection requires a positive standardized statistic exceeding a fixed threshold \(c\): \(S=\{\hat{\beta}/SE>c\}\). This is equivalent to \(\hat{\beta}>cSE\). Under the normal model, write \(Z=(\hat{\beta}-\beta)/SE\), so \(Z\sim N(0,1)\). Selection is then \(Z>a\), where \(a=c-\beta/SE\). The conditional mean of a standard normal variable above \(a\) follows directly by integrating its density:

Here \(\phi\) and \(\Phi\) are the standard normal density and cumulative distribution function. The integral equality uses \(\phi'(z)=-z\phi(z)\), so \(\int_a^\infty z\phi(z)\,dz=\phi(a)\). Since \(\hat{\beta}=\beta+SE\,Z\), the conditional mean and its inflation are

The inflation depends on the true effect relative to its standard error, as well as on the threshold. This formula describes a simple one-sided rule. For a two-sided rule, selection is \(|\hat{\beta}/SE|>c\); for selecting the largest statistic among many variants, the relevant event also depends on the joint distribution of those statistics. A correction must use the rule that was actually applied.

Parametric Bootstrap Correction

A parametric bootstrap specifies a distribution for the data or estimate, generates bootstrap replicates under fitted parameter values, and reapplies the selection procedure to each replicate. In the summary-statistic model, a simple plug-in procedure uses the observed estimate as the generating mean. If \(B\) simulated estimates pass the rule, their average estimates the expected selected estimate under that plug-in model. Subtracting the generating mean estimates the selection bias.

In these expressions, \(\hat{\beta}^{*(b)}\) is the estimate from bootstrap replicate \(b\) that satisfies the selection rule, and \(\hat{\beta}_{\mathrm{gen}}\) is the parameter value used to generate replicates. The rule is applied before calculating the average; averaging all replicates instead would estimate an unconditional mean and would not estimate the bias among selected results. A bootstrap replicate that does not pass the rule is not included in this conditional average, though the selection rate should also be recorded.

This plug-in correction is not guaranteed to remove all bias. In particular, the observed estimate is itself selected, so using it as the generating mean can yield an imperfect estimate of the unknown true effect. Other approaches can fit the distribution conditional on selection or use an external estimate to set the generating parameter. The bootstrap remains a way to quantify the effect of a specified model and rule; it does not make the unknown parameter known.

Resampling the Analysis Pipeline and Assessing Performance

When individual-level data are available, a nonparametric bootstrap can resample individuals, usually with replacement, and rerun the association analysis. For each replicate, the analyst should repeat every step that contributes to selection: relevant filtering, model fitting, test-statistic calculation, and application of the reporting rule. If the rule selects the strongest result across variants, the resampled pipeline should select across the same set of variants rather than treating the winning variant as if it had been specified in advance.

Resampling estimates the behavior of the analysis under the empirical distribution represented by the observed sample. It is not interchangeable with a parametric bootstrap: a parametric bootstrap relies on a specified probability model, while a nonparametric bootstrap resamples the observed units. Neither automatically recreates features missing from its inputs. For example, resampling only a reported significant locus cannot reveal how the full set of tested loci competed for selection. With correlated variants, a summary-statistic simulation should also preserve the relevant dependence rather than generating independent statistics without justification.

Correction quality should be assessed for both bias and variability. Bias is the average corrected estimate minus the true effect across repeated datasets generated under a specified scenario. Variability describes the spread of corrected estimates across those datasets. A correction can reduce average inflation while increasing variability; reporting only a single corrected estimate cannot show this trade-off. In simulation, compare raw and corrected estimates across many independently generated datasets, and vary effect sizes, standard errors, thresholds, and dependence structures that are relevant to the intended use.

Diagnostics, Alternatives, and Limitations

Before interpreting a bootstrap correction, check that the analysis being simulated matches the analysis that generated the result. Verify the scale and direction of the effect, the standard error model, the significance threshold, whether the test is one- or two-sided, and whether selection was based on a single locus or a search over many loci. For resampling of individuals, confirm that the resampling unit matches the sampling design; for example, dependent observations should not be treated as independent individuals. For a parametric model, compare simulated distributions and selection rates with defensible features of the observed data.

• Few selected replicates: a small retained fraction can make the conditional mean noisy. Increase the number of generated replicates or use a simulation strategy suited to rare selection, and report how many pass the rule.
• Correction changes sharply with the generating value: this indicates sensitivity to plug-in assumptions. Evaluate plausible parameter values rather than presenting one correction as certain.
• Selection is not reproduced: a bootstrap that simulates an estimate but omits the threshold or genome-wide winner rule estimates the wrong quantity.
• Unrealistic independence: independently simulated variant statistics can misrepresent selection when variants are correlated. Model or resample the relevant joint structure.
• Unstable corrected estimates: large variation across bootstrap runs or across datasets signals that the correction may be too uncertain for precise interpretation.

Conditional likelihood under a specified selection event is a model-based alternative for estimating an effect from selected data; the earlier material on conditional likelihood develops that approach. The truncated-normal calculation above is another direct option when its normal model and threshold assumptions apply. Bootstrap methods are useful when the complete selection pipeline is easier to simulate than to handle analytically, but they inherit the assumptions and omissions of that simulation. They do not replace independent replication, and a bias-corrected discovery estimate should not be interpreted as if it were an independently observed effect.