Bootstrap Selection Frequencies

Why Measure Selection Stability?

A selection procedure gives one result for the data at hand: a set of predictors chosen by a specified search algorithm and selection rule. But small changes in the observed data might produce a different set. This matters because a predictor chosen once is not necessarily a robust choice. Repeating the full procedure on bootstrap samples lets us see how often each predictor is selected when the data are perturbed in a way that resembles sampling variation.

The resulting frequency describes the behavior of a particular selection procedure applied to a particular data set. It is not a new significance test, and it does not establish that a predictor is important in the population. To interpret it, keep the search algorithm, candidate predictors, stopping or model-size rule, and any constraints clear.

Definition: For predictor \(X_j\), its bootstrap selection frequency is the proportion of bootstrap samples in which the entire specified variable-selection procedure selects \(X_j\). If \(B\) bootstrap samples are generated and \(X_j\) is selected in \(K_j\) of them, its frequency is \(K_j/B\), often reported as a percentage.

How to Construct the Frequencies

For ordinary case resampling, each data row contains one observed outcome and the associated predictor values. A bootstrap sample of size \(n\) is formed by drawing \(n\) rows independently, with replacement, from the original \(n\) rows. Some original rows may appear more than once and some may not appear at all. The resampled rows stay intact: the outcome and predictors from a row are kept together.

This procedure assumes the rows can reasonably be treated as independent, identically distributed observations from the target population. It is not automatically suitable for clustered, repeated-measures, time-series, or fixed-design data; the resampling method needs to respect how those data were collected. In the examples below, the observations are assumed to be independent and identically distributed.

Most importantly, rerun the entire selection procedure in every sample. This means using the same candidate predictors, entry and removal rules or model-size criterion, hierarchy constraints, and treatment of forced or excluded predictors. If the procedure includes preprocessing or tuning choices, those choices must be repeated as part of the procedure rather than fixed using the full data. Keep a consistent rule for ties and unsuccessful fits as well.

Specify the procedure.
Record the candidate predictors, search algorithm, selection criterion, stopping rule, constraints, and any preprocessing steps.

Generate a bootstrap sample.
Draw \(n\) complete observations with replacement from the original data.

Repeat selection from the beginning.
Apply the specified procedure to that sample and record the selected set.

Count selections.
After \(B\) repetitions, divide each predictor's selection count by \(B\).

Worked Example: Calculating Frequencies for Four Predictors

Hypothetical data. A researcher has \(n=120\) independent patient records, with one continuous outcome and four candidate predictors, \(X_1,\ldots,X_4\). The researcher specifies forward selection, a partial \(F\)-test entry rule at \(\alpha=0.05\), and a stop when no remaining predictor meets the entry rule. The procedure is run on \(B=100\) bootstrap samples of 120 rows each. Suppose it selects \(X_1\) in 92 samples, \(X_2\) in 81, \(X_3\) in 47, and \(X_4\) in 18.

For \(X_1\), the frequency is \(92/100=0.92\), or 92%. The corresponding calculations for the other predictors are \(81/100=0.81\), \(47/100=0.47\), and \(18/100=0.18\).

In these bootstrap repetitions, \(X_1\) was selected more consistently than \(X_4\) under this specified search. The 92% figure does not mean there is a 92% probability that \(X_1\) is truly associated with the outcome, nor does it give a p-value. It summarizes how often this particular procedure selected \(X_1\) across resamples of these data.

What a Frequency Estimates

There is a useful way to make the interpretation precise. Imagine drawing a single sample of size \(n\) from the empirical distribution of the observed rows—that is, from the distribution that assigns probability \(1/n\) to each observed row—and then applying the full selection procedure. For predictor \(X_j\), let \(\hat{q}_j\) be the probability, conditional on the observed data, that this process selects \(X_j\). This probability is determined by the data and the specified procedure, though it is usually not calculated directly.

For bootstrap repetition \(b\), define \(I_{bj}=1\) if \(X_j\) is selected and \(I_{bj}=0\) otherwise. The reported frequency is the average of these indicators. Because independent bootstrap samples are drawn from the empirical distribution, each indicator has conditional mean \(\hat{q}_j\), and its conditional variance is \(\hat{q}_j(1-\hat{q}_j)\). Averaging \(B\) independent indicators gives the following justification for the frequency and its simulation error:

\hat{f}_j=\frac{1}{B}\sum_{b=1}^{B} I_{bj}, \qquad E(\hat{f}_j\mid\text{observed data})=\hat{q}_j, \qquad \operatorname{Var}(\hat{f}_j\mid\text{observed data}) =\frac{\hat{q}_j(1-\hat{q}_j)}{B}.

Thus, the frequency estimates the selection probability under repeated sampling from the empirical distribution. Its variability from using a finite number of bootstrap repetitions is Monte Carlo error. A common approximate standard error for that error is \(\sqrt{\hat{f}_j(1-\hat{f}_j)/B}\). This describes the precision of the numerical frequency as a Monte Carlo estimate; it is not a standard error for a regression coefficient or a confidence interval for a population parameter.

Worked Example: Quantifying Monte Carlo Error

Hypothetical result. In \(B=400\) bootstrap samples, a predictor is selected 296 times. Its frequency is \(\hat{f}=296/400=0.74\), or 74%.

An approximate Monte Carlo standard error is \[ \sqrt{\frac{0.74(1-0.74)}{400}} =\sqrt{\frac{0.1924}{400}} =\sqrt{0.000481} \approx 0.0219. \] That is about 2.2 percentage points. Using the rough normal-approximation multiplier 1.96, the Monte Carlo margin is about \(1.96(0.0219)=0.0429\), or 4.3 percentage points.

This calculation indicates the approximate simulation precision of 74% as an estimate of the selection probability under the empirical distribution. It is not a 95% confidence interval for whether the predictor belongs in a population model. If more precision in the simulated frequency is needed, increasing \(B\) reduces Monte Carlo error; multiplying \(B\) by four roughly halves its standard error.

Interpreting Competing Predictors

Frequencies are especially informative when candidate predictors are correlated. Similar predictors can carry overlapping information, so small changes in a resample may lead the procedure to choose one rather than another. Each predictor can then have a moderate frequency even when the search repeatedly chooses at least one member of the competing group.

Worked Example: Two Correlated Predictors

Hypothetical data. Suppose two candidate predictors, \(X_1\) and \(X_2\), measure related aspects of a clinical process. A specified selection procedure is rerun on \(B=300\) independent-row bootstrap samples. It selects both predictors in 43 samples, \(X_1\) alone in 123, \(X_2\) alone in 108, and neither in 26. These counts sum to \(43+123+108+26=300\).

The selection count for \(X_1\) includes the samples selecting \(X_1\) alone and those selecting both. Therefore its frequency is \((123+43)/300=166/300\approx0.553\), or 55.3%. For \(X_2\), it is \((108+43)/300=151/300\approx0.503\), or 50.3%.

The procedure selects at least one of the two predictors in \(300-26=274\) samples, giving \(274/300\approx0.913\), or 91.3%. The individual frequencies are only about 55% and 50%, while the pair as a group appears much more consistently. This pattern is compatible with the predictors competing to represent overlapping information. It does not prove that either predictor has a causal effect or that the pair is significant.

What Selection Frequencies Do—and Do Not—Say

A high frequency is evidence that a predictor is selected consistently by the chosen procedure under the empirical bootstrap perturbations. A low frequency indicates that its selection is sensitive to those perturbations. These statements are deliberately limited: they concern a procedure and a data set, not an intrinsic property of a predictor independent of the analysis.

The same predictor can have different frequencies under different search algorithms, entry rules, or model-size criteria. The candidate set also matters. A frequency does not replace the model-selection criterion, coefficient estimates, uncertainty assessment, or subject-matter reasoning. Nor should it be used as a thresholded significance test: a cutoff such as 80% has no universal interpretation as a level of statistical significance.

Selection frequencies also do not undo the bias and uncertainty that can result from searching many predictors. A predictor selected frequently may still have a biased post-selection coefficient estimate, and a predictor selected infrequently may be closely related to another predictor that is often chosen instead. Frequencies are a stability summary, not a cure for selection effects.

Key takeaway: The bootstrap selection frequency for a predictor is the fraction of bootstrap samples in which the whole specified selection procedure chooses it. Interpret it as a measure of stability conditional on the data, candidate set, and procedure—not as a p-value, a probability that the null hypothesis is false, or proof of importance.

Common Mistakes

Resampling only after selection. Holding the original selected model fixed does not measure how the search would change. Resample the observations and repeat selection from the start.
Calling frequency significance. “Selected in 90% of bootstrap samples” does not mean “significant at the 10% level.” A selection frequency and a p-value answer different questions.
Ignoring competing predictors. Several correlated predictors can divide selections among themselves. Consider whether selection of at least one member of a scientifically meaningful group is more informative than any single frequency.
Changing the procedure across repetitions. Changing entry rules or candidate variables makes the frequencies difficult to interpret. Keep the procedure fixed and state it clearly.
Dropping failed repetitions without explanation. The denominator should be clear. Investigate fitting failures and specify a consistent handling rule rather than silently omitting samples.

Check Your Understanding

Use the definitions and examples above to answer the following.

A predictor is selected in 63 of 150 bootstrap samples. What is its selection frequency?
In one sentence, explain why selection frequency is not a p-value.
Why must the entire specified selection procedure be rerun in every bootstrap sample?
Two correlated predictors each have a moderate selection frequency, but at least one is selected in nearly every sample. What might this pattern suggest, and what does it not establish?
If the number of bootstrap repetitions increases while the observed data and procedure stay fixed, what happens to Monte Carlo error, and what kind of uncertainty does this address?