Bootstrap Stability of Selected Covariate Effects

Why Examine Bootstrap Stability?

A covariate selection procedure can produce one final model, but that model alone does not show how sensitive the choice was to the particular subjects observed. If a small change in the sample leads to a different set of selected covariates, the final model may be a fragile summary of the data. Bootstrap stability analysis makes this sensitivity visible by repeating the entire selection procedure in resampled data sets.

The target is not simply whether a covariate was selected once. It is how often the specified procedure selects it, and how variable its estimated effect is when selected. Those are descriptive measures of the behavior of the selection procedure under resampling. They do not, by themselves, establish that an effect is clinically important or provide a conventional confidence interval.

Resample Subjects and Repeat the Whole Procedure

In an NLME data set, subject \(i\) contributes a collection of observations, potentially at several times. Write that subject’s complete record as \(D_i\), including the response measurements, observation times, and covariates. If subjects are independent sampling units, a nonparametric bootstrap replicate draws \(n\) records \(D_i\) with replacement from the \(n\) observed subject records. All observations belonging to a drawn subject travel together. A subject drawn more than once contributes multiple copies of the complete record to that replicate.

This preserves the observed within-subject pattern and dependence rather than pretending that repeated measurements are independent subjects. If the sampling design requires strata or clusters, the resampling scheme must respect that design; the simple subject bootstrap assumes that the observed subjects are the appropriate independent units.

The selection procedure must be specified before resampling. For example, it may include a candidate covariate set, a fixed entry or retention rule, model fitting choices, and a rule for comparing models. In every replicate, repeat the same procedure from its starting point. Reusing the covariates selected in the original data, or only refitting the final model, does not measure the stability of selection.

The frequency \(\hat{\pi}_j\) is the observed proportion of bootstrap replicates selecting covariate \(j\). It estimates the selection probability under the empirical resampling distribution and the chosen algorithm. It is not the probability that the covariate is truly associated with the outcome. Selection frequencies are also specific to the candidate set, selection rule, model specification, and resampling scheme.

Selection Frequencies and Their Interpretation

A high selection frequency indicates that the procedure often retains a covariate when the observed subjects are perturbed by bootstrap resampling. A low or intermediate frequency indicates that its selection depends more strongly on which subjects are represented. It is useful to report the number of successful replicates, the selection count, and the frequency, rather than only a qualitative label such as “stable.”

Bootstrap replicates are simulated from the observed data, so the frequency itself has Monte Carlo error from using a finite number \(B\). Conditional on the observed data and independent bootstrap draws, each \(I_{jb}\) is a Bernoulli indicator with success probability equal to the bootstrap selection probability. Thus the Monte Carlo standard error of \(\hat{\pi}_j\) is approximately \(\sqrt{\hat{\pi}_j(1-\hat{\pi}_j)/B}\). This quantifies numerical uncertainty in the estimated frequency, not sampling uncertainty about the underlying covariate effect.

Variability of the Estimated Effect

Selection frequency and coefficient variability answer different questions. The frequency measures how often the algorithm includes a covariate. The estimated-effect summary describes the estimates in replicates where it is included. Let \(\hat{\beta}_{jb}\) be the estimated coefficient for covariate \(j\) in replicate \(b\). If the covariate is selected in \(m_j=\sum_b I_{jb}\) replicates, the selected-estimate mean and sample standard deviation are

These summaries are conditional on the procedure selecting the covariate. They can show whether selected estimates are tightly grouped, widely spread, or sometimes opposite in sign. They are not automatically the standard error or confidence interval for the effect in a prespecified model: conditioning on selection changes which estimates are included, and selection itself is data-dependent. Report the selection frequency alongside the conditional estimates so readers can see how much of the bootstrap output those summaries represent.

Why the Bootstrap Frequency Has This Form

For a fixed observed data set, each independently generated bootstrap replicate either selects covariate \(j\) or does not. Treating those outcomes as Bernoulli draws with selection probability \(\pi_j\), the average indicator has expected value \(\pi_j\) and variance \(\pi_j(1-\pi_j)/B\). The standard deviation of this average is therefore \(\sqrt{\pi_j(1-\pi_j)/B}\), estimated by replacing \(\pi_j\) with \(\hat{\pi}_j\). This justifies the Monte Carlo standard error formula and clarifies its scope: it measures variability from the finite simulation, conditional on the original data and fixed selection algorithm.

Increasing \(B\) reduces this Monte Carlo error, but cannot repair a poorly specified covariate model, a selection rule that changes between replicates, or a bootstrap design that fails to match the independent sampling units. A frequency near 0.50 may be genuinely borderline, or its estimate may simply be imprecise because too few replicates were run. Report \(B\) and, when frequencies are central to a conclusion, assess whether increasing \(B\) materially changes them.

Diagnostics, Comparisons, and Limitations

A stability summary is only interpretable if the replicate fits and selection decisions are trustworthy. Track convergence failures, boundary estimates, and the number of successful replicates. If failures are common, the reported frequencies among successful fits may describe a selected subset of bootstrap samples rather than the intended procedure. State how failures were handled; do not silently count failed fits as covariates not selected. Investigate whether sparse categories or unusual resamples make particular candidate effects unidentifiable.

Also check that the procedure is genuinely held fixed. Changing the candidate set, model comparisons, or selection threshold across replicates makes the frequencies difficult to interpret. For categorical covariates, some replicates may contain too few observations in a level to estimate its effect; this is a substantive instability, not just a software nuisance. If the sampling design requires stratified resampling to preserve representation, state and justify that choice, since it changes the resampling distribution.

Bootstrap stability complements rather than replaces other evidence. A Wald test in a fixed full model assesses a coefficient relative to its estimated uncertainty under that specified model; it does not answer how often a selection algorithm would choose the covariate after resampling. Influence diagnostics examine sensitivity to particular influential subjects, whereas bootstrap stability summarizes sensitivity across many resampled data sets. Shrinkage-based selection and bootstrap selection frequencies also differ: one concerns the behavior of a penalized fitting rule, while the other summarizes repeated outputs of the chosen procedure.

The method has limits. It reflects the empirical distribution of the observed subjects, so it cannot reveal effects or subgroup patterns absent from the data. Correlated covariates may substitute for one another across replicates, producing low individual frequencies even when the same general information is repeatedly represented. A high frequency is not proof of causality, clinical relevance, or correct functional form. A low frequency is evidence of sensitivity for this procedure and data set, not proof that the covariate has no effect.

Common Mistake

A common error is to say, “The covariate was selected in 70% of bootstrap samples, so there is a 70% probability that its effect is real.” The calculation does not support that conclusion: 70% is the fraction of resamples in which a particular algorithm selected the covariate, conditional on the observed data and all modeling choices. It is not a posterior probability, a p-value, or an estimate of the probability that a scientific claim is true. Report it as a procedure-specific selection frequency and interpret it alongside effect estimates, model diagnostics, and subject-matter considerations.