Why Examine Selection Stability?
A covariate search can produce a single final model, but that model may not be the only plausible result of the observed data. Small changes in which subjects are represented can change the evidence available to the search, particularly when individual concentration-time information is sparse or candidate covariates are correlated. A covariate retained in one fitted model may therefore disappear when the analysis is repeated on a slightly different sample.
Bootstrap stability analysis makes this sensitivity visible. It repeatedly resamples the study subjects, reruns the same covariate-selection procedure, and records how often each candidate is selected. The resulting frequencies describe the reproducibility of the selection procedure on resamples of the observed data. They do not, by themselves, establish that an effect is true, clinically important, precisely estimated, or likely to replicate in a new population.
Define the Bootstrap Procedure Before Running It
In a typical population PK study, the independent sampling units are subjects, not concentration records. A subject-level bootstrap draws subjects with replacement from the observed subjects and carries along each selected subject’s complete data: covariates, dosing information, concentration records, and relevant design information. If a subject is sampled more than once, each copy represents a bootstrap replicate of that subject. Treating records from the same subject as independent draws would disrupt the within-subject data structure and generally does not represent sampling variation across subjects.
The selection procedure must also be specified in advance. This includes the structural model, candidate covariates and their coding, any transformations or constraints, the search algorithm, selection criteria or thresholds, rules for fitting and convergence, and how ties or failed fits are handled. Each bootstrap dataset must be analyzed using the same procedure. If important choices are changed from run to run, the frequencies combine different procedures and are difficult to interpret.
Resampling may need to respect the study design. For example, if sampling was stratified by a design variable and the target is stability under that design, resampling within strata may be appropriate. That choice changes what is held fixed and therefore what variability the bootstrap represents. It should be justified by the sampling design and the intended interpretation, not selected because it produces preferred frequencies.
Define the candidates, model and search rules, resampling unit, handling of strata, and valid-fit criteria before examining the frequencies.
For each run, sample subjects with replacement and retain each selected subject’s full set of observations and covariates.
Fit the specified models and apply the same selection rules. Record which covariates are retained and whether the run was valid.
Calculate selection frequencies using valid runs, report failed runs and their handling, and examine which candidates tend to be selected together or in place of one another.
The denominator matters. If a planned run fails to produce a valid result, it should not silently be counted as a run in which no covariates were selected. Report the number of planned and valid runs, the reasons for failures, and a prespecified rule for whether to rerun or exclude a failed replicate. If failures are frequent or related to particular data configurations, the valid-run frequencies may describe only a selected subset of bootstrap datasets.
What a Selection Frequency Means
Conditional on the observed data and fixed analysis rules, independently generated bootstrap datasets can be viewed as draws from the empirical distribution of subjects. For a fixed candidate \(j\), each valid run records a binary result: selected or not selected. If the conditional chance of selection under this empirical resampling scheme is \(\pi_j\), then the indicators have mean \(\pi_j\), and their average estimates that chance. The estimated frequency is thus an empirical approximation to how often the specified algorithm selects the covariate when applied to bootstrap samples from this dataset.
A rough measure of Monte Carlo error, due only to using a finite number of bootstrap runs, is the binomial standard error:
This approximation treats the valid bootstrap runs as independent draws under the fixed empirical resampling scheme. It describes simulation noise in estimating the frequency; it is not a confidence interval for a population probability that the covariate is truly relevant. Increasing \(B\) reduces this Monte Carlo error, but it does not remove uncertainty from having observed only one study sample, nor does it correct a poorly specified search procedure.
Worked Example: Calculating Frequencies and Monte Carlo Error
Suppose a hypothetical population PK analysis plans 200 subject-level bootstrap runs. The same prespecified search is applied to each dataset. All 200 runs are valid. A renal-function covariate is selected 146 times, and a sex covariate is selected 82 times. Calculate both frequencies and their approximate Monte Carlo standard errors.
For renal function, \(\hat{\pi}_{\mathrm{renal}}=146/200=0.730\), or 73.0%. Its Monte Carlo standard error is \(\sqrt{0.730(1-0.730)/200}=\sqrt{0.0009855}=0.0314\), about 3.14 percentage points.
For sex, \(\hat{\pi}_{\mathrm{sex}}=82/200=0.410\), or 41.0%. Its Monte Carlo standard error is \(\sqrt{0.410(1-0.410)/200}=\sqrt{0.0012095}=0.0348\), about 3.48 percentage points.
Under this dataset and procedure, renal function was selected more often than sex. The difference in observed frequencies is 32.0 percentage points. These summaries do not show that renal function is a true covariate effect or that sex has no effect. They describe selection reproducibility under the specified empirical resampling procedure. The Monte Carlo errors quantify the finite number of bootstrap runs, not the full uncertainty in either covariate effect.
Unstable Selection Among Correlated Candidates
A low or intermediate frequency can arise even when the data repeatedly support some covariate information. When candidates are strongly associated, small changes in the resampled subjects can shift which candidate provides the more useful conditional contribution. The algorithm may choose one marker in some runs and another marker in other runs. Looking only at each candidate’s marginal frequency can obscure this substitution pattern.
Worked Example: Two Correlated Renal Markers
In a hypothetical analysis with 200 valid bootstrap runs, two correlated renal markers, A and B, are candidates for the same clearance model. Marker A is selected alone in 77 runs, B alone in 69, both are selected in 25, and neither is selected in 29. The counts sum to \(77+69+25+29=200\).
Marker A is selected in \(77+25=102\) runs, so its frequency is \(102/200=0.510\). Marker B is selected in \(69+25=94\) runs, so its frequency is \(94/200=0.470\). At first glance, each appears moderately unstable. Yet at least one marker is selected in \(77+69+25=171\) runs, giving an “either marker” frequency of \(171/200=0.855\). Both are selected together in only \(25/200=0.125\) of runs.
This pattern is consistent with the search often retaining one of the two markers while rarely retaining both. It does not determine which marker has the more credible biological interpretation, nor prove that either has a distinct causal effect. The observed pattern may reflect shared information, conditional selection rules, or other model features. Report the joint selection pattern alongside the individual frequencies, and examine whether the conclusion changes under defensible covariate representations.
This example illustrates why stability belongs to the complete selection process, not only to individual covariates. Useful summaries can include frequencies for prespecified groups of related candidates, the fraction of runs selecting any member of a group, and common pairs or sets of selected covariates. These summaries should be motivated by the candidate structure rather than created after searching for a reassuring pattern.
Selection Stability Is Not Parameter Precision
Selection frequency answers a discrete question: how often did the algorithm retain this covariate? Parameter precision concerns the uncertainty in the estimated size of an effect, expressed on a defined scale and under a stated estimation procedure. A covariate can be selected frequently but have a broad distribution of estimated effects. Conversely, a covariate can have a moderate selection frequency while its estimates, when selected, are relatively similar. The frequency alone does not provide a standard error, confidence interval, or clinically interpretable effect magnitude.
There is an additional complication: a coefficient may not be estimated in runs where its covariate is excluded. Summarizing only estimates from runs where it is selected gives a distribution conditional on selection. That distribution can be narrower or otherwise different from the uncertainty relevant to the full search, because selection itself depends on the data. It must not be presented as though it were an ordinary, selection-free interval for the coefficient.
Worked Example: Frequent Selection With Imprecise Effects
In a hypothetical set of 200 valid bootstrap runs, a standardized continuous covariate is selected in 180 runs, so its selection frequency is \(180/200=0.900\). In those 180 selected runs, the estimated coefficient on log clearance per one standard-deviation increase in the covariate has mean \(0.18\) and standard deviation \(0.16\).
The covariate is selected in 90.0% of these runs, but the standard deviation of the selected-run estimates is also 0.16 log-clearance units per standard-deviation increase. That spread is substantial relative to the mean estimate of 0.18. Therefore, the high selection frequency does not justify calling the coefficient precisely estimated. The coefficient summary is conditional on selection and should be labelled accordingly; it is not automatically a conventional confidence interval.
If, solely for an illustrative unconditional summary, excluded coefficients are assigned the value zero, the mean across all 200 runs would be \(0.90(0.18)+0.10(0)=0.162\). This convention mixes a selection decision with an effect estimate and is not a universal inferential recommendation. It makes clear, however, that any summary involving excluded runs requires an explicit definition of what an unestimated coefficient represents.
Diagnostics, Alternatives, and Limitations
A stability analysis is informative only if the bootstrap runs represent the intended sampling variation and the selection procedure can be applied consistently. Inspect at least the following:
- Convergence and failure rates. Tabulate failed fits, nonconvergence, boundary estimates, and rejected runs. A frequency based on a small or selectively successful subset may be misleading.
- Model and search consistency. Confirm that every valid run used the same structural model, candidate definitions, and selection rules. Unrecorded analyst changes make the frequency hard to reproduce.
- Selection patterns. Review covariates selected together or as alternatives, especially where candidates are correlated. Marginal frequencies alone may conceal substitution.
- Sampling-unit validity. Check that subjects, rather than individual records, were resampled when subjects are the independent units. Preserve design features that the intended bootstrap should condition on.
- Monte Carlo precision. If estimates vary noticeably across repeated batches of bootstrap runs, increase the number of runs. This addresses simulation noise, not limitations of the observed sample.
Bootstrap selection frequencies complement, but do not replace, other analyses. A single fitted search result gives the decision for the original dataset; resampling adds a measure of its sensitivity to subject composition. Bootstrap summaries of coefficient estimates address a different question and require careful treatment of selection. Simulation-based evaluation can assess how a search procedure behaves under explicitly generated scenarios, but its conclusions depend on those scenarios. None of these summaries makes a frequency a probability that a covariate is true.
The bootstrap also has limits. It resamples from the observed empirical distribution, so it cannot create covariate ranges, study designs, or subject types absent from the data. In a small study, some bootstrap samples may contain little information for particular parameters or may reproduce unusual subject counts. If the original data are unrepresentative, bootstrap stability does not guarantee stability in a target population. A reproducibly selected covariate can still be confounded, clinically unimportant, or selected because of model misspecification. Conversely, an unstable candidate is not necessarily irrelevant; the analysis may have limited information to distinguish it from alternatives.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A covariate is selected in 72 of 120 valid bootstrap runs. What is its selection frequency?
- Why should subject records generally be resampled as complete subject clusters rather than sampled independently?
- Two correlated candidates each have a selection frequency near 50%, but at least one is selected in 90% of runs. What does this pattern suggest, and what does it not establish?
- A coefficient has a high selection frequency but a wide distribution among runs where it is selected. Can the covariate be described as precisely estimated from the frequency alone?
Solutions. (1) The frequency is \(72/120=0.600\), or 60.0%, under the specified procedure and resampling scheme. (2) Records from one subject are dependent and form that subject’s data structure; resampling them independently can break the relationship among the subject’s covariates, doses, and concentrations and does not represent sampling subjects. (3) The search often retains at least one of the candidates, while the choice between them may vary across resamples. This is consistent with substitution or shared information, but it does not identify a true or causal marker. (4) No. Selection frequency measures how often the algorithm retains the covariate, whereas the wide selected-run coefficient distribution indicates substantial variation in estimated effects conditional on selection. These are distinct summaries.