What a Bootstrap Distribution Represents
A bootstrap analysis can produce a long list of fitted parameter values, standard errors, and confidence intervals. Those outputs are easy to overinterpret. The key is to remember what varies: each replicate is a resampled or simulated version of the original dataset, analyzed using a specified procedure. Replicates are not newly recruited participants, and their spread does not by itself show that the model describes the data well.
For a nonparametric subject-level bootstrap, the procedure resamples the independent sampling units, usually subjects in a population PK study. If subject \(i\) contributes multiple concentration measurements, those measurements generally remain together as part of that subject’s resampled record. Other designs may require resampling clusters or using strata, as appropriate to how units were sampled. A parametric bootstrap instead simulates data under the fitted model and its assumed sampling structure. In either case, the interpretation depends on the actual procedure; “bootstrap” alone does not specify what population of variation has been represented.
If \(B\) replicates produce estimates \(\hat{\theta}^{*(1)},\ldots,\hat{\theta}^{*(B)}\), their sample standard deviation is commonly used as a bootstrap estimate of the standard error:
This summarizes the spread of the estimates produced by the specified bootstrap procedure. It is not the spread of independent people newly sampled in addition to the original study. Increasing \(B\) can reduce Monte Carlo noise in the numerical summary, but does not increase the original study’s sample size or remove limitations of its design.
Three Interpretation Errors to Avoid
1. Treating replicates as independent subjects
Bootstrap replicates reuse information from the observed dataset. They are alternative resampled datasets for assessing uncertainty, not extra observations to append to the original data. In a repeated-measures study, measurements from the same subject are usually dependent and share subject-level features. Resampling individual concentration records as if they came from unrelated subjects can break this structure and produce an uncertainty estimate for the wrong sampling process.
Worked Example: Resampling Subjects Rather Than Concentration Rows
Hypothetical study. Suppose 60 subjects each contribute three concentration measurements, giving 180 records. The target is the bootstrap uncertainty of estimated typical clearance, \(CL\). The study sampled subjects, and each subject’s measurements form a repeated profile.
For a subject-level nonparametric bootstrap, each replicate draws 60 subjects with replacement and includes all three measurements for each selected subject. If a subject is drawn twice, that subject’s whole profile appears twice in the replicate. This retains the subject as the sampling unit and preserves the within-subject grouping.
An analyst instead resamples 180 individual concentration rows with replacement. That procedure can place one measurement from a subject in a replicate without the other two, and can treat measurements from one subject as though they came from different independent subjects. It no longer represents the stated subject-sampling design.
Assume, purely for illustration, that the subject-level analysis gives a bootstrap standard error of \(0.42\ \text{L/h}\), while the row-level analysis gives \(0.25\ \text{L/h}\). The first value describes variability under subject resampling; the second describes variability under the inappropriate row-resampling scheme. The difference is not guaranteed to have this direction in every dataset. The correction is to choose the resampling unit from the design, not to pick whichever standard error seems more plausible.
This error can also occur in clustered or stratified designs. Resampling subjects independently when they were sampled within clinics, or ignoring prespecified strata, may fail to preserve design features that matter for uncertainty. The bootstrap scheme should match the independent sampling units and any relevant design structure.
2. Reporting only successful fits without the failures
A failed replicate is not simply an inconvenient estimate to omit without comment. If some replicates fail to converge, hit problematic boundaries, or produce unusable results, the successful estimates describe the subset of replicates that did fit. That subset may differ systematically from all attempted replicates. For example, difficult datasets may be more likely to fail precisely when the data provide weak information about a parameter. A summary of successful fits alone can then conceal an important property of the procedure.
The earlier material on bootstrap convergence and failed replicates distinguishes convergence status from the estimated parameter value. A failed fit does not automatically prove that the model is wrong, and it does not make an invalid estimate valid. Report how many replicates were attempted, how many succeeded, how failures were classified, and whether the reported summary is conditional on successful fitting. Investigate failure patterns rather than silently deleting them or substituting arbitrary values.
Worked Example: Quantifying Fit Attrition
Hypothetical demonstration. In 20 bootstrap attempts for clearance, 16 fits converge and four do not. The 16 valid clearance estimates, in \(\text{L/h}\), are \(8, 9, 9, 10, 10, 11, 11, 12\), with each value occurring twice.
The mean among converged fits is
The failure proportion is \(4/20=0.20\), or 20%. Thus, \(10\ \text{L/h}\) is the mean among the 16 successful fits in this demonstration; it is not a summary of 20 valid clearance estimates. The four failed fits have no valid estimates to include in that mean.
A careful report would state the number attempted and the failure proportion alongside any successful-fit summary, and would describe how convergence was defined and failures investigated. Twenty attempts are far too few to support a stable confidence interval; these numbers illustrate the interpretation, not a recommended replicate count. With substantial failures, a percentile interval based only on successful estimates is conditional on convergence and may not represent the intended bootstrap distribution.
The practical response is not to force a single universal rule for every failure. Check whether failures share a cause, whether the fitting algorithm or initial values are at fault, and whether failures cluster in particular data configurations or parameter regions. If estimates are reported after excluding failures, label the resulting analysis as such and discuss how failure could affect its interpretation.
3. Treating bootstrap spread as evidence of model validity
A narrow bootstrap distribution can indicate that estimates are relatively stable under the chosen resampling scheme. It cannot establish that the structural model, residual error model, covariate relationships, or assumptions about the sampling process are adequate. A systematically misspecified model can yield similar parameter estimates across many resamples of the same study. Resampling reproduces the information and limitations of the source data; it does not automatically add information that would reveal every form of misspecification.
Worked Example: Stable Clearance Estimates and a Prediction Discrepancy
Hypothetical model check. A subject-level bootstrap with \(B=1{,}000\) successful refits gives an estimated clearance of \(5.0\ \text{L/h}\) and a bootstrap standard error of \(0.16\ \text{L/h}\). Separately, at a late sampling time, the model predicts a typical concentration of \(8\ \text{mg/L}\), while the corresponding observed typical concentration is \(10\ \text{mg/L}\).
The relative prediction discrepancy, using the observed concentration as the denominator, is
The bootstrap results suggest that this procedure gives a clearance estimate with a reported standard error of \(0.16\ \text{L/h}\) under its resampling scheme. The separate comparison indicates 20% underprediction at the stated time. A small standard error does not cancel that discrepancy or establish adequate predictions. The pattern should be assessed with appropriate model diagnostics and knowledge of the data; the bootstrap spread answers a question about resampled estimate variability, not a general question about model validity.
Interpretation, Reporting, and Limits
A bootstrap standard error is a measure of estimated sampling variability under the specified procedure. A bootstrap confidence interval is constructed from the bootstrap distribution by a stated method, such as the percentile, basic, or BCa method. The interpretation of a confidence interval is about the long-run coverage of the procedure under its assumptions and repeated use; it is not a probability statement that the fixed parameter lies in this particular computed interval. Nor does an interval’s width, by itself, assess whether the model is scientifically adequate.
For a useful report, make the chain of interpretation visible:
- State the resampling or simulation scheme and the unit resampled, including any preserved strata or clusters.
- State whether each replicate refits the model or holds specified parameters fixed, since those procedures target different sources of variation.
- Give the number of attempted and successful replicates, the failure rate, and how failures were handled.
- Name the summary being reported, its units, and—when an interval is given—the interval construction method.
- Describe the result as uncertainty under the stated bootstrap procedure, not as proof of model validity or as additional independent study data.
A parametric bootstrap depends on the fitted model’s simulation assumptions; a nonparametric bootstrap depends on the observed units and the adequacy of the resampling design. Both can be uninformative about features absent or poorly represented in the original study. With few independent subjects or clusters, resampling cannot manufacture broad support for the population. With substantial fitting failures, a successful-fit-only distribution can be especially difficult to interpret. These are limitations to report and investigate, not details that a large value of \(B\) can fix.
A Short Justification for the Replicate Count Distinction
The standard deviation across bootstrap estimates is computed from \(B\) replicate values, but those values are generated from the same observed dataset or fitted model. The formula uses \(B\) to summarize the empirical distribution; it does not turn each replicate into another independently sampled participant. For example, if a study has \(n=60\) subjects and \(B=1{,}000\) bootstrap replicates, the study still contains 60 observed subjects. The 1,000 replicates provide a numerical approximation to the procedure’s bootstrap distribution.
There is also Monte Carlo variation from using a finite number of replicates. For the bootstrap mean, the Monte Carlo standard error is approximately \(SE_{\text{boot}}/\sqrt{B}\), conditional on the bootstrap distribution being sampled as intended. This describes numerical error in approximating that mean; it is not the sampling standard error of the original study estimate. More replicates can improve the numerical approximation, but cannot correct an inappropriate resampling unit, remove systematic model errors, or replace missing study information.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A study samples 45 subjects, each with five concentration measurements. What is the subject count after \(B=800\) subject-level bootstrap replicates?
- In 200 attempted bootstrap fits, 30 fail. What failure proportion should be reported, and what does a summary of the successful fits describe?
- A bootstrap distribution for a parameter is narrow. What does that suggest, and what does it not establish?
- Why can resampling individual concentration rows be inappropriate when subjects contribute repeated measurements?
Solutions. (1) The original study still has 45 subjects; \(B=800\) is the number of resampled datasets, not an added sample size. (2) The failure proportion is \(30/200=0.15\), or 15%. A summary based on the remaining 170 fits describes successful fits under the specified procedure and is conditional on convergence. (3) It suggests limited estimate variability under that bootstrap procedure, but does not establish that the model assumptions or predictions are adequate. (4) Measurements from the same subject are grouped and may be dependent. Row resampling breaks that grouping and represents a different sampling process from resampling subjects with their profiles intact.