Why Bootstrap Shape Matters
A bootstrap distribution is often summarized by one standard error or a confidence interval. Those summaries can be inadequate when bootstrap refits repeatedly land in distinct regions, or when a small fraction of estimates extend far beyond the main body. In population pharmacokinetics, such shapes can arise when the data support more than one region of the parameter space, when parameters are weakly identified, or when fitting is unstable. They can also arise from avoidable implementation problems. The shape is therefore a diagnostic, not merely a plotting detail.
The task here is not to decide that every bump is a separate scientific explanation. It is to establish whether the pattern is reproducible, trace it to plausible causes, and report uncertainty without making the bootstrap appear more decisive than it is.
The resampling scheme, model, estimation procedure, convergence criteria, and rules for retaining replicates define the distribution being examined. A subject-level bootstrap, a parametric bootstrap, and a refit-versus-fixed-parameter bootstrap do not necessarily answer the same uncertainty question. The basic requirement for diagnosing shape is that the replicates be generated and processed consistently with the intended bootstrap workflow. In particular, distinguish successful refits from failures using prespecified criteria; do not silently discard an inconvenient cluster or tail.
Inspect both the parameter scale and, where appropriate, a transformed scale. A positive parameter such as clearance or an exponential variance parameter may be more naturally examined on the log scale. A transformation can clarify relative variation or reduce visual skew, but it cannot make two substantively distinct solution regions into one well-behaved distribution. Retain the original scale for interpretation and state the scale used for each plot or summary.
First Diagnose the Replicates
Begin with replicate-level information, not a preferred summary statistic. Plot a histogram or density estimate, and inspect the ordered estimates or empirical quantiles. Repeat the display with reasonable bin widths or smoothing choices: a visual mode that appears only under one arbitrary plotting choice is weak evidence of a distinct cluster. Mark the original-data estimate and any parameter boundaries. When several parameters are involved, inspect relevant pairs and correlations as well as one-dimensional plots; two apparent modes in a marginal distribution may correspond to different combinations of parameters.
Then link each estimate to its fit information. Useful fields include the objective function or likelihood, convergence status, boundary hits, starting values, gradients or other available convergence diagnostics, and estimates of correlated parameters. Ask whether replicates in different regions have comparable fit quality and whether their parameter combinations are coherent. A parameter at a boundary, a failed convergence check, or a markedly worse objective can suggest a fitting problem. Similar objective values and repeatable, distinct parameter combinations are more consistent with alternative fit solutions, although they do not prove that both regions are scientifically meaningful.
Check the estimates, plot on a scientifically interpretable scale, and test whether the apparent modes or tail persist under reasonable plotting choices.
Compare convergence information, objective values, boundary behavior, and related parameter estimates across the main body, secondary clusters, and tail.
Refit selected replicates from multiple starting values and examine whether the same solution regions recur. Check whether data sparsity or parameter trade-offs make the estimates sensitive to small resampling changes.
Give the number of valid replicates and the relevant clusters or tail behavior alongside suitable quantiles and clear limitations.
Worked Examples
Worked Example: Two Clearance Clusters
Suppose 200 hypothetical subject-level bootstrap refits produce two visible clusters for clearance, \(CL\), measured in L/h. There are 132 estimates in a lower cluster with mean 4.80 L/h and within-cluster standard deviation 0.350 L/h, and 68 in a higher cluster with mean 6.10 L/h and within-cluster standard deviation 0.400 L/h. Assume the two groups are defined from the observed separation for this descriptive calculation, not as known biological subpopulations.
The overall bootstrap mean is the count-weighted mean:
The mean alone lies between the clusters and describes neither one particularly well. To see how the separation affects spread, use the variance decomposition for a two-group mixture. The within-group contribution is \(0.66(0.350^2)+0.34(0.400^2)=0.13525\ (\text{L/h})^2\). The between-group contribution is \(0.66(0.34)(6.10-4.80)^2=0.37924\ (\text{L/h})^2\). Thus the mixture variance, using the reported within-group variances as descriptive component variances, is approximately \(0.51449\ (\text{L/h})^2\), corresponding to a standard deviation of \(\sqrt{0.51449}=0.7173\ \text{L/h}\).
Most of this variance comes from separation between clusters, not variation within them. The appropriate next step is to compare fit diagnostics and parameter combinations in each cluster and to refit representative replicates from multiple starting values. Reporting only \(5.242\) L/h and \(0.7173\) L/h would conceal the structure that prompted the investigation.
Worked Example: Is a Second Mode a Fitting Artifact?
In a hypothetical bootstrap analysis, 120 replicates yield 83 estimates of a volume parameter near 28 L and 37 estimates near 46 L. The two groups have similar objective values to the precision of the fit, and both pass the prespecified convergence checks. To investigate, an analyst selects 10 replicates from each group and refits each from four dispersed initial values.
Suppose the refits show that, for each selected replicate, at least two starting values converge successfully to each of the two solution regions, with objective values differing by less than 0.02 objective units. This pattern makes a single-start optimizer failure a less persuasive explanation: both regions can be reached repeatedly from valid starts, and neither is consistently inferior by the recorded objective. It is evidence of alternative fit solutions under this model and dataset, not proof that either region represents a distinct biological state.
The analyst should also inspect related parameters. If the high-volume cluster systematically pairs with lower clearance and the same concentration data fit nearly as well, that relationship may indicate a parameter trade-off. If, instead, one region occurs only when a variance parameter hits a boundary or a convergence diagnostic fails, the apparent mode may be driven by numerical problems. The evidence determines whether to report a stable alternative-solution pattern, investigate estimation further, or classify affected replicates as invalid under the prespecified criteria. The 37-to-83 split is a bootstrap frequency, not the probability that either parameter value or biological explanation is true.
Worked Example: A Long Tail in Absorption Estimates
Consider 100 hypothetical successful bootstrap estimates of an absorption-rate parameter \(k_a\), measured in h\(^{-1}\). Ninety estimates form a central group with mean 1.02 h\(^{-1}\); the remaining 10 form a right tail with mean 4.10 h\(^{-1}\). The overall mean is
The tail raises the mean by 0.308 h\(^{-1}\) above the central-group mean, despite containing only 10% of the replicates. A reasonable investigation would check whether those 10 refits have similar objective values and convergence quality, whether their estimates of other absorption or disposition parameters differ systematically, and whether the tail recurs under a larger bootstrap run. Sparse early sampling could leave absorption timing weakly identified, but that explanation must be checked against the study design and fit diagnostics rather than inferred from the plot alone.
A responsible summary would show the empirical distribution, report the number of valid replicates and the upper-tail behavior, and give appropriate quantiles with their units. It would not replace the tail by the mean alone, nor delete the 10 estimates simply because they are inconvenient. If they fail an explicit validity criterion, report their number and the criterion; if they are valid, retain them in the uncertainty summary and explain the resulting limitation.
A Short Justification: Why a Single Standard Deviation Can Mislead
For a distribution consisting of two groups, let the first group have proportion \(p\), mean \(\mu_1\), and variance \(\sigma_1^2\), and let the second have proportion \(1-p\), mean \(\mu_2\), and variance \(\sigma_2^2\). The overall mean is \(\mu=p\mu_1+(1-p)\mu_2\). The total variance is the weighted sum of within-group variances plus the variance of the group means:
To justify the decomposition, write \(X-\mu=(X-\mu_g)+(\mu_g-\mu)\), where \(\mu_g\) is the mean of the group containing \(X\). Squaring and taking the expectation gives the within-group squared deviation, the between-group squared deviation, and a cross-product. The cross-product has expectation zero because the within-group deviations average to zero within each group. The between-group term for two groups simplifies to \(p(1-p)(\mu_1-\mu_2)^2\). Thus a large overall standard deviation can reflect separation of groups rather than broad uncertainty around one central value. A normal approximation based on that overall standard deviation may then describe neither group well.
Summarize Without Hiding Structure
Choose summaries that match the observed shape and scientific purpose. A median and quantiles can describe the center and spread without being dominated by a few extreme values, but they do not explain why a cluster or tail exists. Report a plot or clear cluster description when modes are evident. If a cluster distinction is defensible and reproducible, give its replicate count or proportion and within-cluster summaries in addition to the overall distribution. Make clear that cluster frequencies describe outcomes under the chosen resampling and fitting procedure; they are not posterior probabilities or probabilities that a scientific mechanism is true.
If the analysis uses percentile or other bootstrap intervals, inspect whether their endpoints span a gap between clusters or include a long tail. An interval can be numerically valid as a quantile summary and still obscure that the bootstrap estimates do not form one compact region. Do not choose a single interval to imply a simple, symmetric uncertainty distribution when the plotted replicates contradict that impression. The reporting choice should reflect the estimand and the purpose of the analysis.
| Observed pattern | Useful checks | Reporting emphasis |
|---|---|---|
| Separated, repeatable clusters | Multiple starts, objective values, correlated parameters | Show both clusters and their replicate frequencies; avoid presenting one center as representative of both. |
| Small number of extreme estimates | Convergence, boundaries, replicate data, sensitivity to resampling | State how many estimates form the tail and whether they meet the validity criteria. |
| Pattern changes with plot settings or more replicates | Bin widths, smoothing choices, bootstrap stability | Describe the evidence as uncertain rather than asserting distinct modes. |
The main alternatives are not interchangeable. A single mean and standard deviation are compact but can conceal clusters. Quantiles are robust to some extremes but can still span a gap without revealing it. Separate cluster summaries are informative when the clustering is stable and diagnostically meaningful, but can create false certainty if the grouping depends on arbitrary cut points. A transformed-scale analysis can help with skewness but does not remove the need to inspect original-scale values. In every case, state how many replicates were attempted, how many were valid under the stated rules, and how excluded failures were handled.
Common Mistakes and Limitations
- Calling every bump a separate solution. Density plots depend on smoothing and finite replicate counts. Check stability across reasonable displays and inspect the replicate estimates directly.
- Discarding a tail without a rule. A large estimate is not automatically a failed fit. Apply prespecified fit-validity criteria, and report exclusions and their reasons.
- Interpreting cluster frequency as truth probability. A bootstrap frequency depends on the resampling scheme and fitting process. It does not give the probability that a parameter value or mechanism is true.
- Assuming the optimizer explains every mode. Alternative numerical solutions are one possibility; weak identification or data sensitivity can also produce distinct valid estimates. Diagnostics guide the explanation but may not resolve it.
- Assuming a larger bootstrap removes the problem. More replicates can clarify whether a pattern is stable, but do not correct weak identification, misspecification, or inadequate sampling design.
A bootstrap distribution characterizes the behavior of a particular estimation procedure under a particular resampling scheme and fitted model. It cannot, by itself, establish that the model is scientifically correct or determine which of several valid fit regions is preferable. When the cause remains unresolved, report that uncertainty explicitly rather than forcing the results into a single uncomplicated summary.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A bootstrap distribution has two apparent peaks, but one disappears when the histogram bin width changes. What should you conclude and check next?
- In 150 valid replicates, 30 estimates lie in a separated cluster. What does the value 30/150 describe, and what does it not describe?
- Why can a long right tail materially raise the bootstrap mean even when most estimates are near one value?
- Name two diagnostics that can help distinguish alternative fit solutions from optimizer problems.
Solutions. (1) The evidence for a distinct mode is not yet stable; inspect the replicate values and ordered estimates, try reasonable plotting choices, and check whether the pattern recurs with additional replicates. (2) The ratio is 0.20, or 20%, of the valid replicates under that resampling and fitting procedure. It is not the probability that the cluster is true or that a biological mechanism exists. (3) The mean uses every estimate and is sensitive to distance from the center; a small number of sufficiently large values can contribute substantially to the sum. (4) Examples include comparing convergence checks and objective values across regions, refitting selected replicates from multiple starting values, checking boundary hits, and examining related parameter estimates.