From Bootstrap Replicates to a Standard Error
A bootstrap analysis produces repeated estimates of a model parameter by refitting the model to resampled data or to data simulated under a specified bootstrap scheme. The resulting estimates show how much the fitted parameter can vary under that scheme. Their spread can be used to estimate a standard error, and that standard error can then be used to construct a normal-based interval.
For an NLME analysis, each replicate must follow the intended resampling design. For example, a subject-level bootstrap resamples subjects as clusters when subjects are the independent sampling units. The standard error calculated from the replicate estimates reflects the uncertainty represented by that design; it is not a design-free measure of uncertainty.
This standard deviation estimates the spread of the bootstrap distribution. It estimates uncertainty in \(\hat{\theta}\) under the sampling scheme represented by the bootstrap, rather than uncertainty in the bootstrap mean itself. The bootstrap mean can also be informative: \(\bar{\theta}^{*}-\hat{\theta}\) is a simple estimate of bootstrap bias. But the usual normal-based bootstrap interval is centered on the original estimate \(\hat{\theta}\), not automatically on \(\bar{\theta}^{*}\).
Why the Sample Standard Deviation Is Used
Conditional on the observed data and the specified bootstrap procedure, imagine drawing independent bootstrap replicates from the bootstrap distribution. The replicate estimates then act as a sample from that distribution. The sample variance, with denominator \(B-1\), estimates the variance of that distribution without the downward bias that would result from using \(B\) when estimating variance around the sample mean.
The centering matters. For any fixed value \(c\), $$ \sum_{b=1}^{B}(\hat{\theta}^{*(b)}-c)^2 = \sum_{b=1}^{B}(\hat{\theta}^{*(b)}-\bar{\theta}^{*})^2 + B(\bar{\theta}^{*}-c)^2. $$ The second term is nonnegative, so the sum of squared deviations is minimized at \(c=\bar{\theta}^{*}\). This is why the bootstrap standard deviation measures spread around the replicate mean, even when the interval itself is centered on \(\hat{\theta}\).
Worked Example: Calculating a Bootstrap Standard Error
Suppose the original model estimates a clearance parameter as \(\hat{\theta}=10.0\) L/h. For a deliberately small teaching example, five valid bootstrap fits give estimates of 8.0, 9.0, 10.0, 11.0, and 12.0 L/h. The replicate mean is $$ \bar{\theta}^{*}=\frac{8+9+10+11+12}{5}=10.0\ \text{L/h}. $$
The squared deviations from the mean are \(4,1,0,1,4\), with total \(10\ (\text{L/h})^2\). Thus $$ SE_{\mathrm{boot}} = \sqrt{\frac{10}{5-1}} = \sqrt{2.5} = 1.581\ \text{L/h}. $$ This is the sample standard deviation of the five replicate estimates. Five replicates are far too few for a reliable practical interval; the example illustrates the calculation, not a recommended number of fits.
Constructing a Normal-Based Bootstrap Interval
A two-sided normal-based interval uses the bootstrap standard error as its scale and the original-data estimate as its center. For a nominal confidence level \(1-\alpha\), let \(z_{1-\alpha/2}\) be the corresponding standard normal quantile. The interval is
For a 95% interval, \(z_{0.975}\) is approximately 1.96. Using the teaching example above gives $$ 10.0\pm1.96(1.581) = 10.0\pm3.099, $$ so the interval is approximately 6.901 to 13.10 L/h. Under the assumptions supporting the procedure, this is an interval procedure intended to cover the fixed parameter in 95% of repeated applications of the sampling and analysis process. It does not mean there is a 95% probability that the fixed parameter lies in this particular computed interval.
The interval uses the bootstrap estimates to estimate the standard error, but the critical value comes from the standard normal distribution. The bootstrap does not make the resulting interval automatically robust to skewness, boundaries, bias, or model problems.
Worked Example: A Normal-Based Interval for Clearance
Suppose a hypothetical NLME analysis estimates clearance as \(\hat{\theta}=12.00\) L/h. Across a sufficiently large set of valid subject-level bootstrap fits, the replicate estimates have mean \(12.04\) L/h and sample standard deviation \(0.180\) L/h. Assume the bootstrap design matches the study’s independent sampling units and the distribution of estimates is reasonably symmetric and not strongly affected by a boundary.
For a 95% normal-based interval, use the original estimate as the center: $$ 12.00\pm1.96(0.180) = 12.00\pm0.3528. $$ The endpoints are \(11.6472\) and \(12.3528\) L/h, or approximately 11.65 to 12.35 L/h.
The bootstrap mean is \(12.04\) L/h, not \(12.00\) L/h, but that difference does not change the center of this conventional interval. The estimated bootstrap bias is \(12.04-12.00=0.040\) L/h. If bias is substantial, simply ignoring it may be questionable; changing the center is a different interval choice and should be stated explicitly.
Scale, Shape, and the Interval’s Assumptions
A normal-based interval is most defensible when the estimator’s sampling distribution is approximately normal on the scale used, its standard error is meaningfully estimated by the bootstrap spread, and the fitted bootstrap replicates are a relevant approximation to repeated samples from the target study design. In an NLME setting, this requires more than obtaining a numerical standard deviation: the resampling scheme must respect the design, and the refits must provide valid estimates for the parameter being summarized.
A parameter constrained to be positive can be poorly represented by a symmetric interval on its original scale, especially near zero. One option is to work with its logarithm. If \(\hat{\phi}=\log(\hat{\theta})\) and the bootstrap standard error is computed for log-scale estimates, a normal-based interval on that scale is $$ \hat{\phi}\pm z_{1-\alpha/2}SE_{\mathrm{boot}}(\hat{\phi}). $$ Exponentiating both endpoints gives an interval for \(\theta\). This procedure requires interpreting the bootstrap distribution on the log scale; it does not justify using a log-scale standard error with an untransformed estimate.
Worked Example: A Normal-Based Interval on the Log Scale
Suppose the original clearance estimate is \(8.00\) L/h, and the bootstrap standard deviation of the replicate log-clearance estimates is \(0.200\). The original log estimate is $$ \hat{\phi}=\log(8.00)=2.0794. $$ A 95% interval on the log scale is $$ 2.0794\pm1.96(0.200) = 2.0794\pm0.3920, $$ giving endpoints \(1.6874\) and \(2.4714\). Exponentiating, $$ [\exp(1.6874),\exp(2.4714)] \approx [5.405,11.84]\ \text{L/h}. $$
The interval is positive and asymmetric around 8.00 L/h on the original scale. Its interpretation depends on the normal approximation being reasonable for log clearance, not necessarily for clearance itself.
How This Differs from Quantile-Based Intervals
A normal-based interval uses one summary of spread—the bootstrap standard error—and normal critical values. A percentile interval instead takes lower and upper quantiles of the bootstrap estimates themselves. These procedures can give different endpoints because they use different information and assumptions.
| Feature | Normal-based bootstrap interval | Percentile bootstrap interval |
|---|---|---|
| Construction | Original estimate plus or minus a normal critical value times bootstrap standard error | Lower and upper quantiles of the bootstrap estimates |
| Center and shape | Symmetric around the original estimate on the analysis scale | Endpoints follow the observed bootstrap distribution’s quantiles |
| Main vulnerability | Can perform poorly when normal approximation, symmetry, or scale assumptions are unsuitable | Can perform poorly when bootstrap quantiles do not adequately approximate the desired sampling quantiles |
For the clearance analysis above, suppose the bootstrap 2.5th and 97.5th percentiles are 11.70 and 12.41 L/h. The percentile interval is 11.70 to 12.41 L/h, while the normal-based interval is 11.65 to 12.35 L/h. The difference is not an arithmetic contradiction: the first uses empirical quantiles, whereas the second uses the replicate standard deviation and normal critical values. Neither interval is guaranteed to be preferable in every setting.
Other uncertainty summaries have different constructions as well. Wald intervals use a model-based standard error, commonly derived from the fitted likelihood curvature; a bootstrap standard error replaces that source of scale information with the spread of bootstrap estimates. BCa intervals adjust bootstrap quantiles for bias and acceleration. These approaches should not be treated as interchangeable labels for the same calculation.
Diagnostics, Failure Modes, and Limitations
Before reporting a bootstrap standard error and normal-based interval, check that the replicate estimates refer to the same parameter on the same scale and units, and that the replicates were generated under the intended design. Also confirm that the estimates used in the calculation are valid fits under the stated analysis procedure. A spread computed after silently excluding difficult fits can misrepresent uncertainty; the treatment of failed fits must follow a defensible prespecified procedure.
A histogram or other summary of replicate estimates can reveal features that make a normal-based interval doubtful. Strong skewness, multiple clusters, extreme estimates, or a pile-up near a parameter boundary are warnings that a symmetric interval may be misleading. These patterns do not, by themselves, identify the cause; they may reflect weak information, model-fitting issues, or the behavior of the estimator. The standard deviation alone cannot show these features.
- Do not mistake a small standard error for sound inference. A narrow spread can result from a bootstrap design that fails to represent the actual sampling process.
- Do not center the interval on the replicate mean without saying so. The conventional normal-based bootstrap interval is centered on \(\hat{\theta}\); a bias-adjusted center defines a different procedure.
- Do not assume symmetry from the formula. The formula creates symmetric endpoints on the chosen scale even if the replicate distribution is skewed.
- Do not ignore parameter constraints. A symmetric interval for a positive quantity can include impossible values; consider whether a justified transformation is more suitable.
The normal-based interval is attractive because it is simple and reports uncertainty in familiar estimate-plus-or-minus-standard-error form. Its limitation is equally important: a bootstrap estimate of spread does not establish that normal critical values give suitable coverage. Use it when the approximation is credible, and compare its implications with alternative bootstrap summaries when the replicate distribution raises concerns.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Five hypothetical bootstrap estimates of a parameter, in mg/L, are 4.6, 4.8, 5.0, 5.2, and 5.4. Calculate their bootstrap standard error.
- If the original estimate is 5.1 mg/L, use the standard error from question 1 to calculate a 95% normal-based interval. State its center and units.
- Why might a percentile interval differ from the normal-based interval even when both use the same bootstrap replicates?
- Bootstrap estimates of a positive parameter are strongly right-skewed and many are close to zero. Give one reason to question a normal-based interval on the original scale.
Solutions. (1) The replicate mean is \(5.0\) mg/L. The squared deviations sum to \(0.16+0.04+0+0.04+0.16=0.40\ (\text{mg/L})^2\), so the sample standard deviation is \(\sqrt{0.40/(5-1)}=\sqrt{0.10}=0.3162\) mg/L. (2) The interval is \(5.1\pm1.96(0.3162)=5.1\pm0.6198\), or approximately 4.480 to 5.720 mg/L. It is centered on the original estimate, 5.1 mg/L. (3) The normal-based interval uses the replicate standard deviation and normal critical values; the percentile interval uses empirical quantiles, so skewness or other distributional features can give different endpoints. (4) A symmetric interval on the original scale may poorly approximate a skewed distribution and may extend into values that are impossible for a positive parameter. A transformation or a different interval method may be more appropriate, subject to its own assumptions.