Bootstrap Replicate Counts and Monte Carlo Error

Why the Number of Bootstrap Replicates Matters

A bootstrap analysis approximates a sampling distribution by repeatedly resampling the observed data and recalculating a statistic. The number of resamples, $B$, determines how much simulation noise remains in quantities computed from those replicates. A small $B$ can make a bootstrap standard error or interval endpoint noticeably variable across runs, even when the original data have not changed.

This simulation noise is Monte Carlo error. It is separate from the uncertainty represented by the bootstrap distribution and from any error caused by the bootstrap approximation itself. Increasing $B$ can reduce Monte Carlo error; it does not add information to the original data or automatically improve the bootstrap method’s approximation to the sampling distribution.

Definition: Conditional on the observed data, let $T_1^*,\ldots,T_B^*$ be statistics calculated from $B$ independently generated bootstrap samples. Each $T_b^*$ is drawn from the same conditional bootstrap distribution. Monte Carlo error is the variation in a computed bootstrap quantity, such as a standard error or quantile, caused by using this finite set of $B$ replicates to approximate that distribution.

The replicates are not $B$ new observations from the population. They are repeated calculations based on resampling the same observed data. Treating $B$ as though it were the original sample size confuses computational precision with statistical information.

Monte Carlo Error in a Bootstrap Standard Error

The bootstrap standard error is usually estimated by the sample standard deviation of the bootstrap statistics:

\widehat{\operatorname{SE}}_{\mathrm{boot}} = s^* = \sqrt{\frac{1}{B-1}\sum_{b=1}^{B}(T_b^*-\bar{T}^*)^2}, \qquad \bar{T}^*=\frac{1}{B}\sum_{b=1}^{B}T_b^*.

This estimate itself varies from one set of bootstrap replicates to another. If the conditional bootstrap distribution is approximately normal with standard deviation $\sigma^*$, the sample variance has variance $2(\sigma^*)^4/(B-1)$. Applying a first-order approximation to the square root gives the relative Monte Carlo standard error of $s^*$:

\frac{\operatorname{MCSE}(s^*)}{\sigma^*} \approx \frac{1}{\sqrt{2(B-1)}}.

In practice, $\sigma^*$ is unknown, so a useful estimated absolute Monte Carlo standard error is $\widehat{\operatorname{MCSE}}(s^*)\approx s^*/\sqrt{2(B-1)}$. This is an approximation, most appropriate when the bootstrap distribution is reasonably close to normal and has a finite variance. Strong skewness or heavy tails can make the approximation less accurate.

Worked Example: Assessing Monte Carlo Error in a Bootstrap Standard Error

A hypothetical claim-severity analysis uses $B=500$ independent bootstrap resamples. The standard deviation of the 500 bootstrap estimates is $s^*=\$12{,}000$. Estimate the relative and absolute Monte Carlo standard error of this bootstrap standard error, using the approximate normal-distribution formula.

Method and assumptions. Treat the conditional distribution of the bootstrap estimates as approximately normal, and use $\widehat{\operatorname{MCSE}}(s^*)\approx s^*/\sqrt{2(B-1)}$. The observed data and the resampling procedure are held fixed; only the finite replicate set contributes to this calculation.

First calculate the denominator: \[ 2(B-1)=2(500-1)=998, \qquad \sqrt{998}\approx31.591. \]

The estimated relative Monte Carlo standard error is \[ \frac{1}{\sqrt{998}}\approx0.03166, \] or about $3.17\%$. The estimated absolute Monte Carlo standard error is \[ \frac{\$12{,}000}{31.591}\approx \$379.85. \] Rounded to the nearest dollar, the Monte Carlo standard error is $\$380$.

Check. The absolute error estimate divided by the reported bootstrap standard error is $\$379.85/\$12{,}000\approx0.03165$, consistent with the relative estimate. The method avoids interpreting $B=500$ as 500 additional claims; it quantifies only the simulation variability of the standard-error estimate.

A practical calculation can also be made backwards. To target an approximate relative Monte Carlo standard error no greater than $r$, require \[ 1/\sqrt{2(B-1)}\leq r, \] which gives $B\geq 1+1/(2r^2)$. The result is a planning guide, not a guarantee: it depends on the approximate normality assumption and concerns the standard error estimate, not every bootstrap output.

Monte Carlo Error in a Quantile Endpoint

A percentile endpoint is a quantile of the conditional bootstrap distribution. Let $F$ be the conditional cumulative distribution function of a bootstrap statistic $T^*$, and define its $p$-quantile by $q_p=\inf\{t:F(t)\geq p\}$. The empirical quantile from $B$ replicates approximates $q_p$, but its accuracy depends on both $B$ and the shape of $F$ near the endpoint.

For a continuous distribution with a positive, smooth density $f$ at $q_p$, so that $F(q_p)=p$, the approximate Monte Carlo standard error of the empirical quantile is

\operatorname{MCSE}(\hat q_p) \approx \sqrt{\frac{p(1-p)}{B\,f(q_p)^2}}.

The formula is useful only under its stated conditions. It follows by tracking the empirical cumulative probability at $q_p$, whose variance is $p(1-p)/B$, and converting a small probability error into a value-scale error using the local slope $f(q_p)$. A low density near the endpoint means that a small change in cumulative probability can correspond to a large change in the endpoint. Thus, a large $B$ may still leave substantial endpoint error when the distribution is sparse near the quantile.

Identify the target quantile.
For a lower or upper percentile endpoint, specify its probability $p$, such as $0.025$ or $0.975$.

Check the distributional condition.
The density-based approximation requires a continuous distribution with a positive, reasonably smooth density at the target quantile.

Estimate endpoint precision.
Use the formula with an appropriate estimate of $f(q_p)$, and interpret the result in the units of the statistic.

Verify stability computationally.
Increase $B$, or compare independently generated batches, and check whether endpoint changes are small relative to the precision needed for the analysis.

Worked Example: Estimating Error in a Smooth Bootstrap Endpoint

A hypothetical reserve estimator has a bootstrap distribution that is treated as continuous near its $2.5\%$ quantile. Its estimated density there is $0.020$ per thousand dollars. With $B=1{,}000$ replicates, estimate the Monte Carlo standard error of this lower endpoint and the expected replicate counts on either side of it.

Method and assumptions. Use the smooth-density quantile approximation. The probability is $p=0.025$, the density is $f(q_p)=0.020$ per thousand dollars, and the replicates are independently drawn from the same conditional bootstrap distribution.

The estimated endpoint Monte Carlo standard error is \[ \sqrt{\frac{0.025(1-0.025)}{1{,}000(0.020)^2}} = \sqrt{\frac{0.024375}{0.4}} = \sqrt{0.0609375} \approx0.2469 \] thousand dollars. This is approximately $\$247$.

Because the distribution is assumed continuous and $F(q_p)=p$, the expected number of replicates at or below the endpoint is $Bp=1{,}000(0.025)=25$. The expected number above it is $B(1-p)=1{,}000(0.975)=975$.

Check. The expected counts sum to $25+975=1{,}000$, and the endpoint error is in the same units as the estimator. The density assumption matters: if the bootstrap distribution has an atom at the endpoint, the density-based calculation and the $Bp$ count need not apply.

Discrete Bootstrap Distributions and Quantile Counts

Bootstrap distributions are often discrete, even when the population quantity of interest is continuous. For example, resampling a finite data set can produce only finitely many distinct samples and therefore finitely many values of a statistic. In a discrete distribution, a quantile can satisfy $F(q_p)>p$. The correct expected number of replicates at or below a fixed endpoint $q_p$ is $B F(q_p)$, and the expected number above it is $B[1-F(q_p)]$. These counts do not generally equal $Bp$ and $B(1-p)$.

The smooth-density quantile formula also does not directly apply at an atom: there may be no positive density there, and a small change in the empirical cumulative probability may leave the empirical quantile unchanged or move it to a neighboring support value. Reported quantiles can also depend on the convention used to interpolate between ordered replicates. When the distribution is discrete or highly tied, assess endpoint stability directly across larger runs or independent batches rather than presenting a density-based approximation as exact.

Worked Example: Correct Counts at a Discrete Bootstrap Quantile

Suppose a hypothetical bootstrap statistic has conditional distribution $P(T^*=0)=0.20$, $P(T^*=0.5)=0.60$, and $P(T^*=1)=0.20$. For $p=0.50$, its quantile is $q_{0.50}=0.5$. If $B=200$, find the expected number of replicates at or below the quantile and the expected number above it.

Method. Apply the general count formulas using the actual cumulative probability at the quantile. Here \[ F(0.5)=P(T^*\leq0.5)=0.20+0.60=0.80. \]

Therefore, the expected count at or below $0.5$ is \[ B F(q_{0.50})=200(0.80)=160, \] and the expected count above $0.5$ is \[ B[1-F(q_{0.50})]=200(0.20)=40. \]

Check and common mistake avoided. The counts total $160+40=200$. Using $Bp=100$ and $B(1-p)=100$ would be wrong: those expressions require a continuous distribution with $F(q_p)=p$, whereas this discrete distribution has $F(0.5)=0.80$.

Choosing and Checking a Replicate Count

A defensible choice of $B$ starts with the quantity that must be stable. A standard-error estimate and a tail quantile do not necessarily need the same number of replicates. For a standard error, a target relative Monte Carlo error can provide a first planning value. For a smooth quantile, the local density and the desired value-scale precision also matter. Tail endpoints can be especially demanding because relatively few replicates fall in the corresponding tail.

A stability check complements a formula-based estimate. One approach is to generate an initial batch, calculate the target quantity, then add replicates and recalculate it. Another is to generate independent batches and compare their results. In either case, specify a practical tolerance before deciding that the result is stable. For example, an endpoint that changes by a small amount in the units relevant to the analysis may be adequate, even if it is not identical between runs.

Independent batches are a diagnostic, not independent samples of the underlying population. Their variation reflects Monte Carlo error conditional on the same observed data. If a percentile interval has two endpoints, assess each endpoint, and consider the interval width as well; the two endpoints are calculated from the same replicates and their simulation errors need not be independent.

Worked Example: Planning a Replicate Count for a Standard Error

An analyst wants the approximate relative Monte Carlo standard error of a bootstrap standard deviation to be no more than $2\%$. Assume the bootstrap distribution is approximately normal. Find the smallest integer $B$ meeting this target, and check the resulting relative error.

Method. Use $1/\sqrt{2(B-1)}\leq0.02$. Squaring both sides gives \[ 2(B-1)\geq\frac{1}{0.02^2}=\frac{1}{0.0004}=2{,}500. \] Thus $B-1\geq1{,}250$, so $B\geq1{,}251$.

At $B=1{,}251$, the approximation gives \[ \frac{1}{\sqrt{2(1{,}251-1)}}= \frac{1}{\sqrt{2{,}500}}=0.02. \] Therefore $1{,}251$ is the smallest integer meeting the stated approximate target.

Check and common mistake avoided. Substituting $B=1{,}251$ returns exactly $2\%$ under the formula. The calculation does not mean that the underlying standard error is known to $2\%$ in a statistical sense; it is a conditional Monte Carlo precision target under the approximate normality assumption.

Common Traps

Confusing replicate count with sample size. $B$ controls the numerical approximation to the bootstrap distribution. It is not the number of independent observations collected from the population.
Assuming more replicates fix every problem. Increasing $B$ reduces Monte Carlo error, but does not remove bias from an unsuitable bootstrap procedure or correct limitations in the original data.
Using $Bp$ for every quantile. This expected count applies when the distribution is continuous and $F(q_p)=p$. In general, use $B F(q_p)$ at or below the quantile and $B[1-F(q_p)]$ above it.
Ignoring the density near a smooth endpoint. The endpoint’s Monte Carlo error depends on $f(q_p)$, not only on $B$. Sparse tails can require substantially more replicates for a given value-scale precision.
Reporting a stability check without a tolerance. State what was compared and what change was acceptable in the units of the result. “The estimate looked stable” is not a reproducible precision criterion.

Key takeaway: Bootstrap replicates approximate a conditional distribution; they are not additional data. Quantify the simulation error for the particular output of interest, respect the assumptions behind each approximation, and use increasing-$B$ or independent-batch checks to confirm practical stability.

Check Your Understanding

Use the stated assumptions in each question. Each answer concerns Monte Carlo precision conditional on the observed data.

A bootstrap standard deviation is $5$ units from $B=200$ replicates. Under the approximate normal formula, what is its estimated absolute Monte Carlo standard error? (Answer: $5/\sqrt{398}\approx0.251$ units.)
A continuous bootstrap distribution has a $97.5\%$ quantile $q$, with $F(q)=0.975$. For $B=800$, what are the expected counts at or below $q$ and above $q$? (Answer: $780$ at or below and $20$ above.)
A discrete bootstrap distribution has $F(q)=0.70$ at its $p=0.50$ quantile. With $B=300$, what are the expected counts at or below $q$ and above $q$? (Answer: $210$ at or below and $90$ above.)
For a continuous endpoint with $p=0.975$, $B=2{,}000$, and $f(q_p)=0.05$ per unit, estimate its Monte Carlo standard error using the smooth-quantile formula. (Answer: $\sqrt{0.975(0.025)/[2{,}000(0.05)^2]}\approx0.0698$ units.)
Under the approximate normal formula, what is the smallest integer replicate count for a relative Monte Carlo standard error no greater than $1.5\%$ for a bootstrap standard deviation? (Answer: $B=2{,}224$, since $1/\sqrt{2(2{,}224-1)}\approx0.0150$, while $B=2{,}223$ is just above $0.015$.)

What You'll Learn