Bootstrap Performance In Small Samples

Why Small Samples Challenge the Bootstrap

A nonparametric bootstrap resamples observed losses, with replacement, to approximate the variation an estimator would have across repeated samples. Its appeal is that it can be applied without specifying a parametric severity distribution. Its limitation is equally important: the resampling distribution is built only from the observations in hand. When there are few observations, that empirical distribution may be a poor representation of the population, especially if the observed sample does not capture its range or skewness.

For severity analysis, this means a bootstrap can calculate variation precisely conditional on the observed sample while still giving unreliable information about the population. A large number of resamples reduces computational noise in a bootstrap calculation; it does not add new information about losses that were never observed.

Definition: Given independent, identically distributed observations \(x_1,\ldots,x_n\), a nonparametric bootstrap replicate is a sample of size \(n\) drawn with replacement from the observed values. A bootstrap replicate of an estimator is that estimator recalculated on the resampled data. The bootstrap distribution is the conditional distribution of these replicate estimates, given the observed data.

Two different questions must therefore be kept separate. First, how variable are the replicate estimates for this particular observed sample? Second, if the data collection and interval procedure were repeated under a specified population model, how often would the resulting interval contain the true parameter? The first is a conditional bootstrap calculation; the second is a coverage property of the procedure. Small-sample performance is assessed by paying attention to both.

Exact Bootstrap Variability for a Sample Mean

For a sample mean, the bootstrap standard error can be derived directly. Let \(\bar{x}\) be the observed sample mean and let \(X^*\) be one draw from the empirical distribution that assigns probability \(1/n\) to each observed value. A bootstrap sample mean \(\bar{X}^*\) is the average of \(n\) independent draws from this empirical distribution.

E^*[X^*]=\bar{x}, \qquad \operatorname{Var}^*(X^*)=\frac{1}{n}\sum_{j=1}^n (x_j-\bar{x})^2

Here \(E^*\) and \(\operatorname{Var}^*\) refer to expectation and variance conditional on the observed data. Since the \(n\) draws in a bootstrap sample are independent conditional on those data, the variance of their average is the variance of one draw divided by \(n\). Thus:

\operatorname{Var}^*(\bar{X}^*) =\frac{1}{n^2}\sum_{j=1}^n (x_j-\bar{x})^2, \qquad \operatorname{SE}^*(\bar{X}^*) =\sqrt{\frac{1}{n^2}\sum_{j=1}^n (x_j-\bar{x})^2}.

If \(s^2=\sum_{j=1}^n(x_j-\bar{x})^2/(n-1)\) denotes the usual sample variance, the same result is \(\operatorname{Var}^*(\bar{X}^*)=(n-1)s^2/n^2\). The denominator matters: the exact conditional bootstrap variance uses the empirical distribution, whose probabilities are \(1/n\), not an unbiased estimate of a population variance.

Worked Example: Exact Bootstrap Standard Error With Three Losses

A hypothetical insurer observes three covered losses of \(2\), \(4\), and \(8\) thousand dollars. Estimate the mean severity and calculate the exact nonparametric bootstrap standard error of that estimate. State one small-sample limitation.

Method. Use the sample mean as the estimator and calculate its conditional bootstrap variance from the empirical distribution. The observations are \(2,4,8\), each with bootstrap draw probability \(1/3\).

The observed mean is:

\bar{x}=\frac{2+4+8}{3}=\frac{14}{3}=4.6667

The deviations from the mean are \(-8/3\), \(-2/3\), and \(10/3\). Their squared deviations sum to:

\left(-\frac{8}{3}\right)^2+ \left(-\frac{2}{3}\right)^2+ \left(\frac{10}{3}\right)^2 =\frac{64+4+100}{9} =\frac{56}{3}.

There are \(n=3\) draws in each bootstrap sample, so:

\operatorname{Var}^*(\bar{X}^*) =\frac{56/3}{3^2} =\frac{56}{27} =2.0741, \qquad \operatorname{SE}^*(\bar{X}^*) =\sqrt{\frac{56}{27}} =1.4402

The estimate is \(4.6667\) thousand dollars and the exact conditional bootstrap standard error is \(1.4402\) thousand dollars. As a check, the bootstrap mean is \(14/3\): the expected value of each draw is the observed mean, so the expected average of three draws is also \(14/3\).

Common mistake. Using the usual sample variance gives \(s^2=(56/3)/2=28/3\). Substituting this directly into \(s/\sqrt{n}\) gives \(\sqrt{28/9}=1.7638\), not the exact conditional bootstrap standard error. The bootstrap calculation uses \((n-1)s^2/n^2=56/27\), giving \(1.4402\). The discrepancy occurs because these are different variance calculations, not because the bootstrap resamples were simulated inaccurately.

Limitation. Every replicate uses only \(2,4,8\). No bootstrap replicate can contain a loss above \(8\) thousand dollars, so this resampling calculation cannot represent unobserved larger losses.

Small Samples Produce Coarse Intervals

A percentile bootstrap interval uses quantiles of the bootstrap replicate estimates as its endpoints. In a small sample, those estimates can take only a limited set of values, and their probabilities can be uneven. Consequently, the interval may have coarse, asymmetric endpoints. It is important not to interpret a nominal level, such as 95%, as a guarantee that the procedure actually covers the population parameter 95% of the time.

Worked Example: A Percentile Interval From a Skewed Sample

A hypothetical severity sample, in dollars, is \(100,100,100,700\). Find the exact bootstrap standard error of the sample mean and the 2.5th and 97.5th percentile bootstrap quantiles. Use the inverse empirical distribution convention: select the first value whose cumulative probability is at least the stated percentile.

Method. Let \(K\) be the number of \(700\)-dollar observations in a bootstrap sample of size four. Each draw is \(700\) with probability \(1/4\), so \(K\sim\operatorname{Binomial}(4,1/4)\). A replicate mean is \(100+150K\).

The observed mean is \((100+100+100+700)/4=250\) dollars. The possible replicate means and their probabilities are:

\begin{array}{c|ccccc} K & 0 & 1 & 2 & 3 & 4\\ \hline 100+150K & 100 & 250 & 400 & 550 & 700\\ P^*(K) & 81/256 & 108/256 & 54/256 & 12/256 & 1/256 \end{array}

For example, \(P^*(K=2)=\binom{4}{2}(1/4)^2(3/4)^2=54/256\). The bootstrap variance of \(K\) is \(4(1/4)(3/4)=3/4\). Therefore:

\operatorname{SE}^*(\bar{X}^*) =150\sqrt{\frac{3}{4}} =129.90\text{ dollars}.

The cumulative probability at a replicate mean of \(100\) dollars is \(81/256=0.3164\), so the 2.5th percentile is \(100\). The cumulative probability through \(400\) dollars is \((81+108+54)/256=243/256=0.9492\). This is below \(0.975\); including the probability \(12/256\) at \(550\) dollars raises the cumulative probability to \(255/256=0.9961\). Thus the 97.5th percentile is \(550\), and this convention gives the percentile interval \([100,550]\) dollars.

Check. The probabilities sum to \((81+108+54+12+1)/256=1\), and the bootstrap mean is \(100+150(4/4)=250\) dollars, matching the observed mean.

Common mistake. Treating the endpoints as if the bootstrap distribution were continuous and smoothly spread between observed values would miss its actual discrete probabilities. The interval is determined by the specified quantile rule; its nominal percentile labels alone do not establish its repeated-sample coverage.

What the Bootstrap Cannot Learn About an Unseen Tail

The preceding examples show that small-sample limitations are not merely about rough numerical estimates. The empirical distribution gives zero probability to every value not in the data. If the population severity distribution has a substantial upper tail, a small observed sample may omit important losses, and the nonparametric bootstrap has no mechanism for generating them.

Worked Example: A Small Exponential Sample Misses Population Tail Risk

Suppose a hypothetical sample of three severities is \(1,2,9\) thousand dollars. For a diagnostic comparison, assume the losses were generated independently from an exponential distribution with mean \(4\) thousand dollars. Calculate the observed mean, its exact bootstrap standard error, the probability a bootstrap sample omits the observed maximum, and the population probability of a loss exceeding \(9\) thousand dollars.

Method. Use the empirical distribution for the bootstrap calculations and the stated exponential model only for the population tail probability. The sample mean is \((1+2+9)/3=4\) thousand dollars.

The squared deviations from \(4\) are \(9,4,25\), with sum \(38\). Hence:

\operatorname{Var}^*(\bar{X}^*)=\frac{38}{3^2}=\frac{38}{9}=4.2222, \qquad \operatorname{SE}^*(\bar{X}^*)=\sqrt{\frac{38}{9}}=2.0548

Each bootstrap draw selects one of the three observations with probability \(1/3\). The probability that all three draws omit the observed maximum \(9\) is therefore:

\left(\frac{2}{3}\right)^3=\frac{8}{27}=0.2963.

Under the assumed exponential model with mean \(4\), the rate is \(1/4\) per thousand dollars. Its survival function is \(P(X>x)=e^{-x/4}\), so:

P(X>9)=e^{-9/4}=0.1054.

The bootstrap probability of a loss greater than \(9\) is exactly zero because \(9\) is the largest available resampling value. Under the stated population model, that probability is \(0.1054\). This contrast illustrates why the bootstrap standard error is not a complete measure of uncertainty about the population tail.

Check. The bootstrap mean equals the observed mean \(4\), while the bootstrap support cannot exceed the observed maximum \(9\). These two facts are consistent: reproducing the sample mean does not ensure that the resampling distribution reproduces the population distribution.

Common mistake. A large number of bootstrap replicates cannot fix the missing-tail problem. It can estimate the conditional resampling distribution more accurately, but every replicate still draws from \(1,2,9\), not from the unobserved population.

Assessing Coverage With Repeated Samples

Coverage is a repeated-sample property. For an interval procedure intended to have nominal confidence level \(1-\alpha\), coverage is the probability that an interval constructed from a random sample contains the true parameter. A single dataset and its bootstrap interval cannot reveal this probability. When a data-generating model is specified, a simulation study can estimate coverage by generating many independent datasets, applying the complete interval procedure to each, and recording the fraction of intervals that contain the known parameter.

The estimated coverage also has Monte Carlo error because it is based on a finite number of simulated datasets. If \(C\) of \(M\) independently generated datasets produce intervals covering the parameter, the estimated coverage is \(\hat{p}=C/M\), with estimated Monte Carlo standard error \(\sqrt{\hat{p}(1-\hat{p})/M}\). This error describes simulation precision; it does not correct for a poor bootstrap procedure or a questionable data-generating model.

Worked Example: Interpreting a Small-Sample Coverage Audit

In a hypothetical audit, a specified severity model is used to generate \(1{,}000\) independent datasets, each of size \(8\). A 95% bootstrap interval for the mean severity contains the known model mean for \(870\) datasets. Calculate the observed coverage, its estimated Monte Carlo standard error, and assess whether the difference from 95% is plausibly explained by this Monte Carlo error alone.

Method. Divide the number of covering intervals by the number of datasets. Then use the binomial standard-error formula for the estimated coverage.

\hat{p}=\frac{870}{1000}=0.8700

The estimated Monte Carlo standard error is:

\sqrt{\frac{0.8700(1-0.8700)}{1000}} =\sqrt{0.0001131} =0.0106.

The absolute difference from the nominal coverage is \(0.9500-0.8700=0.0800\). This is about \(0.0800/0.0106=7.5\) estimated Monte Carlo standard errors. Thus, under the stated independent simulation setup, the discrepancy is much larger than would ordinarily be attributed to Monte Carlo error alone. It is evidence that this interval procedure may under-cover in the specified small-sample scenario; it does not, by itself, establish how it behaves under other severity models or sample sizes.

Check. Coverage is calculated across generated datasets, not across bootstrap replicates within one dataset. The \(1{,}000\) datasets assess repeated-sample performance; resamples within each dataset are part of constructing its interval.

Common mistake. Reporting that \(870\) of \(1{,}000\) bootstrap replicates covered the parameter would answer a different question. Coverage counts the intervals from independent datasets that include the true parameter.

Practical Checks for Small-Sample Results

When interpreting bootstrap output from a limited severity sample, ask what the resampling distribution can and cannot represent. The following checks help keep a numerical result in perspective:

Identify the conditional target. A bootstrap standard error describes variation under resampling from the observed empirical distribution.
Inspect the observed range. For nonparametric resampling, no replicate can exceed the sample maximum or fall below its minimum.
Check whether the replicate distribution is coarse. With few observations, a small number of possible replicate outcomes can strongly determine interval endpoints.
Separate precision from validity. More resamples can stabilize a computed bootstrap quantity, but cannot establish that the empirical distribution represents the population well.
Use a stated model to evaluate coverage. Coverage claims require a repeated-sample assessment under a defined scenario; one interval is not a coverage study.

These checks do not mean that every bootstrap result from a small sample is unusable. They mean that the result should be interpreted as evidence conditional on limited data, and that its adequacy for the actuarial estimand should be evaluated rather than assumed. The distinction is particularly important when severity is skewed and a few observations have substantial influence on the mean.

Key takeaway: In a small sample, the bootstrap can quantify resampling variability accurately while still understating uncertainty about the population. Its standard errors and intervals inherit the empirical sample’s limited support; repeated-sample coverage assessment is needed to evaluate interval performance under a specified model.

Check Your Understanding

For each problem, distinguish calculations conditional on the observed sample from conclusions about a population model.

A hypothetical sample of severities, in thousands of dollars, is \(3,7\). Calculate the exact nonparametric bootstrap standard error of the sample mean. (Answer: \(\sqrt{2}=1.4142\) thousand dollars)
A sample of \(2,2,8\) thousand dollars is resampled \(3\) times per bootstrap replicate. Calculate the exact bootstrap mean and standard error. (Answer: Mean \(4.0000\) thousand dollars; standard error \(\sqrt{8/3}=1.6330\) thousand dollars)
A sample consists of \(50,50,250\) dollars. Let \(K\) count the number of \(250\)-dollar observations in a size-three bootstrap sample. Express the replicate mean in terms of \(K\) and calculate the bootstrap standard error. (Answer: \(50+(200/3)K\); standard error \(54.4331\) dollars)
Four independent exponential severities have mean \(5\) thousand dollars. In one observed sample the maximum is \(10\) thousand dollars. What is the nonparametric bootstrap probability that a size-four resample omits that maximum, and what is the model probability of a severity exceeding \(10\) thousand dollars? (Answer: Bootstrap omission probability \(0.3164\); model exceedance probability \(e^{-2}=0.1353\))
A hypothetical coverage audit produces \(920\) covering intervals from \(1{,}000\) independent datasets. Calculate the estimated coverage and its Monte Carlo standard error. How many estimated standard errors is it below nominal 95% coverage? (Answer: Coverage \(0.9200\); standard error \(0.0086\); approximately \(3.50\) standard errors below nominal)

What You'll Learn