Bootstrap Bias And Bias-Corrected Point Estimates

Why Estimate Bias?

An estimator can be systematically high or low in repeated samples, even when it is calculated correctly each time. The bootstrap gives a way to estimate that bias using the observed data: repeatedly resample the observations, recalculate the estimator, and compare the average of those bootstrap estimates with the estimate from the original sample.

This comparison produces a bias-corrected point estimate. It is an adjustment based on the empirical bootstrap distribution, not proof that the adjusted value is closer to the unknown quantity. In particular, bias and variability are different features of an estimator, and correcting estimated bias can increase variability.

Definition: Let $X_1,\ldots,X_n$ be independent observations from a population, and let $\hat{\theta}=T(X_1,\ldots,X_n)$ estimate a quantity $\theta$. In a nonparametric bootstrap, a bootstrap sample of size $n$ is drawn with replacement from the observed values. If $\hat{\theta}^{*(b)}$ is the estimate from bootstrap replicate $b$, the estimated bootstrap bias based on $B$ replicates is $$ \widehat{\operatorname{Bias}}_{\mathrm{boot}}(\hat{\theta}) = \frac{1}{B}\sum_{b=1}^{B}\hat{\theta}^{*(b)}-\hat{\theta}. $$ The corresponding bias-corrected point estimate is $$ \hat{\theta}_{\mathrm{bc}} = \hat{\theta}-\widehat{\operatorname{Bias}}_{\mathrm{boot}}(\hat{\theta}). $$

The first quantity is the bootstrap average minus the observed estimate; the correction subtracts that estimated bias from the observed estimate. Consequently, the corrected estimate can also be written as $2\hat{\theta}-\frac{1}{B}\sum_{b=1}^{B}\hat{\theta}^{*(b)}$. If the bootstrap average is above the observed estimate, the estimated bias is positive and the correction moves the estimate down. If the bootstrap average is below it, the estimated bias is negative and the correction moves the estimate up.

With a finite number $B$ of replicates, the computed bootstrap average has Monte Carlo error. Increasing $B$ can reduce that simulation error, but it does not remove the limitations of using the observed sample as a stand-in for the population. When all possible resamples are considered, the average is the exact mean under the empirical bootstrap distribution; with a finite number of resamples, it estimates that mean.

Calculation Process

The calculation is easiest to audit when the original estimate, bootstrap average, bias, and corrected estimate are kept as distinct steps. Use the same estimator and the same statistic scale in the original sample and in every replicate.

Calculate the observed estimate.
Apply the specified estimator to the original data to obtain $\hat{\theta}$.

Resample and recalculate.
Draw samples of size $n$ with replacement from the observed data, and calculate the estimator for each sample.

Find the bootstrap average and estimated bias.
Average the replicate estimates, then subtract the original estimate: bootstrap average minus $\hat{\theta}$.

Correct the point estimate.
Subtract the estimated bias from $\hat{\theta}$, retaining the sign of the bias.

Worked Examples

Worked Example: Bootstrap Bias of a Sample Median

A hypothetical claim portfolio has three observed claim amounts, measured in thousands of dollars: $1,2,9$. The estimator of interest is the sample median, and each bootstrap sample contains three draws with replacement. Find the exact empirical-bootstrap bias and the bias-corrected estimate.

Method. Enumerate the $3^3=27$ equally likely ordered bootstrap samples by their resulting median. The observed median is $2$.

A bootstrap median is $1$ when at least two of the three draws equal $1$. There are $\binom{3}{2}(2)+1=7$ such samples: choose which two positions are $1$, fill the remaining position with either other value, or use three $1$s. Thus $P^*(\text{median}=1)=7/27$, where $P^*$ denotes probability under resampling from the observed data. By the same count, $P^*(\text{median}=9)=7/27$. The remaining $27-7-7=13$ samples have median $2$.

The exact bootstrap average is therefore $$ \frac{7(1)+13(2)+7(9)}{27} = \frac{7+26+63}{27} = \frac{96}{27} = \frac{32}{9} = 3.5556 $$ in thousands of dollars. The estimated bias is $$ \frac{32}{9}-2=\frac{14}{9}=1.5556 $$ in thousands of dollars. Hence the corrected estimate is $$ 2-\frac{14}{9}=\frac{4}{9}=0.4444 $$ in thousands of dollars. The correction is internally consistent: the bootstrap average exceeds the observed median, so subtracting the positive estimated bias moves the point estimate down. Values are rounded to four decimal places only at the final step.

Common mistake avoided: The bootstrap average is not itself the corrected estimate. It estimates the bootstrap center; the correction is the observed median minus the difference between that center and the observed median.

Worked Example: Correcting the Usual Sample Variance

A hypothetical set of annual claim amounts, in thousands of dollars, is $2,4,10$. Use the usual sample variance, with denominator $n-1$, as the estimator. Find its exact bootstrap bias and corrected point estimate for the variance.

Method. First calculate the observed sample variance. Then use the expected sample variance of a bootstrap sample drawn from the empirical distribution. The observations have mean $$ \bar{x}=\frac{2+4+10}{3}=\frac{16}{3}. $$ Their sum of squared deviations is $$ \left(2-\frac{16}{3}\right)^2+ \left(4-\frac{16}{3}\right)^2+ \left(10-\frac{16}{3}\right)^2 = \frac{100+16+196}{9} = \frac{104}{3}. $$ Thus the observed sample variance is $$ s^2=\frac{104/3}{3-1}=\frac{52}{3}. $$

To obtain the bootstrap average, let $Y_1,Y_2,Y_3$ be the three draws from the empirical distribution on $2,4,10$. One draw has variance $$ \operatorname{Var}^*(Y_1) = \frac{1}{3}\left[ \left(2-\frac{16}{3}\right)^2+ \left(4-\frac{16}{3}\right)^2+ \left(10-\frac{16}{3}\right)^2 \right] = \frac{104}{9}. $$ For three independent draws from any distribution with finite variance $\sigma^2$, the expected sample variance with denominator $n-1$ is $\sigma^2$. To see why, use the identity $\sum_{j=1}^{n}(Y_j-\bar{Y})^2=\sum_{j=1}^{n}(Y_j-\mu)^2-n(\bar{Y}-\mu)^2$, where $\mu=E[Y_j]$. Taking expectations gives $n\sigma^2-n(\sigma^2/n)=(n-1)\sigma^2$; division by $n-1$ gives $\sigma^2$. Here the bootstrap draws have variance $104/9$, so the exact bootstrap average of their sample variances is $104/9$.

The estimated bootstrap bias and correction are $$ \widehat{\operatorname{Bias}}_{\mathrm{boot}}(s^2) = \frac{104}{9}-\frac{52}{3} = -\frac{52}{9}, \qquad s^2_{\mathrm{bc}} = \frac{52}{3}-\left(-\frac{52}{9}\right) = \frac{208}{9} = 23.1111. $$ Both variance estimates are in squared thousands of dollars. The negative bias means that the bootstrap average is below the observed sample variance, so the correction increases the estimate. Values are rounded to four decimal places only at the final step.

Common mistake avoided: The bootstrap variance here is the average of replicate sample variances, not the variance of the replicate estimates. Those are different calculations.

Worked Example: Correcting a Sample Maximum

A hypothetical sample of three claim amounts, in thousands of dollars, is $2,5,8$. Use the sample maximum as an estimator of a population upper endpoint. Each bootstrap sample has size three. Find the bootstrap bias and corrected point estimate.

Method. The observed maximum is $\hat{\theta}=8$. In an ordered bootstrap sample, the maximum is $2$ only if all three draws are $2$, which has probability $1/27$. The maximum is $5$ if every draw is either $2$ or $5$, but not all are $2$, which has probability $$ \left(\frac{2}{3}\right)^3-\left(\frac{1}{3}\right)^3 = \frac{8-1}{27} = \frac{7}{27}. $$ The remaining probability, $1-1/27-7/27=19/27$, gives a maximum of $8$.

The exact bootstrap average is $$ \frac{1(2)+7(5)+19(8)}{27} = \frac{2+35+152}{27} = \frac{189}{27} = 7. $$ Therefore the estimated bias is $7-8=-1$ thousand dollars, and the corrected estimate is $$ 8-(-1)=9 $$ thousand dollars. The arithmetic checks: a negative estimated bias calls for an upward adjustment. Values are exact here, so no rounding is needed.

Common mistake avoided: Reversing the bias subtraction would give $8-7=1$, then incorrectly produce $8-1=7$. Bias is bootstrap average minus observed estimate, so the correct bias is $-1$, not $1$.

What the Correction Does—and Does Not Do

The bootstrap bias estimate describes how the estimator behaves when samples are drawn from the empirical distribution represented by the observed data. It is not generally the true repeated-sampling bias under the unknown population. The bootstrap resamples cannot introduce values absent from the original data, a limitation that can matter for statistics such as a sample maximum. In the maximum example, the correction moves beyond the largest observed claim; that is permitted algebraically, but it should be interpreted cautiously because the resampling data contained no larger values.

A bias correction also does not guarantee lower mean squared error. Mean squared error depends on both squared bias and variance. The adjustment may reduce an estimated bias while increasing variability, especially if the bootstrap average is unstable or the estimator is highly sensitive to individual observations. A corrected value can also violate a natural constraint, such as nonnegativity, so check whether the estimator and target have restrictions before reporting the result.

For an estimator such as a sample mean, the nonparametric bootstrap average is exactly the observed sample mean: each bootstrap draw has conditional mean $\bar{x}$, so the mean of $n$ draws also has conditional mean $\bar{x}$. Its exact bootstrap bias is therefore zero. A zero estimated bias is a valid outcome; it does not imply that the estimate has no sampling uncertainty.

Key takeaway: Calculate bootstrap bias as bootstrap average minus observed estimate, then subtract that signed bias from the observed estimate. State whether the bootstrap average is exact or based on a finite number of replicates, and interpret the correction as a data-based adjustment rather than a guarantee of improved accuracy.

Check Your Understanding

For each problem, use the exact empirical bootstrap unless a finite number of replicates is specified.

A sample is $4,6,8,10$, and the estimator is the sample mean. What are the exact bootstrap bias and corrected estimate? (Answer: bias $0$; corrected estimate $7$.)
A sample is $1,4$, and the estimator is the sample maximum. Bootstrap samples have size two. Find the bootstrap average, bias, and corrected estimate. (Answer: bootstrap average $13/4=3.25$; bias $-0.75$; corrected estimate $4.75$.)
A sample is $2,6$, and the estimator is the median, defined for two observations as their average. Find the exact bootstrap bias and corrected estimate. (Answer: bootstrap average $4$; bias $0$; corrected estimate $4$.)
A sample is $3,9$, and the estimator is the sample variance with denominator $n-1$. Find the exact bootstrap bias and corrected estimate. (Answer: observed variance $18$; bootstrap average $9$; bias $-9$; corrected estimate $27$.)
A sample is $2,5,11$, and the estimator is the sample minimum. Bootstrap samples have size three. Find the bootstrap average, bias, and corrected estimate. (Answer: bootstrap average $28/9$; bias $10/9$; corrected estimate $8/9$.)

What You'll Learn