Why Compare Bootstrap and Jackknife?
The bootstrap and the jackknife are resampling methods for assessing how a statistic might vary across repeated samples. Both begin with the observed data, but they create different sets of perturbed data: the bootstrap samples observations with replacement, while the delete-one jackknife leaves out one observation at a time. Their standard error and bias estimates can be similar for some statistics, but they are not interchangeable.
The distinction matters when interpreting uncertainty. A bootstrap distribution can reflect the empirical distribution’s repeated-sampling behavior, whereas the jackknife summarizes how sensitive the statistic is to removing individual observations. That sensitivity can be informative for a smooth statistic such as a mean and misleading for a statistic that changes abruptly, such as a median or maximum.
These standard forms of the methods presume independent observations from a common population. For dependent data, ordinary resampling of individual observations does not preserve the dependence structure; a dependence-aware method is needed instead. The results below compare the ordinary bootstrap and delete-one jackknife under their independent-sample setup.
What the Two Methods Estimate
For \(B\) bootstrap replicates, write the replicate statistics as \(\hat\theta_1^*,\ldots,\hat\theta_B^*\), and let \(\bar{\theta}^*\) be their average. The usual simulation estimates of bootstrap bias and standard error are:
The bias convention here is “resampling estimate minus original estimate.” The standard error is the sample standard deviation of the bootstrap replicate statistics. If all possible resamples are enumerated, the bootstrap variance is calculated as a variance under that resampling distribution; if only \(B\) random replicates are used, the displayed formula estimates it with a denominator of \(B-1\).
For the jackknife, first calculate the average of the delete-one estimates, \(\bar{\theta}_{(-)}=n^{-1}\sum_{i=1}^n\hat\theta_{(-i)}\). The standard delete-one jackknife estimates are:
The bias estimate is an approximation based on how the statistic changes when one observation is removed. Bias correction is not automatically an improvement: it can increase error when the approximation is poor or the estimate is already unstable.
Deriving the Jackknife Formula for a Mean
Suppose the statistic is the sample mean \(\bar X=n^{-1}\sum_{j=1}^n X_j\), with \(n>1\). Removing \(X_i\) gives:
Averaging these \(n\) delete-one means gives \(\bar X_{(-)}=\bar X\). Therefore the deviation of the \(i\)th delete-one mean from their average is:
Substitute this difference into the jackknife variance formula. If \(s^2=(n-1)^{-1}\sum_{i=1}^n(X_i-\bar X)^2\) is the sample variance, the result is:
This is the usual estimated variance of a sample mean under independent, identically distributed sampling with finite variance. Also, since \(\bar X_{(-)}=\bar X\), the delete-one jackknife bias estimate for the mean is zero. The derivation explains why the jackknife works especially neatly for a mean: deleting an observation changes it by an exact, linear amount.
Worked Example: Exact Bootstrap and Jackknife for a Mean
A hypothetical analyst records two independent claim-processing times, in days: 2 and 6. Use the exact size-two bootstrap distribution to find the bootstrap standard error and bias of the sample mean. Then find the delete-one jackknife standard error and bias. The method is exact enumeration, not a random simulation; a common mistake is to treat the bootstrap variance and jackknife variance as necessarily identical.
Original estimate. The observed mean is \(\hat\theta=(2+6)/2=4\) days. The four ordered bootstrap samples, each with probability \(1/4\), are \((2,2),(2,6),(6,2),(6,6)\). Their means are 2, 4, 4, and 6 days.
Bootstrap bias. The mean of the four replicate means is \(\bar\theta^*=(2+4+4+6)/4=4\). Thus the bootstrap bias estimate is \(4-4=0\) days.
Bootstrap standard error. The exact bootstrap variance is the average squared deviation from 4: \(\{(2-4)^2+(4-4)^2+(4-4)^2+(6-4)^2\}/4=8/4=2\) days squared. The bootstrap standard error is \(\sqrt{2}=1.4142\) days, rounded to four decimals.
Jackknife. Deleting the first observation leaves a mean of 6; deleting the second leaves a mean of 2. Their average is 4, so the jackknife bias estimate is \((2-1)(4-4)=0\) days. The jackknife variance is \((1/2)\{(6-4)^2+(2-4)^2\}=4\) days squared, giving a standard error of 2 days.
Check. Both methods estimate zero bias here, but their standard errors differ: 1.4142 days for the exact bootstrap and 2 days for the jackknife. The jackknife result also equals \(s/\sqrt{n}\), since the sample variance is \(s^2=8\) and \(s^2/n=8/2=4\).
Where the Methods Can Disagree
For a sample mean, the bootstrap variance under the empirical distribution is \((n-1)s^2/n^2\), while the delete-one jackknife variance is \(s^2/n\). These differ by a factor of \((n-1)/n\); they become closer as \(n\) grows, but need not agree in a small sample. For statistics that are not smooth functions of the observations, the difference can be more pronounced: deleting one value may produce only a few distinct jackknife estimates, while resampling can produce a broader and irregular distribution.
A statistic is often called smooth, informally, when small changes to the data produce controlled, gradual changes in the statistic. The mean is smooth in this sense. Quantiles and extremes can be nonsmooth: the identity of the observation determining the statistic may change abruptly. Jackknife bias and variance approximations are not guaranteed to be reliable for such statistics.
Worked Example: Comparing Methods for a Sample Median
A hypothetical sample of three independent repair durations, in hours, is \(1,2,9\). Compare the exact bootstrap and delete-one jackknife bias and standard error for the median. This example uses all \(3^3=27\) ordered bootstrap samples. The trap is assuming that a jackknife bias correction is automatically credible just because it can be calculated.
Original median. The sample median is \(\hat\theta=2\) hours.
Exact bootstrap distribution. The bootstrap median is 1 if at least two of the three draws are 1. There is 1 all-1 sample and \(3\cdot2=6\) samples with exactly two 1s, giving 7 samples. By the same count, 7 samples have median 9. The remaining \(27-7-7=13\) samples have median 2. Therefore the bootstrap mean median is \((7\cdot1+13\cdot2+7\cdot9)/27=96/27=32/9\) hours.
Bootstrap bias and standard error. The bias estimate is \(32/9-2=14/9=1.5556\) hours, rounded to four decimals. The bootstrap second moment is \(\{7(1^2)+13(2^2)+7(9^2)\}/27=626/27\). Thus the variance is \(626/27-(32/9)^2=854/81\) hours squared, and the standard error is \(\sqrt{854/81}=3.2463\) hours, rounded to four decimals.
Delete-one jackknife. Omitting 1 gives the median \((2+9)/2=5.5\); omitting 2 gives \((1+9)/2=5\); omitting 9 gives \((1+2)/2=1.5\). Their average is \((5.5+5+1.5)/3=4\) hours. The jackknife bias estimate is \(2(4-2)=4\) hours, and the bias-corrected estimate is \(2-4=-2\) hours.
Jackknife standard error. The squared deviations of the leave-one medians from 4 sum to \(1.5^2+1^2+(-2.5)^2=9.5\). The jackknife variance is \((2/3)(9.5)=19/3\) hours squared, so the standard error is \(\sqrt{19/3}=2.5166\) hours, rounded to four decimals.
Check and interpretation. The methods give different bias estimates, and the jackknife correction produces an implausible negative median duration. The calculation is valid as a jackknife calculation, but the median’s abrupt changes make that bias approximation untrustworthy in this tiny sample. The result is a warning about the method, not a reason to interpret a negative duration as meaningful.
Worked Example: Comparing Methods for a Sample Maximum
A hypothetical independent sample of four equipment lifetimes, in months, is \(2,4,7,10\). Compare the exact bootstrap and delete-one jackknife estimates for the maximum. This example demonstrates that different perturbations of the sample can yield different uncertainty estimates for a statistic determined by an extreme observation.
Original estimate and jackknife. The sample maximum is \(\hat\theta=10\). The leave-one maxima are 7, 10, 10, and 10, with average \(37/4=9.25\). The jackknife bias estimate is \(3(9.25-10)=-2.25\) months. The squared deviations from 9.25 sum to \((-2.25)^2+3(0.75)^2=6.75\). The jackknife variance is \((3/4)(6.75)=5.0625\) months squared, giving a standard error of 2.25 months.
Exact bootstrap distribution. There are \(4^4=256\) equally likely ordered size-four resamples. The bootstrap maximum is 2 for 1 sample, 4 for \(2^4-1^4=15\) samples, 7 for \(3^4-2^4=65\) samples, and 10 for \(256-3^4=175\) samples. The counts sum to \(1+15+65+175=256\).
Bootstrap bias and standard error. The bootstrap mean maximum is \((2+15\cdot4+65\cdot7+175\cdot10)/256=2267/256=8.8555\) months. The bias estimate is \(8.8555-10=-1.1445\) months, rounded to four decimals. The second moment is \((2^2+15\cdot4^2+65\cdot7^2+175\cdot10^2)/256=20929/256\). Subtracting \((2267/256)^2\) gives a variance of \(218535/65536=3.3346\) months squared and a standard error of 1.8261 months, rounded to four decimals.
Check. The bootstrap and jackknife bias estimates are both negative, but their magnitudes differ. Their standard errors also differ: 1.8261 months and 2.25 months, respectively. Neither calculation implies that the maximum’s uncertainty is known exactly; both are estimates conditional on the observed sample.
Choosing and Interpreting a Method
The bootstrap is flexible because it creates many resamples and can display features such as skewness or multiple modes in a statistic’s resampling distribution. Its accuracy still depends on whether the empirical distribution represents the population features relevant to the statistic. In particular, an observed sample may contain little information about unobserved extremes. Increasing the number of bootstrap replicates reduces simulation noise, but it cannot supply information absent from the original data.
The jackknife is computationally straightforward when the statistic can be recomputed after each deletion, and its formula is especially useful for smooth statistics. But it uses only \(n\) delete-one perturbations; it is not a substitute for a bootstrap distribution. For a nonsmooth statistic, those \(n\) perturbations can miss important aspects of resampling variation or yield an unreliable bias approximation.
| Question | Bootstrap | Delete-one jackknife |
|---|---|---|
| How are perturbed data sets made? | Draw \(n\) observations with replacement from the observed data. | Remove each observation once, producing \(n\) samples of size \(n-1\). |
| What is the standard error based on? | Variation among bootstrap replicate statistics. | Variation among delete-one statistics, scaled by \((n-1)/n\). |
| Where is it particularly convenient? | Exploring a resampling distribution for many statistics. | Assessing sensitivity of smooth statistics, including the mean. |
| What should prompt caution? | Dependence, unobserved extremes, or a statistic poorly represented by the empirical sample. | Nonsmooth statistics or an unstable bias correction. |
A disciplined comparison begins by stating the sampling assumptions, calculating each method with its own formula, and then interpreting discrepancies in light of the statistic. For smooth statistics, close results can be reassuring, but agreement does not prove that the assumptions hold. For medians, maxima, and other nonsmooth statistics, disagreement is a diagnostic to investigate rather than an arithmetic error to remove. These distinctions are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, identify the method and apply its estimate carefully.
- A sample of four independent values is \(1,3,5,7\). For the sample mean, find the exact empirical-bootstrap standard error and delete-one jackknife standard error. (Answer: Bootstrap SE \(=\sqrt{1.25}=1.1180\); jackknife SE \(=\sqrt{5/3}=1.2910\).)
- A sample of five independent observations has mean 6 and sum of squared deviations from the mean equal to 40. Find the delete-one jackknife bias estimate and standard error for the mean. (Answer: Bias estimate \(=0\); jackknife variance \(=2\), so SE \(=\sqrt{2}=1.4142\).)
- For the sample \(1,3,4,8,20\), the statistic is the median. The five delete-one medians are 6, 6, 5.5, 3.5, and 3.5. Find the jackknife bias estimate and bias-corrected estimate. (Answer: Their average is 4.9; bias estimate \(=4(4.9-4)=3.6\); corrected estimate \(=0.4\).)
- For the sample \(1,2,4\), find the delete-one jackknife standard error and bias estimate for the maximum. (Answer: Leave-one maxima are 4, 4, and 2; jackknife variance \(=16/9\), SE \(=4/3=1.3333\), and bias estimate \(=2(10/3-4)=-4/3\).)
- A statistic is calculated from a time-ordered sequence with serial dependence. Should ordinary resampling of individual observations be used as if the data were independent? (Answer: No; individual resampling does not preserve the dependence structure, so use a method appropriate for dependent observations.)