Why Diagnose the Bootstrap Distribution?
A bootstrap standard error compresses many resampled estimates into one number, and an interval compresses them into two endpoints. Those summaries are useful only if they reflect the shape of the replicate distribution. A long tail, strong skewness, or a few unusually extreme replicates can make a symmetric summary misleading or make the reported standard error unusually sensitive to particular resamples.
Here, a bootstrap replicate is an estimate \(\hat{\theta}^{*(b)}\) calculated from the \(b\)th resampled data set, for \(b=1,\ldots,B\). The empirical distribution of the \(B\) replicate estimates is a diagnostic picture of how the estimation procedure behaves under resampling the observed data. It is not a guarantee that the observed data or the resampling scheme captures every source of uncertainty.
A histogram or dot plot helps reveal features that a standard error does not show. Quantiles describe where the distribution lies, while skewness summarizes one aspect of its asymmetry. Extreme replicates deserve investigation because they can substantially affect a mean or standard deviation. They should not be removed merely because a diagnostic rule labels them extreme.
Shape, Quantiles, and a Simple Outlier Rule
For a concise numerical description of skewness, define the empirical central moments \(m_r=B^{-1}\sum_{b=1}^B(\hat{\theta}^{*(b)}-\bar{\theta}^*)^r\). When \(m_2>0\), the moment coefficient of skewness is \(g_1=m_3/m_2^{3/2}\). A positive value indicates a longer or heavier right side in the third-moment sense; a negative value indicates a longer or heavier left side. A value near zero does not prove that the distribution is symmetric, so inspect the plot as well.
For quartiles, a convention must be specified, particularly when \(B\) is small. In the worked examples below, use the nearest-rank convention: the empirical \(p\)-quantile is the ordered replicate at rank \(\lceil pB\rceil\), with ranks starting at 1. Other software may interpolate between ordered values, so numerical quantiles can differ slightly across conventions.
The interquartile range is \(\mathrm{IQR}=Q_3-Q_1\). A common descriptive rule flags values below \(Q_1-1.5\mathrm{IQR}\) or above \(Q_3+1.5\mathrm{IQR}\) for inspection. This is a screening rule, not a test that a replicate is erroneous. In a skewed bootstrap distribution, a valid replicate in a long tail may cross a fence.
Worked Example: Correcting for a Single Large Replicate
A hypothetical bootstrap analysis of a claim-cost functional gives five replicate estimates, in thousands of dollars: \(1,1,1,1,5\). Calculate the bootstrap mean, moment skewness, and sample standard error. Then use the nearest-rank quartiles to check for a flagged high replicate. State what the diagnostics suggest, without deleting any replicate.
Method. Calculate the mean and central moments from all five values. Use the sample standard deviation for the bootstrap standard error and the stated nearest-rank convention for the quartiles.
Mean and moments. The mean is \((1+1+1+1+5)/5=9/5=1.8\). The four deviations from the mean are \(-0.8\), and the fifth is \(3.2\). Thus
and
Therefore \(g_1=6.144/(2.56)^{3/2}=6.144/4.096=1.5\), indicating positive, or right, skewness.
Standard error. The sum of squared deviations is \(5m_2=12.8\). The sample variance is \(12.8/(5-1)=3.2\), so \(s^*=\sqrt{3.2}=1.789\) thousand dollars, rounded to three decimals.
Quartiles and check. The ordered values are \(1,1,1,1,5\). The nearest-rank \(Q_1\) is at rank \(\lceil0.25(5)\rceil=2\), so \(Q_1=1\); \(Q_3\) is at rank \(\lceil0.75(5)\rceil=4\), so \(Q_3=1\). The IQR is zero and the upper fence is 1. The value 5 is flagged by this rule. The large replicate strongly affects the mean and standard error, but these five values alone do not establish that it is invalid. The calculation avoids the common error of reporting skewness as 1.2: the central moments give \(g_1=1.5\).
Why Skewness Matters for Interval Summaries
A symmetric, mean-based summary places equal distances on either side of its center. For example, a normal-approximation interval has the form \(\bar{\theta}^*\pm z s^*\), where \(z\) is a chosen standard-normal quantile. Its equal-tail structure is a reasonable description only when a symmetric approximation is appropriate. If the bootstrap distribution has a long right tail, the upper and lower tails need not be represented equally by such an interval.
The reason is visible from the signed deviations. The mean and variance use first and second powers of deviations, while the third central moment preserves their signs. A few large positive deviations can therefore make \(m_3\) positive and large, even when most replicates lie close to the center. In the same situation, squared deviations can also inflate the standard error. Quantiles add useful information because they show where specified fractions of the replicates fall without relying on the mean and variance alone.
Check for asymmetry, multiple clusters, gaps, and isolated values. A plot often reveals patterns that a single skewness coefficient cannot summarize.
Calculate the mean, standard error, median, and relevant quantiles. A noticeable difference between the mean and median can be a sign of asymmetry or influential values.
Check the resampled data and the estimate calculation. Determine whether the value is a valid consequence of resampling or a coding or data-handling error.
If the distribution is markedly asymmetric, report that feature and examine quantile-based summaries alongside any symmetric summary. Do not treat a diagnostic flag as permission to discard a valid replicate.
Worked Example: Contrasting Symmetric and Right-Skewed Replicates
Two hypothetical bootstrap analyses estimate a payment rate. Their replicates are: set A, \(8,9,9,10,10,11,11,12\); set B, \(9,9,9,9,10,10,10,14\). For each set, calculate the mean, \(m_2\), \(m_3\), and \(g_1\). Then compare the nearest-rank 10th and 90th percentiles. Values are in percentage points.
Set A. The sum is 80, so the mean is \(80/8=10\). The deviations are \(-2,-1,-1,0,0,1,1,2\). Their squared deviations sum to 12, giving \(m_2=12/8=1.5\). Their cubed deviations cancel, so \(m_3=0\) and \(g_1=0/(1.5)^{3/2}=0\).
The ordered set A has 8 as its first value and 12 as its eighth. The nearest-rank 10th percentile is rank \(\lceil0.10(8)\rceil=1\), or 8; the 90th percentile is rank \(\lceil0.90(8)\rceil=8\), or 12.
Set B. Its sum is \(4(9)+3(10)+14=80\), so its mean is also 10. The deviations are \(-1,-1,-1,-1,0,0,0,4\). The squared deviations sum to \(4+16=20\), giving \(m_2=20/8=2.5\). The cubed deviations sum to \(-4+64=60\), giving \(m_3=60/8=7.5\). Hence
The nearest-rank 10th and 90th percentiles are 9 (rank 1) and 14 (rank 8). Set B has a longer upper side and a positive skewness coefficient, while set A is symmetric about 10. Set B’s 14 is important to describing that asymmetry; replacing the distribution with a symmetric summary centered at 10 would conceal its upper tail. These small sets illustrate the calculations, not reliable interval estimation from only eight replicates.
Extreme Replicates: Flag, Then Investigate
An extreme replicate may arise because a resample contains an unusual combination of observations, particularly for nonlinear estimates or statistics sensitive to large losses. It may also arise from a mistake, such as applying a transformation inconsistently or using the wrong denominator. The appropriate response is to inspect the replicate and the calculation, not to remove it automatically.
A useful sensitivity check is to compare the mean and standard error with and without the flagged value, while clearly labeling the second calculation as a diagnostic only. A large change signals that the summary depends heavily on that replicate. It does not establish that the original calculation is wrong or that exclusion is justified. If the value is valid, the primary description should retain it and explain the resulting shape or sensitivity.
Worked Example: A Flagged Loss-Ratio Replicate
In a hypothetical bootstrap of an insurer’s loss-ratio estimator, ten replicate values, expressed as percentages, are \(48,49,49,50,50,50,51,51,52,80\). Using nearest-rank quartiles, calculate the mean, median, IQR fences, and sample standard error. Then calculate the mean after omitting 80 solely as a sensitivity check, and interpret the difference.
Center and quartiles. The sum of all ten values is 530, so the mean is \(530/10=53\%\). The middle two ordered values are both 50, so the median is 50%. The nearest-rank \(Q_1\) is rank \(\lceil0.25(10)\rceil=3\), giving 49%; \(Q_3\) is rank \(\lceil0.75(10)\rceil=8\), giving 51%. Thus the IQR is \(51-49=2\) percentage points. The fences are \(49-1.5(2)=46\%\) and \(51+1.5(2)=54\%\). The replicate 80% is above the upper fence.
Standard error. The deviations from the mean 53 are \(-5,-4,-4,-3,-3,-3,-2,-2,-1,27\). Their squares sum to \(25+16+16+9+9+9+4+4+1+729=822\). The sample variance is \(822/9=91.333\), so the sample standard error is \(\sqrt{91.333}=9.557\) percentage points, rounded to three decimals.
Sensitivity check. The first nine values sum to \(530-80=450\), so their mean is \(450/9=50\%\). Omitting the flagged replicate lowers the mean from 53% to 50%, a three-percentage-point change. That is evidence that the mean is sensitive to the high value. The replicate should be checked against its resampled data and calculation; unless it is shown to be invalid, the diagnostic does not justify excluding it.
Common Mistakes and Practical Interpretation
- Calling every flagged replicate an error. The IQR rule identifies a value for review; it does not determine whether the value is valid.
- Assuming a small standard error means a well-behaved distribution. A single standard deviation cannot show whether the distribution is skewed or has distinct clusters.
- Reporting skewness without checking the arithmetic. Compute the mean first, then the signed third central moment and the second central moment. In particular, use \(m_3/m_2^{3/2}\), not \(m_3/m_2\).
- Comparing quantiles calculated under different conventions. State the convention or use the same software and setting for all comparisons.
- Interpreting asymmetry as proof of failure. Skewness is a diagnostic feature. It signals that a symmetric summary may be unrepresentative; it does not by itself establish that the bootstrap procedure or estimate is invalid.
In practice, report enough information for a reader to see what the standard error alone omits: the number and spread of replicates, a plot or shape description, selected quantiles, and any influential values that were investigated. When the distribution is strongly asymmetric, make that limitation explicit when presenting a symmetric interval summary. The aim is not to force the replicates into a preferred shape, but to describe faithfully what the resampling calculation produced.
These diagnostics are aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus. The central exam skill is to connect numerical summaries to the empirical distribution: recognize when a mean or standard error is sensitive to extremes, distinguish skewness from symmetric spread, and interpret flagged replicates cautiously.
Check Your Understanding
Use the stated nearest-rank convention where quartiles are requested. Treat an outlier flag as a prompt for investigation, not automatic deletion.
- A bootstrap produces \(2,2,2,2,10\). Calculate its mean and moment skewness, and apply the IQR upper fence. (Answer: Mean \(=3.6\), \(g_1=1.5\), and 10 is above the upper fence of 2.)
- For replicates \(4,5,5,6,6,7\), calculate the mean, moment skewness, and sample standard error. Are the replicates symmetric about their mean? (Answer: Mean \(=5.5\), \(g_1=0\), sample standard error \(=\sqrt{1.1}\approx1.049\), and the replicates are symmetric about 5.5.)
- A sorted set of seven replicate estimates is \(20,21,21,22,22,23,35\). Find its mean, median, IQR fences, and whether 35 is flagged. (Answer: Mean \(=164/7\approx23.429\), median \(=22\), fences \(18\) and \(26\), and 35 is flagged above the upper fence.)
- Eight sorted bootstrap estimates of a rate are \(0.08,0.09,0.09,0.10,0.10,0.11,0.12,0.21\). Calculate the mean, median, nearest-rank \(Q_1\) and \(Q_3\), and upper fence. What does the flag suggest? (Answer: Mean \(=0.1125\), median \(=0.10\), \(Q_1=0.09\), \(Q_3=0.11\), upper fence \(=0.14\); 0.21 is flagged, suggesting a high replicate to investigate and possible right-tail influence.)
- A valid high replicate is found to account for much of a bootstrap standard error. Should it be discarded to obtain a more representative standard error? Explain the appropriate diagnostic response. (Answer: No. Retain a valid replicate; inspect its data and calculation, report the distribution’s asymmetry or sensitivity, and do not treat the IQR flag alone as justification for deletion.)