Why Reporting Is Part of the Analysis
An uncertainty calculation is useful only if a reader can tell what quantity was estimated, what procedure produced the uncertainty summary, and which sources of uncertainty it includes. In actuarial work, a bootstrap interval based on observed claims and a simulation error bound from a fixed model answer different questions. Reporting either one as a general measure of “uncertainty” can lead to a materially misleading conclusion.
A clear report identifies the point estimate, the uncertainty measure, the procedure used to obtain it, and the assumptions that support the procedure. It also distinguishes the original estimate from summaries of the bootstrap replicates. Those quantities need not be identical, particularly when only a finite number of bootstrap samples has been generated.
These labels matter. The point estimate from the observed data is \(\hat\theta\); the bootstrap-replicate mean is a separate summary. The bootstrap standard error describes the spread of the replicate estimates under the stated resampling procedure. A percentile interval uses selected quantiles of the replicate estimates. None of these summaries, by itself, removes uncertainty about whether the data or model adequately represents the risk under study.
A Reporting Framework
A concise, reproducible report should answer four questions. First, what is the estimand—for example, a mean severity, a probability of a specified event, or an estimator’s bias under a data-generating model? Second, what estimate or simulation procedure was used? Third, what uncertainty summary is reported, and how was it calculated? Fourth, what assumptions and limitations govern its interpretation?
State the target and its units. Distinguish a parameter being estimated from a statistic calculated from the data.
For a bootstrap, specify the resampling unit and scheme. For a simulation study, state the data-generating assumptions and what was held fixed.
Give the original estimate, bootstrap bias or standard error, and interval as applicable. For estimator-performance studies, report bias, variance, or root mean squared error with their definitions.
State what uncertainty is represented, what is not represented, and any limitations from sample size, model choice, or simulation count.
Bootstrap Summaries and Their Interpretation
For \(B\) bootstrap estimates, the usual estimated bootstrap standard error is
The denominator \(B-1\) gives the sample standard deviation of the generated replicates. A percentile interval is formed from specified quantiles of those replicates. The report should identify the interval type, confidence level, and quantile convention when that convention could affect the result. A bootstrap standard error is a measure of spread, not an interval; it should not be presented as though it were a confidence interval.
Why the Bootstrap Standard Error Uses Replicate Spread
The bootstrap treats each resample as a realization from an estimated sampling process, conditional on the observed data and the resampling assumptions. Applying the same estimator to each resample produces values that approximate the estimator’s variation under that process. Their center is \(\bar{\hat\theta}^*\), so variation around that center is measured by squared deviations from \(\bar{\hat\theta}^*\), not from the original estimate. Dividing their sum by \(B-1\) and taking the square root yields the sample standard deviation of the generated values.
This derivation also explains why the replicate mean is not automatically the original estimate: a finite collection of resamples is only a finite Monte Carlo sample from the bootstrap distribution. Its mean can differ from \(\hat\theta\), even when the bootstrap procedure is appropriately constructed. A report must give each value its correct label.
Worked Example: Reporting a Bootstrap Severity Estimate
A hypothetical insurer observes five uncensored, independent claim severities, in thousands of dollars: \(60, 80, 100, 110,\) and \(130\). There is no discounting or mortality adjustment. The empirical distribution assigns probability \(1/5\) to each observed claim. Five bootstrap samples of size five, drawn with replacement, produce mean severities of \(90, 94, 98, 102,\) and \(106\), also in thousands of dollars. Report the observed estimate, bootstrap bias, bootstrap standard error, and an 80% percentile interval. For the interval, use linear interpolation with quantile position \(1+(B-1)p\).
Observed estimate. The original sample mean is \((60+80+100+110+130)/5=480/5=96\). Thus, \(\hat\theta=\$96{,}000\).
Bootstrap-replicate mean and bias. The replicate mean is \((90+94+98+102+106)/5=490/5=98\) thousand dollars. The estimated bootstrap bias is \(98-96=2\) thousand dollars. The replicate mean of \(\$98{,}000\) is not the observed estimate, which remains \(\$96{,}000\).
Bootstrap standard error. Deviations from the replicate mean are \(-8,-4,0,4,8\). The sum of squared deviations is \(64+16+0+16+64=160\). Therefore, \(\widehat{\operatorname{se}}_{\mathrm{boot}}=\sqrt{160/(5-1)}=\sqrt{40}=6.3249\) thousand dollars.
80% percentile interval. The lower quantile is at \(p=0.10\), with position \(1+4(0.10)=1.4\). Interpolating 40% of the way from 90 to 94 gives \(90+0.4(4)=91.6\). The upper quantile is at \(p=0.90\), with position \(1+4(0.90)=4.6\). Interpolating 60% of the way from 102 to 106 gives \(102+0.6(4)=104.4\). The interval is \((\$91{,}600,\ \$104{,}400)\).
Report and check. A transparent summary is: “The observed sample mean severity was \(\$96{,}000\). The mean of five nonparametric bootstrap replicate estimates was \(\$98{,}000\), giving an estimated bootstrap bias of \(\$2{,}000\) and a bootstrap standard error of \(\$6{,}324.9\). The 80% percentile interval, using linear interpolation, was \((\$91{,}600,\ \$104{,}400)\).” The interval brackets both the observed estimate and replicate mean. Five replicates are far too few for a stable practical interval; this calculation illustrates reporting and interpolation, not a recommended bootstrap run size. Common mistake avoided: calling the replicate mean the estimate from the original data.
The bootstrap conclusions also depend on the resampling scheme. Resampling individual claims assumes that those observations are exchangeable realizations of the relevant sampling process. If observations are clustered, dependent, stratified, or otherwise heterogeneous, the resampling unit or scheme must reflect that structure. The resampling method should be named rather than hidden behind the word “bootstrap.” An interval’s numerical endpoints do not reveal whether its assumptions were appropriate.
Reporting Simulation Performance and Monte Carlo Error
A simulation study can serve different purposes. In a performance study, a parameter value is fixed by a specified data-generating model, and repeated simulated data sets are used to assess an estimator. If the true value is \(\theta\), and the resulting estimates are \(\hat\theta_1,\ldots,\hat\theta_R\), the simulated mean estimate is \(\bar{\hat\theta}=R^{-1}\sum_{r=1}^R\hat\theta_r\). A common empirical bias summary is \(\bar{\hat\theta}-\theta\). The simulation variance and root mean squared error can be summarized as
Here the variance summary uses divisor \(R\) because it describes the empirical distribution of the simulated estimates; it is not the usual unbiased sample-variance estimator. The RMSE combines dispersion and bias relative to the specified true value. A report should state the definitions or divisor used, especially when comparing results from different sources.
In a different setting, simulation estimates a quantity such as an event probability under a fixed model. Random simulation error—often called Monte Carlo error—arises because only finitely many trials are run. It is distinct from model uncertainty: increasing the number of trials can reduce Monte Carlo error, but does not show that the assumed model is correct or capture uncertainty about its parameters unless that uncertainty is explicitly included.
Worked Example: Summarizing Simulated Estimator Performance
A hypothetical, undiscounted severity model has independent claim severities \(X_i\sim\operatorname{Gamma}(25,10)\), using shape 25 and scale \(\$10{,}000\). Thus, \(E[X_i]=\$250{,}000\) and \(\operatorname{Var}(X_i)=25(\$10{,}000)^2\). In each of five independent simulated data sets of 100 claims, the sample mean estimates the model mean. The five estimates, in thousands of dollars, are \(242,246,250,254,\) and \(258\). There are no mortality or interest assumptions. Calculate the simulated bias, variance, and RMSE.
Simulated mean estimate. The average is \((242+246+250+254+258)/5=1{,}250/5=250\) thousand dollars. Since the stipulated true mean is 250 thousand dollars, the estimated bias is \(250-250=0\) thousand dollars.
Simulation variance. Deviations from 250 are \(-8,-4,0,4,8\), with squared deviations summing to \(160\). Using the stated divisor \(R=5\), the variance summary is \(160/5=32\) \((\text{thousand dollars})^2\).
RMSE. The squared errors from the true value also sum to 160. Hence the RMSE is \(\sqrt{160/5}=\sqrt{32}=5.6569\) thousand dollars.
Report and check. “Under the specified Gamma model, five simulated samples of 100 claims gave a mean estimated severity of \(\$250{,}000\), empirical bias of \(\$0\), simulation variance of \(32\) \((\text{thousand dollars})^2\), and RMSE of \(\$5{,}656.9\).” The zero bias is the result for these five replicates, not proof that the estimator has zero bias in general. Common mistake avoided: reporting an observed finite-run bias as a guaranteed property of the estimator.
Worked Example: Reporting Monte Carlo Error for an Event Probability
A hypothetical loss model is held fixed, and a simulation generates \(N=400\) independent annual outcomes. An outcome is marked when aggregate claims exceed a specified threshold; 36 outcomes are marked. No interest or mortality adjustment is made. Estimate the model probability of exceeding the threshold and give an approximate 95% Monte Carlo interval using the normal approximation.
Estimate. The estimated probability is \(\hat p=36/400=0.09\), or \(0.0900\) to four decimal places.
Monte Carlo standard error. Under independent Bernoulli trials with probability \(p\), the sample proportion has estimated standard error \(\sqrt{\hat p(1-\hat p)/N}\). Here this is \(\sqrt{0.09(0.91)/400}=\sqrt{0.00020475}=0.014309\).
Approximate interval. The normal-approximation margin is \(1.96(0.014309)=0.028046\). The endpoints are \(0.09-0.028046=0.061954\) and \(0.09+0.028046=0.118046\). Rounded to four decimal places, the approximate interval is \((0.0620,0.1180)\).
Interpretation and check. This interval summarizes finite-simulation uncertainty for the event probability under the fixed model and independent-trial assumption. It is not an interval for real-world risk that incorporates model uncertainty. The estimated probability lies between the endpoints. Common mistake avoided: interpreting a Monte Carlo interval as evidence that the chosen loss model is correct.
What a Useful Report Should Make Explicit
A result is easier to review when units, rounding, and qualifications are consistent. Report money in dollars or thousands as stated, and do not switch units within a calculation. Give probabilities to four decimal places when that precision is appropriate, and avoid displaying extra digits that imply unjustified accuracy. State the number of bootstrap replicates or simulation runs; a standard error or interval without its computational basis is incomplete.
Bootstrap and simulation summaries should also be kept conceptually separate. A bootstrap interval addresses sampling variation under a resampling scheme applied to observed data. A simulation performance study addresses estimator behavior under a stipulated data-generating model. A Monte Carlo standard error describes numerical variability from a finite simulation. These summaries may all appear in one analysis, but they should not be combined under one undifferentiated label.
| Quantity | What it summarizes | What it does not establish by itself |
|---|---|---|
| Bootstrap standard error | Spread of estimates over the stated bootstrap resamples | That the resampling assumptions or data model are adequate |
| Bootstrap percentile interval | Specified quantiles of the bootstrap replicate distribution | That the interval has its intended coverage in every setting |
| Simulation bias or RMSE | Estimator performance under the specified data-generating model | Performance under materially different models or populations |
| Monte Carlo standard error | Finite-run simulation variability for the stated target | Uncertainty about fixed model assumptions that were not varied |
A practical limitation should be stated when it affects interpretation. Examples include a small observed sample, very few bootstrap replicates, rare events represented by few simulated occurrences, or sensitivity to a parametric model. These are not repaired by a polished table. They are reasons to explain what the result can support and what requires further investigation. Accurate, qualified reporting is aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.
Check Your Understanding
For each problem, identify the relevant uncertainty summary and keep estimates distinct from replicate summaries.
- A hypothetical sample of claim severities has an observed mean of \(\$48{,}000\). Five bootstrap replicate means, in thousands of dollars, are \(45,47,49,51,53\). Calculate the replicate mean, estimated bootstrap bias, and bootstrap standard error using divisor \(B-1\). (Answer: replicate mean \(\$49{,}000\); bias \(+\$1{,}000\); bootstrap standard error \(\$3{,}162.3\).)
- A simulation study has true parameter \(\theta=12\) and five estimates \(10,11,12,13,14\). Using divisor \(R\) for simulation variance, calculate bias, variance, and RMSE. (Answer: bias \(0\); variance \(2\); RMSE \(\sqrt{2}\approx1.4142\).)
- In 900 independent simulation trials under a fixed model, an event occurs 81 times. Calculate the estimated event probability and its plug-in Monte Carlo standard error. (Answer: probability \(0.0900\); standard error \(\sqrt{0.09(0.91)/900}=0.0095\), to four decimal places.)
- A report gives a bootstrap-replicate mean of \(\$76{,}500\), an original-data estimate of \(\$75{,}000\), and a bootstrap standard error of \(\$2{,}000\). Which value is the point estimate, and what is the estimated bootstrap bias? (Answer: point estimate \(\$75{,}000\); estimated bootstrap bias \(+\$1{,}500\).)
- A simulation report gives a narrow Monte Carlo interval for a probability but uses one fixed severity model and does not vary its parameters. What does the interval describe, and what uncertainty does it omit? (Answer: it describes finite-run simulation uncertainty for the probability under the fixed model; it omits uncertainty about the model assumptions and parameters.)