Separating Monte Carlo Error From Model Uncertainty

Why Two Kinds of Uncertainty Must Be Separated

A simulation result is calculated using a model and a finite collection of random draws. Its variation can therefore have more than one source. Even if the model and all its parameters are fixed, a new set of random draws will usually produce a slightly different answer. That variation is Monte Carlo error. Separately, the model may be an imperfect description of the risk, or its parameters may be uncertain because they were estimated from limited data. Those sources of uncertainty are not removed merely by generating more draws.

This distinction matters when simulation is used to estimate a claim probability, summarize a bootstrap distribution, or compare actuarial assumptions. A result with very small Monte Carlo error can still be highly uncertain as a description of future losses if the data or model assumptions are uncertain. Conversely, a model can be well specified for a particular purpose while a simulation estimate remains noisy because it uses too few replicates.

Definition: Monte Carlo error is the random error in a computed simulation summary caused by using a finite number of random replicates, conditional on the specified data, model, parameter values, and simulation procedure. Model or parameter uncertainty concerns the assumptions, structure, or parameter values used to represent the underlying risk. Increasing the number of replicates reduces Monte Carlo error; it does not, by itself, reduce model or parameter uncertainty.

For example, a bootstrap distribution generated by resampling one observed data set is conditional on that data set and on the resampling scheme. The spread of the bootstrap estimates describes the variability represented by that procedure. A finite number of bootstrap replicates adds Monte Carlo error to summaries such as the bootstrap mean or standard deviation. These are different quantities: the spread of the bootstrap distribution is not the Monte Carlo standard error of its mean.

Conditional Simulation Precision

Suppose a fixed model assigns probability $p$ to an event, and $R$ independent simulations are used to estimate that probability. Let $I_r$ equal 1 when replicate $r$ has the event and 0 otherwise. Then the estimate is $\hat p=R^{-1}\sum_{r=1}^{R} I_r$. Because each indicator has variance $p(1-p)$, the variance of their average is $p(1-p)/R$. In practice, $p$ is unknown, so the estimated Monte Carlo standard error uses $\hat p$.

\widehat{\operatorname{SE}}_{\mathrm{MC}}(\hat p) = \sqrt{\frac{\hat p(1-\hat p)}{R}}.

This standard error describes the simulation precision of $\hat p$ for the fixed probability in the specified model. A normal-approximation interval $\hat p\pm1.96\,\widehat{\operatorname{SE}}_{\mathrm{MC}}(\hat p)$ is an approximate interval for the model probability, arising from simulation randomness. It is not automatically a confidence interval for the real-world probability when the model or its parameters are uncertain.

For a simulated quantity $Z$ with finite variance, the mean of $R$ independent replicates has Monte Carlo standard error estimated by $s_Z/\sqrt R$, where $s_Z$ is the sample standard deviation of the replicates. This describes the precision of the simulated mean, not the variability of individual outcomes. For instance, the simulated mean claim amount may be estimated precisely even though individual claim amounts vary widely.

Worked Example: A Simulated Probability and Its Monte Carlo Interval

A hypothetical insurer uses a fixed claim model to estimate the probability that a policy’s annual covered loss exceeds a specified threshold. In $R=2{,}500$ independent simulations, 175 losses exceed the threshold. Estimate the model probability, calculate its Monte Carlo standard error, and form a 95% normal-approximation interval for the model probability.

Step 1: Estimate the probability. The observed fraction is

\hat p=\frac{175}{2{,}500}=0.0700.

Step 2: Estimate the Monte Carlo standard error. Substitute $\hat p=0.0700$ and $R=2{,}500$:

\widehat{\operatorname{SE}}_{\mathrm{MC}}(\hat p) = \sqrt{\frac{0.0700(1-0.0700)}{2{,}500}} = \sqrt{\frac{0.0651}{2{,}500}} = \sqrt{0.00002604} \approx 0.0051039.

Step 3: Form the approximate interval. The normal-approximation margin is $1.96(0.0051039)=0.0100036$. Therefore,

0.0700\pm0.0100036 = (0.0599964,\ 0.0800036) \approx (0.0600,\ 0.0800).

The normal approximation is reasonable as a simulation approximation here because the estimated counts are $2{,}500(0.0700)=175$ event replicates and $2{,}500(0.9300)=2{,}325$ non-event replicates. The interval describes finite-simulation precision conditional on the fixed model. It does not show that the model’s threshold probability is known to within those limits in the real portfolio.

Common mistake and correction. A tempting arithmetic error is to calculate $1.96(0.0051039)$ as $0.0090036$, leading to endpoints near $(0.0610,0.0790)$. The product is $0.0100036$, not $0.0090036$, so the corrected interval is $(0.0600,0.0800)$ to four decimal places.

Parameter Uncertainty and the Variance Decomposition

When a model parameter is uncertain, the risk quantity of interest can vary because the parameter value is unknown, even before any simulation is run. If a probability distribution for the parameter is explicitly specified, the law of total variance separates variation conditional on that parameter from variation in the conditional mean across parameter values.

Theorem: For a random quantity $Z$ and an uncertain parameter $\Theta$, provided the variances are finite,

\operatorname{Var}(Z) = E[\operatorname{Var}(Z\mid\Theta)] + \operatorname{Var}(E[Z\mid\Theta]).

To see why this holds, write $Z-E[Z]$ as $(Z-E[Z\mid\Theta])+(E[Z\mid\Theta]-E[Z])$. Squaring and taking expectations gives two squared terms and a cross term. The expected cross term is zero: conditional on $\Theta$, $E[Z-E[Z\mid\Theta]\mid\Theta]=0$, while $E[Z\mid\Theta]-E[Z]$ is fixed. The first squared term has expectation $E[\operatorname{Var}(Z\mid\Theta)]$; the second has expectation $\operatorname{Var}(E[Z\mid\Theta])$. This gives the stated decomposition.

The first term is the average conditional process variation. The second is variation in the conditional mean caused by uncertainty about $\Theta$. If a simulation averages $R$ independent draws from the full mixture over parameter values, the Monte Carlo standard error of that simulated mean is based on the variance of a draw from the mixture divided by $R$. It is still a finite-replication error; it does not make the assumed parameter distribution more credible.

Worked Example: Separating Claim Variation From Parameter Uncertainty

For a hypothetical portfolio, annual claim count $N$, conditional on a severity parameter $\Theta=\theta$, is Poisson with mean $\lambda=2$. Claim amounts are independent exponential random variables with mean $\theta$, independent of $N$. An analyst assigns equal probability to $\Theta=900$ and $\Theta=1{,}100$ dollars. Let $S$ be annual aggregate claims. Calculate the mean and variance of $S$ across this parameter mixture, and identify the parameter-uncertainty component.

Step 1: Find conditional moments. For an exponential severity with mean $\theta$, $E[X\mid\theta]=\theta$ and $E[X^2\mid\theta]=2\theta^2$. For a compound Poisson aggregate, $E[S\mid\theta]=\lambda\theta=2\theta$, and $\operatorname{Var}(S\mid\theta)=\lambda E[X^2\mid\theta]=4\theta^2$.

Step 2: Average the conditional means. The conditional means are $1{,}800$ dollars at $\theta=900$ and $2{,}200$ dollars at $\theta=1{,}100$. Thus

E[S]=\frac{1}{2}(1{,}800)+\frac{1}{2}(2{,}200)=2{,}000\text{ dollars}.

Step 3: Calculate the two variance components. The average conditional variance is

E[\operatorname{Var}(S\mid\Theta)] = \frac{1}{2}(4(900)^2)+\frac{1}{2}(4(1{,}100)^2) = \frac{1}{2}(3{,}240{,}000+4{,}840{,}000) = 4{,}040{,}000.

The parameter component is the variance of the conditional means:

\operatorname{Var}(E[S\mid\Theta]) = \frac{1}{2}(1{,}800-2{,}000)^2 + \frac{1}{2}(2{,}200-2{,}000)^2 = 40{,}000.

Therefore, $\operatorname{Var}(S)=4{,}040{,}000+40{,}000=4{,}080{,}000$ dollars squared. Of this total, $40{,}000$ dollars squared is attributable to the specified uncertainty in the conditional mean across parameter values. The remaining $4{,}040{,}000$ is average conditional claim variation. Neither component is the Monte Carlo error of a simulated estimate; that error also depends on how many aggregate-loss replicates are used.

Check. The conditional means are symmetrically 200 dollars below and above 2,000, so their variance must be $200^2=40{,}000$. Adding this to the average conditional variance gives the stated total. The equal scenario weights and the conditional compound Poisson formulas are essential to the result.

Bootstrap Replicates: Resampling Uncertainty Versus Replication Error

A bootstrap procedure uses the observed data and a specified resampling rule to approximate the sampling behavior of an estimator. Its bootstrap distribution has a spread, which represents variability under that procedure. When only $B$ bootstrap replicates are generated, a summary of that distribution is itself estimated with Monte Carlo error. Increasing $B$ can stabilize the computed summary, but it does not add new observed data or remove uncertainty about whether the resampling procedure represents the risk adequately.

For a bootstrap mean, the Monte Carlo standard error is estimated by the sample standard deviation of the $B$ bootstrap estimates divided by $\sqrt B$. This direct formula concerns the mean. Other summaries, such as a quantile endpoint, have their own simulation precision behavior; the standard deviation of bootstrap estimates divided by $\sqrt B$ should not be casually used as the Monte Carlo standard error of every possible summary.

Worked Example: Monte Carlo Error in an Estimated Bootstrap Bias

A hypothetical premium estimator has value $0.80$ on the original data. The analyst generates $B=500$ independent bootstrap replicates of the estimator. Their mean is $0.84$, and their sample standard deviation is $0.12$. Estimate the bootstrap bias, its Monte Carlo standard error, and an approximate 95% interval reflecting only Monte Carlo error.

Step 1: Estimate the bootstrap bias. The bootstrap estimate of bias is the bootstrap mean minus the original estimate:

\widehat{\operatorname{Bias}}_{\mathrm{boot}}=0.84-0.80=0.04.

Step 2: Calculate Monte Carlo error in the bootstrap mean. The original estimate $0.80$ is fixed for these replicates, so subtracting it does not change the Monte Carlo standard error:

\widehat{\operatorname{SE}}_{\mathrm{MC}} = \frac{0.12}{\sqrt{500}} \approx \frac{0.12}{22.3607} \approx 0.005367.

Step 3: Form the approximate Monte Carlo interval. The margin is $1.96(0.005367)\approx0.01052$. Thus the interval for the computed bootstrap mean is approximately $(0.8295,0.8505)$; subtracting $0.80$ gives an interval for the estimated bootstrap bias of $(0.0295,0.0505)$, to four decimal places.

The bootstrap standard deviation $0.12$ describes the spread of the bootstrap estimates, whereas $0.005367$ describes the estimated Monte Carlo error in their mean from using 500 replicates. The approximate interval above is not a confidence interval for the true bias of the estimator under every possible data-generating process. It is conditional on the observed data and bootstrap procedure.

Check. If the number of replicates were increased from 500 to 2,000 while the underlying bootstrap distribution stayed the same, the standard error of the mean would be halved, since $\sqrt{2{,}000}=2\sqrt{500}$. It would be about $0.002683$, not $0.12/2{,}000$.

Comparing Model Scenarios

Uncertainty about model structure is often represented by comparing plausible scenarios, such as alternative severity tails or frequency assumptions. Unless probabilities are assigned to those scenarios, their difference is a sensitivity result, not a variance component that can automatically be combined with a Monte Carlo standard error. Report both the simulation precision within each scenario and the difference across scenarios.

Worked Example: Is a Difference Between Models Just Simulation Noise?

Two hypothetical models estimate the same annual expected covered loss. Each is simulated independently $R=900$ times. Model A produces a mean of \$1,200 and a replicate standard deviation of \$600; Model B produces a mean of \$1,350 and a replicate standard deviation of \$900. Calculate the Monte Carlo standard errors and assess the difference between the estimated means.

Step 1: Calculate each mean’s Monte Carlo standard error.

\widehat{\operatorname{SE}}_{\mathrm{MC},A} = \frac{600}{\sqrt{900}} = \frac{600}{30} = \$20, \qquad \widehat{\operatorname{SE}}_{\mathrm{MC},B} = \frac{900}{30} = \$30.

Step 2: Calculate the difference and its simulation standard error. The estimated difference, B minus A, is $\$1{,}350-\$1{,}200=\$150$. Because the runs are independent, the variance of the difference is the sum of the two simulation variances:

\widehat{\operatorname{SE}}_{\mathrm{MC}}(\hat\mu_B-\hat\mu_A) = \sqrt{20^2+30^2} = \sqrt{1{,}300} \approx \$36.06.

A 95% normal-approximation interval for the difference due to simulation error is $\$150\pm1.96(\$36.06)$, or approximately $(\$79.3,\$220.7)$. The estimated difference is larger than its Monte Carlo standard error, so finite simulation noise alone is not a persuasive explanation for the observed gap under these assumptions.

This calculation does not establish which model better describes actual losses. The difference is a model-scenario sensitivity; the interval quantifies only the finite-replication error in the simulated difference. More replicates would narrow that interval around the scenario difference, but would not resolve which model is appropriate.

A Practical Separation Checklist

Before presenting a simulation-based result, identify what is being held fixed and what is allowed to vary. A useful sequence is:

Name the target.
State whether the output is a model probability, an expected loss, a bootstrap bias estimate, or another quantity.

State the conditioning.
Specify the data, model structure, parameter values, and simulation procedure held fixed when describing Monte Carlo error.

Quantify finite-replication error.
Use a standard error appropriate to the summary, and report the number of independent replicates.

Report substantive uncertainty separately.
Describe parameter uncertainty or model-scenario results on their own terms. Combine components only when a defensible joint probability model has been specified.

A common exam trap is to describe every interval produced by simulation as a confidence interval for the real-world risk. The interval’s interpretation depends on what varied to create it. A Monte Carlo interval concerns random simulation draws; a bootstrap interval represents sampling uncertainty under a resampling procedure; a parameter-mixture interval relies on a specified parameter distribution. These interpretations are not interchangeable.

Key takeaway: Increasing the number of replicates reduces numerical noise in a simulation summary, typically at a rate proportional to $1/\sqrt R$. It does not correct a misspecified model, supply missing data, or eliminate uncertainty in estimated parameters. State the source of each uncertainty before interpreting its size.

Check Your Understanding

For each problem, keep simulation precision distinct from the uncertainty represented by the assumed model or resampling procedure.

A fixed model produces 48 event outcomes in 800 independent simulations. Calculate the estimated event probability, its Monte Carlo standard error, and a 95% normal-approximation interval. (Answer: $\hat p=0.0600$, Monte Carlo standard error $=0.0084$, interval $(0.0435,0.0765)$.)
An original estimator is $0.65$. From 400 bootstrap replicates, the mean is $0.71$ and the sample standard deviation is $0.20$. Find the estimated bootstrap bias and an approximate 95% interval for that bias reflecting only Monte Carlo error. (Answer: bias $=0.0600$, Monte Carlo standard error $=0.0100$, interval $(0.0404,0.0796)$.)
A compound Poisson model has $\lambda=1$ and exponential claim severity with mean $\Theta$. Suppose $\Theta$ is 80 or 120 dollars with equal probability. Find the mean aggregate claim, the parameter component $\operatorname{Var}(E[S\mid\Theta])$, and the total variance of aggregate claims. (Answer: mean $=\$100$; parameter component $=400$ dollars squared; total variance $=21{,}200$ dollars squared.)
Two independent model runs each use 5,000 replicates. Model A has simulated mean \$240 and replicate standard deviation \$100; Model B has mean \$270 and replicate standard deviation \$200. Find the Monte Carlo standard error of the difference B minus A and a 95% normal-approximation interval for that difference. (Answer: standard error $\$3.162$; interval approximately $(\$23.8,\$36.2)$.)
In one sentence, explain why multiplying the number of simulation replicates by four does not eliminate model uncertainty. (Answer: It approximately halves Monte Carlo standard error, but does not change the model assumptions or the uncertainty about whether those assumptions represent the risk.)

What You'll Learn