Separating Parameter Uncertainty from Simulation Error

Why Two Sources of Uncertainty Must Be Separated

A simulated rate or capital estimate can vary for two different reasons. First, a simulation is a finite random calculation: repeating it with the same fitted model may produce a different answer. Second, the fitted parameters themselves are estimates from data; a different plausible sample could produce a different fitted model and therefore a different indicated rate or capital amount. These are simulation error and parameter uncertainty, respectively.

Neither should be confused with the random variation in actual insured losses. For example, an aggregate loss model describes the distribution of losses from one period to another. Monte Carlo error is instead the numerical uncertainty in estimating a feature of that distribution from a finite number of simulated periods. Parameter uncertainty concerns uncertainty about which distribution to use.

Definition: With fitted parameters held fixed, simulation error is the variation in a numerical estimate caused by using a finite number of random draws. Parameter uncertainty is variation in the target estimate caused by uncertainty in the fitted model parameters. In a nested bootstrap simulation, the inner simulation estimates a target for each outer bootstrap refit.

The distinction matters in decisions. Increasing the number of simulated outcomes can reduce simulation error, but it does not supply new information about the data used to fit the model. If parameter uncertainty is the dominant source of variation, running the same fitted model for longer can create an appearance of precision without resolving the main uncertainty.

Repeated Simulation With Parameters Held Fixed

Suppose the model parameters have been estimated and are now held fixed. Run $B$ independent simulation batches, each using the same fitted model and the same number of simulated policies. Let $\hat{\psi}_b$ be the estimate from batch $b$, such as a pure premium or a simulated capital measure. The average across batches is $\bar{\psi}$, and the sample variance of the batch estimates is

s^2_{\mathrm{MC}}=\frac{1}{B-1}\sum_{b=1}^{B}(\hat{\psi}_b-\bar{\psi})^2.

If the batches are independent and have the same simulation variance, the estimated standard error of their average is $s_{\mathrm{MC}}/\sqrt{B}$. This is the simulation error of the average of the $B$ batch estimates, not the standard deviation of losses in the underlying insurance model.

Worked Example: Repeating a Fixed-Parameter Pure-Premium Simulation

A hypothetical annual portfolio is modeled with claim count $N\sim\operatorname{Poisson}(\lambda)$, where $\lambda=0.10$ claims per policy-year. Claim severities are independent exponential random variables with mean $\theta=\$500$, independent of $N$. The fitted parameters are held fixed. Four independent simulation batches each contain 100 policy-years and produce pure-premium estimates of $\$30$, $\$40$, $\$60$, and $\$70$. Find the average estimate and the estimated simulation standard error of that average.

Method. First check the model’s expected pure premium. For a compound Poisson aggregate loss per policy-year, $E[S]=\lambda E[X]$. Thus the model expectation is $0.10(\$500)=\$50$, consistent with the scale of the simulated results.

The average of the four batch estimates is

\bar{\psi}=\frac{30+40+60+70}{4}=\$50.

The deviations from $\$50$ are $-\$20,-\$10,\$10,\$20$. Their squared sum is $400+100+100+400=1{,}000$ dollars squared. Therefore, the sample variance of the batch estimates is

s^2_{\mathrm{MC}}=\frac{1{,}000}{4-1}=333.333\ldots\ \text{dollars}^2.

The estimated standard error of the average of four independent batches is

\frac{s_{\mathrm{MC}}}{\sqrt{4}} =\sqrt{\frac{333.333\ldots}{4}} =\$9.13.

The average simulation estimate is $\$50.00$, and its estimated Monte Carlo standard error is $\$9.13$. This describes numerical uncertainty in this average; it does not say that the actual loss per policy-year has a standard deviation of $\$9.13$. Common mistake avoided: reporting the sample standard deviation of the four batch estimates, about $\$18.26$, as the standard error of their average. The standard error divides that standard deviation by $\sqrt{4}$.

Adding Parameter Bootstrap Replications

To assess parameter uncertainty, resample the observed data according to an appropriate bootstrap procedure, refit the model to each resampled data set, and calculate the target for each refit. If simulation is used to calculate the target, run an inner simulation conditional on that refitted model. The outer bootstrap captures how refitting changes the target; the inner simulation contributes additional numerical noise.

Let $\theta_b^*$ denote the fitted parameter vector in bootstrap replication $b$, and let $\hat{\psi}_b$ be the inner-simulation estimate for that refit. In the variance decomposition below, assume that the bootstrap replications are independent, the inner estimator is conditionally unbiased for the target under each refit, and the inner simulation error is independent across replications conditional on the refitted parameters.

Theorem: Under those assumptions, the variance across repeated nested estimates has two components:

\operatorname{Var}(\hat{\psi}) = \operatorname{Var}\!\left(E[\hat{\psi}\mid\theta^*]\right) + E\!\left[\operatorname{Var}(\hat{\psi}\mid\theta^*)\right].

The first term is the parameter-uncertainty component: it measures how the conditional target changes across bootstrap refits. The second term is the average inner simulation-error variance. This is the law of total variance, applied by conditioning on the refitted parameters.

To see why the identity holds, write $\hat{\psi}-E[\hat{\psi}]$ as the sum of $\hat{\psi}-E[\hat{\psi}\mid\theta^*]$ and $E[\hat{\psi}\mid\theta^*]-E[\hat{\psi}]$. The first term has conditional mean zero given $\theta^*$, so its product with the second term has expectation zero. Taking the expected square therefore gives the sum of the conditional variance averaged over refits and the variance of the conditional expectation.

With $B$ outer replications, calculate the sample variance $s^2_{\mathrm{nested}}$ of the $B$ inner estimates. If the inner simulation variance is approximately the same in each replication and is estimated as $s^2_{\mathrm{inner}}$, a useful estimate of the parameter component is

\widehat{\operatorname{Var}}_{\mathrm{parameter}} \approx s^2_{\mathrm{nested}}-s^2_{\mathrm{inner}}.

If inner variances differ by replication, subtract their average instead. This subtraction estimates a variance component, not a standard deviation. With few bootstrap replications it can be noisy and even negative; a negative result indicates that the data are insufficient to distinguish the components reliably, not that true variance is negative. Any practical truncation at zero should be identified as a constrained estimate.

Worked Example: Separating the Variance in Bootstrap Rate Estimates

A hypothetical portfolio’s pure-premium model uses Poisson claim frequency and exponential severity. Four independent parameter-bootstrap refits are followed by inner simulations of the rate. The resulting rate estimates, in dollars per exposure, are $\$42,\$48,\$52,\$58$. Each inner simulation has estimated variance $4$ dollars squared. Estimate the parameter variance and report the total variance represented by the observed spread.

Method. The bootstrap estimates have mean

\bar{\psi}=\frac{42+48+52+58}{4}=\$50.

The deviations from the mean are $-8,-2,2,8$ dollars. Their squared sum is $64+4+4+64=136$ dollars squared. The sample variance across the four nested estimates is therefore

s^2_{\mathrm{nested}}=\frac{136}{4-1}=45.333\ldots\ \text{dollars}^2.

The average inner simulation-error variance is $4$ dollars squared, so the estimated parameter variance is

\widehat{\operatorname{Var}}_{\mathrm{parameter}} =45.333\ldots-4 =41.333\ldots\ \text{dollars}^2.

Thus the estimated parameter standard deviation is $\sqrt{41.333\ldots}=\$6.43$, while the inner simulation standard deviation is $\sqrt{4}=\$2.00$. The total variance across the nested estimates is $45.33$ dollars squared, with total standard deviation $\sqrt{45.333\ldots}=\$6.73$. Rounded to the nearest cent, the sample variance is $\$45.33$ squared and the estimated parameter variance is $\$41.33$ squared. Check: adding the components gives $41.333\ldots+4=45.333\ldots$, as required. Common mistake avoided: dividing the sum of squared deviations by $B$ instead of $B-1$ when calculating the sample variance.

Interpreting the Effect on a Capital Estimate

The same separation applies to a simulated capital measure, although its inner simulation may estimate a tail statistic rather than a mean. The following hypothetical example illustrates the arithmetic for a 99th-percentile estimate. The numerical quantiles are simulation outputs; no closed-form quantile calculation is assumed.

Worked Example: Comparing Fixed-Model and Bootstrap Capital Variation

A hypothetical annual aggregate loss model has $N\sim\operatorname{Poisson}(0.08)$ claims per policy-year and independent exponential claim severities with mean $\$1{,}000$. These fitted parameters are fixed for the first calculation. Four independent batches, each simulating 100,000 policy-years, estimate the 99th percentile of annual aggregate loss as $\$2{,}180$, $\$2{,}240$, $\$2{,}210$, and $\$2{,}270$. The estimated variance of one such batch’s quantile estimate is $900$ dollars squared. Separately, four parameter-bootstrap refits produce inner-simulation quantile estimates of $\$1{,}900$, $\$2{,}110$, $\$2{,}260$, and $\$2{,}530$, each with the same estimated inner variance of $900$ dollars squared. Calculate the fixed-model simulation standard error of the four-batch average and the estimated parameter variance from the bootstrap results.

Fixed fitted parameters. The four estimates average

\frac{2{,}180+2{,}240+2{,}210+2{,}270}{4} =\$2{,}225.

The deviations from $\$2{,}225$ are $-45,15,-15,45$ dollars. The squared deviations sum to $2{,}025+225+225+2{,}025=4{,}500$ dollars squared, so the sample variance is $4{,}500/3=1{,}500$ dollars squared. The estimated standard error of the four-batch average is

\sqrt{\frac{1{,}500}{4}}=\$19.36.

Parameter bootstrap. The bootstrap estimates average

\frac{1{,}900+2{,}110+2{,}260+2{,}530}{4} =\$2{,}200.

Their deviations are $-300,-90,60,330$ dollars. The squared deviations sum to $90{,}000+8{,}100+3{,}600+108{,}900=210{,}600$ dollars squared, giving a sample variance of $210{,}600/3=70{,}200$ dollars squared. Subtract the inner simulation variance:

\widehat{\operatorname{Var}}_{\mathrm{parameter}} =70{,}200-900 =69{,}300\ \text{dollars}^2.

The estimated parameter standard deviation is $\sqrt{69{,}300}=\$263.25$, to the nearest cent. This small illustrative bootstrap sample suggests that refitting may contribute much more variation than the fixed-model simulation error. It does not establish a precise capital range: four bootstrap replications are far too few for reliable tail quantiles or stable variance estimates. Check: the bootstrap total variance decomposes as $69{,}300+900=70{,}200$ dollars squared. Common mistake avoided: treating the observed bootstrap spread as wholly parameter-driven without removing the inner simulation variance.

Practical Interpretation and Common Traps

Variance decomposition is useful only when the simulation design matches the question. Use fixed-parameter repeats to assess numerical stability conditional on one fitted model. Use refitting across bootstrap samples when the goal is to reflect sampling uncertainty in the parameters. If the target can be calculated exactly for each refit, there is no need to add inner simulation error for that target.

Do not confuse loss volatility with simulation error. Aggregate-loss variability is a feature of the assumed risk model. Simulation error concerns estimation of a feature, such as its mean or quantile.
Do not expect more inner simulations to remove parameter uncertainty. Larger inner runs reduce conditional Monte Carlo error, but the spread across plausible refits remains.
Do not subtract variances from incompatible designs. The inner variance must refer to the same simulated statistic and comparable run size as the nested estimates.
Do not report a variance as a standard error. Variance is in squared dollars; its square root is in dollars. The standard error of an average also depends on the number of independent estimates being averaged.
Do not overinterpret a small bootstrap. The component estimates can themselves be unstable when the number of outer replications is small, particularly for tail measures.

Key takeaway: Hold fitted parameters fixed to measure simulation error; refit across bootstrap samples to assess parameter uncertainty. In a nested design, the observed variance across estimates includes both components, so estimate and interpret them separately before drawing conclusions about a rate or capital indication.

Check Your Understanding

For each problem, distinguish the relevant variance components and show the arithmetic before checking the answer.

Four independent fixed-parameter simulation batches estimate a pure premium as $\$18,\$20,\$22,\$24$. Find the average and the estimated standard error of that average. (Answer: average $\$21.00$; standard error $\$1.29$)
Five bootstrap rate estimates are $\$30,\$34,\$36,\$40,\$40$. Each has inner simulation-error variance $2$ dollars squared. Estimate the parameter variance by subtracting the inner variance from the sample variance across estimates. (Answer: sample variance $18$ dollars squared; estimated parameter variance $16$ dollars squared)
Four nested capital estimates have sample variance $25$ thousand-dollars squared, and the estimated average inner simulation variance is $9$ thousand-dollars squared. Give the estimated parameter variance and the total standard deviation across nested estimates. (Answer: parameter variance $16$ thousand-dollars squared; total standard deviation $5$ thousand dollars)
A rate estimate has parameter variance $64$ dollars squared and inner simulation variance $36$ dollars squared when each inner run uses 1,000 simulations. Assume inner variance is inversely proportional to the number of simulations. What are the total variance and standard deviation if each inner run is increased to 4,000 simulations? (Answer: total variance $73$ dollars squared; standard deviation $\$8.54$)
Four bootstrap estimates of a prospective rate are $\$72,\$76,\$80,\$84$, with inner simulation-error variance $4$ dollars squared per estimate. Calculate the sample variance, estimated parameter variance, and parameter standard deviation. (Answer: sample variance $26.67$ dollars squared; estimated parameter variance $22.67$ dollars squared; parameter standard deviation $\$4.76$)

What You'll Learn