Tutorials › Exam ASTAM - Advanced Short-Term Actuarial Mathematics › Separating Monte Carlo Error From Parameter Uncertainty

Home  /  Tutorials  /  Exam ASTAM / Simulation Precision and Design

Exam ASTAM · Simulation Precision and Design · Tutorial 673 of 979

Separating Monte Carlo Error From Parameter Uncertainty

Learn to separate parameter uncertainty from conditional simulation error and to report the uncertainty of a nested simulation estimate without double-counting.

Intermediate 12 min read

What You'll Learn

  • Define the target when fitted parameters are represented by a distribution of plausible values
  • Derive the variance decomposition for nested simulation
  • Estimate between-parameter and conditional simulation components from nested runs
  • Explain why the variance of inner averages includes inner simulation noise
  • Compare the precision effects of increasing outer and inner simulation counts
  • Identify common double-counting and uncertainty-reporting errors

Introduction

A simulation based on fitted model parameters can be precise about the wrong source of uncertainty. Repeating the simulated losses many times with the parameters held fixed measures Monte Carlo error conditional on those parameters. It does not show how much the result might change if the parameters themselves were uncertain. A nested simulation can represent both sources, but its reported standard error must account for how the inner simulation affects the outer averages.

This distinction matters when a result is derived from estimated frequencies, severities, or other model inputs. The fitted values may be treated as fixed for a conditional simulation, or uncertainty about them may be represented by a specified distribution of plausible parameter values. These are different analyses with different targets; a large number of inner simulations alone does not remove parameter uncertainty.

Definition: Let \(\Theta\) have a specified distribution \(Q\) representing uncertainty in the model parameters. Conditional on \(\Theta=\theta\), let \(Y\) be a simulated output, with conditional mean \(m(\theta)=E[Y\mid\Theta=\theta]\) and conditional variance \(\sigma^2(\theta)=\operatorname{Var}(Y\mid\Theta=\theta)\). Parameter uncertainty is variation in \(m(\Theta)\) under \(Q\). Conditional Monte Carlo error is the sampling variation from simulating \(Y\) while holding \(\Theta\) fixed.

The distribution \(Q\) must be justified by the analysis. It might arise from a bootstrap procedure, a posterior distribution, or an explicitly specified approximation to parameter-estimation uncertainty. A fitted parameter is not automatically random: if the question is only the conditional expected output at one fitted value \(\hat\theta\), then the target is \(m(\hat\theta)\), and uncertainty in \(\hat\theta\) is outside that conditional calculation.

The Variance Decomposition

The law of total variance separates the variation in \(Y\) into the variation of its conditional mean and the average of its conditional variance.

$$ \operatorname{Var}(Y) = \operatorname{Var}_{Q}\!\left(m(\Theta)\right) + E_{Q}\!\left[\sigma^2(\Theta)\right]. $$

For a nested simulation, draw \(B\) independent parameter values \(\Theta_1,\ldots,\Theta_B\) from \(Q\). For each \(\Theta_b\), simulate \(M\) conditionally independent outputs \(Y_{b1},\ldots,Y_{bM}\). The inner average and grand average are

$$ \bar Y_b=\frac{1}{M}\sum_{j=1}^{M}Y_{bj}, \qquad \hat\mu=\frac{1}{B}\sum_{b=1}^{B}\bar Y_b. $$

Here \(\hat\mu\) estimates \(\mu=E_Q[m(\Theta)]\). Conditional on \(\Theta_b\), the inner average has mean \(m(\Theta_b)\) and variance \(\sigma^2(\Theta_b)/M\). Applying the total-variance identity to \(\bar Y_b\), then using independence between outer groups, gives

$$ \operatorname{Var}(\hat\mu) = \frac{1}{B} \left\{ \operatorname{Var}_{Q}\!\left(m(\Theta)\right) + \frac{E_{Q}\!\left[\sigma^2(\Theta)\right]}{M} \right\}. $$

This formula has an important interpretation. Increasing \(M\) reduces the conditional simulation contribution, but it does not reduce the between-parameter contribution for a fixed \(B\). Increasing \(B\) reduces both contributions to the variance of the grand average. In the limit as \(M\) grows, the inner averages approach their conditional means, but the finite-\(B\) estimate still varies because only \(B\) parameter values were drawn.

Key result: With independent outer parameter draws and conditionally independent inner simulations, the variance of the grand average is \(\{\operatorname{Var}_{Q}(m(\Theta))+E_Q[\sigma^2(\Theta)]/M\}/B\). The result assumes finite variances and a common inner count \(M\). Dependence between outer draws or inner outputs requires a variance calculation that accounts for that dependence.

Estimating Both Components From Nested Runs

The sample variance of the \(B\) inner averages is not an estimate of between-parameter variance alone. Each inner average contains simulation noise. In fact, under the nested design,

$$ E\!\left[s_{\bar Y}^{\,2}\right] = \operatorname{Var}_{Q}\!\left(m(\Theta)\right) + \frac{E_Q[\sigma^2(\Theta)]}{M}, $$

where \(s_{\bar Y}^{\,2}\) is the usual sample variance of \(\bar Y_1,\ldots,\bar Y_B\). If the aim is to estimate the between-parameter component separately, estimate the average within-group variance and subtract its contribution to the inner averages:

$$ \widehat{\operatorname{Var}}_{\!Q}(m(\Theta)) = s_{\bar Y}^{\,2}-\frac{\bar s_{\mathrm{within}}^{\,2}}{M}, \qquad \bar s_{\mathrm{within}}^{\,2} = \frac{1}{B}\sum_{b=1}^{B}s_b^2. $$

Here \(s_b^2\) is the sample variance of the \(M\) outputs in group \(b\). Under the stated sampling assumptions, \(\bar s_{\mathrm{within}}^{\,2}\) estimates \(E_Q[\sigma^2(\Theta)]\). Subtracting its estimated contribution removes the inner simulation noise already present in the variance of the inner averages.

Alternatively, if the goal is only the standard error of the grand average, estimate its variance directly as \(s_{\bar Y}^{\,2}/B\). This already includes both sources of variation. Do not add a separate within-group term to it: doing so would count inner simulation error twice. With small samples, a corrected estimate of the between-parameter component can be negative. That does not imply a negative true variance; it signals sampling variability in the component estimates. Report the estimation method and do not silently treat the raw difference as a meaningful negative variance.

Worked Example: Separating Components Without Double-Counting

A hypothetical portfolio simulation uses \(B=4\) independent parameter draws. Conditional on each parameter draw, the output is modeled as normally distributed, with a group-specific mean and variance. Two inner simulations are run for each draw, producing the pairs \((7,9)\), \((9,11)\), \((11,13)\), and \((13,15)\). Estimate the between-parameter variance, the average conditional variance, and the standard error of the grand average. Assume no interest or mortality adjustment is involved.

Method. Calculate the four inner averages and within-group sample variances. Use the subtraction correction for the between-parameter component, then calculate the variance of the grand average. This avoids treating noisy inner averages as if they were exact conditional means.

The inner averages are \(8,10,12,14\), so the grand average is

$$ \hat\mu=\frac{8+10+12+14}{4}=11. $$

Each pair is separated from its mean by 1. With two observations, the sample variance is the sum of squared deviations divided by \(2-1\), so each group has \(s_b^2=1^2+1^2=2\). Thus \(\bar s_{\mathrm{within}}^{\,2}=2\).

The sample variance of the inner averages is

$$ s_{\bar Y}^{\,2} = \frac{(8-11)^2+(10-11)^2+(12-11)^2+(14-11)^2}{4-1} = \frac{20}{3} \approx 6.6667. $$

The estimated between-parameter component is therefore

$$ \widehat{\operatorname{Var}}_{\!Q}(m(\Theta)) = \frac{20}{3}-\frac{2}{2} = \frac{17}{3} \approx 5.6667. $$

The estimated variance of the grand average can be obtained directly:

$$ \widehat{\operatorname{Var}}(\hat\mu) = \frac{s_{\bar Y}^{\,2}}{B} = \frac{20/3}{4} = \frac{5}{3} \approx 1.6667, \qquad \widehat{\operatorname{SE}}(\hat\mu)=\sqrt{5/3}\approx 1.291. $$

As a check, the component calculation gives a between-parameter contribution of \((17/3)/4=17/12\), and a conditional simulation contribution of \(2/(2\cdot4)=1/4\). Their sum is \(17/12+1/4=20/12=5/3\), as required. Common mistake avoided: adding \(2/(2\cdot4)\) to \(s_{\bar Y}^{\,2}/B\) would double-count the conditional simulation contribution, because \(s_{\bar Y}^{\,2}/B\) already includes it.

Worked Examples of Interpreting and Planning Precision

Worked Example: Conditional Error Versus Parameter Uncertainty

For a hypothetical claim indicator, a parameter \(\Theta\) equals \(0.20\) or \(0.40\), each with probability \(1/2\). Given \(\Theta=\theta\), the output \(Z\) is Bernoulli with success probability \(\theta\). A nested estimate uses \(B=100\) parameter draws and \(M=10\) indicators per draw. Calculate the target mean and the variance and standard error of the grand average. No interest or mortality assumption is relevant.

Method. Use \(m(\theta)=\theta\) and \(\sigma^2(\theta)=\theta(1-\theta)\), then apply the nested variance formula.

The target mean is \(\mu=(0.20+0.40)/2=0.30\). The between-parameter variance is

$$ \operatorname{Var}_{Q}(m(\Theta)) = \frac{(0.20-0.30)^2+(0.40-0.30)^2}{2} = 0.01. $$

The conditional variances at the two parameter values are \(0.20(0.80)=0.16\) and \(0.40(0.60)=0.24\). Their average is \(0.20\). Therefore,

$$ \operatorname{Var}(\hat\mu) = \frac{0.01+0.20/10}{100} = 0.0003, \qquad \operatorname{SE}(\hat\mu)=\sqrt{0.0003}\approx 0.0173. $$

The parameter component contributes \(0.01/100=0.0001\), and the conditional simulation component contributes \(0.20/(10\cdot100)=0.0002\); together they give \(0.0003\). If \(\Theta\) were instead fixed at \(0.30\), then 1,000 Bernoulli simulations would estimate the conditional mean with variance \(0.30(0.70)/1000=0.00021\). That conditional calculation answers a different question: it does not include uncertainty represented by the two-point distribution of \(\Theta\). Common mistake avoided: reporting only a standard error conditional on one fitted value and describing it as including parameter uncertainty.

Worked Example: Choosing Nested Simulation Counts

A hypothetical model has two equally likely parameter states. Under them, the conditional output distributions are \(Y\mid\Theta=1\sim N(49.4,9)\) and \(Y\mid\Theta=2\sim N(50.6,9)\), where the second normal argument is the variance. No interest or mortality adjustment is used. A pilot analysis estimates \(\operatorname{Var}_Q(m(\Theta))=0.36\) and \(E_Q[\sigma^2(\Theta)]=9\). Compare the standard errors for \(B=40\) with \(M=9\) and \(M=36\); then find the minimum \(B\) for \(M=36\) to achieve standard error at most \(0.12\).

Method. Use the pilot variance components in \(\operatorname{Var}(\hat\mu)=(0.36+9/M)/B\). The stated conditional normal distributions have means \(49.4\) and \(50.6\), whose equally weighted mean is \(50\) and whose variance is \(\{(-0.6)^2+(0.6)^2\}/2=0.36\), consistent with the between-parameter component.

For \(M=9\) and \(B=40\),

$$ \widehat{\operatorname{Var}}(\hat\mu) = \frac{0.36+9/9}{40} = \frac{1.36}{40} = 0.034, \qquad \widehat{\operatorname{SE}}(\hat\mu)\approx 0.1844. $$

For \(M=36\) and \(B=40\),

$$ \widehat{\operatorname{Var}}(\hat\mu) = \frac{0.36+9/36}{40} = \frac{0.61}{40} = 0.01525, \qquad \widehat{\operatorname{SE}}(\hat\mu)\approx 0.1235. $$

At \(M=36\), the requirement is \(0.61/B\leq 0.12^2=0.0144\). Hence \(B\geq 0.61/0.0144\approx42.3611\), so the minimum integer is \(B=43\). At that value, \(0.61/43\approx0.014186\) and its square root is approximately \(0.1191\), which meets the target. Common mistake avoided: increasing \(M\) alone cannot eliminate the \(0.36/B\) contribution from uncertainty across parameter states.

Reporting Results and Common Traps

A clear report identifies the target, the source of parameter uncertainty, and the simulation design. State whether the estimate concerns a conditional mean at fixed fitted parameters or an average over a specified parameter distribution. For nested simulation, give \(B\) and \(M\), describe how parameter draws were obtained, and say whether the reported standard error is conditional or includes variation across parameter draws.

  • Do not label all simulation variability “parameter uncertainty.” Inner outcomes vary even when their parameter draw is fixed.
  • Do not call the variance of the inner averages a pure between-parameter estimate. It includes approximately \(E_Q[\sigma^2(\Theta)]/M\). Subtract the estimated noise contribution if estimating the between component separately.
  • Do not add the within-group contribution to the direct grand-average variance estimate. The estimate \(s_{\bar Y}^{\,2}/B\) already includes both sources.
  • Do not assume that more inner runs resolve uncertainty in fitted parameters. At fixed \(B\), the between-parameter contribution remains.
  • Check independence and finite variance. Correlated parameter draws, shared random numbers, or heavy-tailed outputs may invalidate the simple variance estimate or make it unstable.

These distinctions help an analyst decide what a reported precision measure actually supports. A small conditional Monte Carlo standard error is evidence that repeated simulation at the same parameter values is stable; it is not by itself evidence that the fitted model inputs are known precisely. Conversely, a parameter-averaged result is only as meaningful as the distribution \(Q\) used to represent uncertainty in those inputs.

The variance decomposition and its careful estimation are aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus. On an exam problem, first identify whether the requested uncertainty is conditional or parameter-inclusive; then match the variance formula and simulation design to that target.

Check Your Understanding

For each problem, identify the relevant variance components before calculating. Assume independent outer draws and conditionally independent inner outputs unless stated otherwise.

  1. A hypothetical model has \(\operatorname{Var}_Q(m(\Theta))=0.25\), \(E_Q[\sigma^2(\Theta)]=4\), \(B=25\), and \(M=16\). Find the variance and standard error of the grand average. (Answer: variance \(=0.02\); standard error \(\approx0.1414\).)
  2. For \(B=5\) groups, the inner averages are \(3,5,7,9,11\), and the average within-group sample variance is \(6\). Find the sample variance of the inner averages and the corrected estimate of between-parameter variance for \(M=3\). (Answer: \(s_{\bar Y}^{\,2}=10\); corrected between-parameter estimate \(=10-6/3=8\).)
  3. A nested design has estimated between-parameter variance \(1.2\) and average conditional variance \(8\). With \(B=50\), compare the standard errors for \(M=4\) and \(M=16\). (Answer: \(M=4\), \(\sqrt{(1.2+2)/50}\approx0.2530\); \(M=16\), \(\sqrt{(1.2+0.5)/50}\approx0.1844\).)
  4. For \(B=8\), \(M=2\), the sample variance of the inner averages is \(5\), and the average within-group sample variance is \(3\). Calculate the direct estimated variance of the grand average and the corrected between-parameter component. (Answer: direct variance estimate \(=5/8=0.625\); corrected between-parameter estimate \(=5-3/2=3.5\).)
  5. A model has between-parameter variance \(0.16\) and average conditional variance \(4\). For \(M=20\), what is the minimum integer \(B\) needed for standard error at most \(0.10\)? (Answer: \(B\geq(0.16+4/20)/0.10^2=36\), so the minimum is \(36\).)

Previous

Next

Back to Tutorials
Clinical Biostats