Checking Simulated Means And Variances

Why Check Simulated Means and Variances?

A simulation can generate plausible-looking individual outcomes and still produce the wrong aggregate-loss distribution. The frequency may have the wrong parameter, a severity may be scaled twice, or the aggregate may be formed incorrectly. Comparing simulated summaries with known model moments is a basic validation step: it checks whether the simulation behaves as the specified model predicts, while allowing for ordinary variation between finite simulation runs.

The key is not to expect exact agreement. A finite set of simulated aggregates will usually have a sample mean and sample variance different from their theoretical counterparts. The size of the discrepancy matters: a small difference may be ordinary simulation variation, while a large, systematic difference calls for checking the model inputs and implementation.

Definition: Let $N$ be a claim count, and let $X_1,X_2,\ldots$ be independent, identically distributed claim severities, independent of $N$. The aggregate loss is $S=\sum_{j=1}^{N}X_j$, with $S=0$ when $N=0$. Write $E[X]=\mu_X$, $\operatorname{Var}(X)=\sigma_X^2$, $E[N]=\mu_N$, and $\operatorname{Var}(N)=\sigma_N^2$. Provided the moments exist, $E[S]=\mu_N\mu_X$ and $\operatorname{Var}(S)=\mu_N\sigma_X^2+\sigma_N^2\mu_X^2$.

These formulas apply to an ordinary compound model with independent frequency and severities, and with the stated moments finite. They are not generally valid if severity depends on frequency, if claim severities are dependent, or if the simulation applies a coverage modification that has not also been reflected in the severity moments. The input severity moments must describe the payments actually being aggregated, not a different underlying loss amount.

Why the Compound Variance Formula Holds

Condition on the claim count. Given $N=n$, the aggregate is a sum of $n$ independent severities, so its conditional mean is $n\mu_X$ and its conditional variance is $n\sigma_X^2$. The law of total expectation gives the aggregate mean. The law of total variance separates variability among severities at a fixed count from variability in the count itself.

E[S]=E[E[S\mid N]]=E[N\mu_X]=\mu_N\mu_X

\begin{aligned} \operatorname{Var}(S) &=E[\operatorname{Var}(S\mid N)]+\operatorname{Var}(E[S\mid N])\\ &=E[N\sigma_X^2]+\operatorname{Var}(N\mu_X)\\ &=\mu_N\sigma_X^2+\sigma_N^2\mu_X^2. \end{aligned}

The second variance term is easy to overlook. Even if individual severities were constant, aggregate loss would still vary whenever the number of claims varies. For a Poisson count with parameter $\lambda$, $\mu_N=\sigma_N^2=\lambda$, so the formulas simplify to $E[S]=\lambda E[X]$ and $\operatorname{Var}(S)=\lambda E[X^2]$. For a binomial count with parameters $m$ and $p$, $\mu_N=mp$ and $\sigma_N^2=mp(1-p)$.

Summarizing Replicated Aggregate Losses

Suppose the simulation generates $R$ independent aggregate losses $S_1,\ldots,S_R$, each from the same specified model. The natural checks are the sample mean and unbiased sample variance:

\bar S=\frac{1}{R}\sum_{r=1}^{R}S_r, \qquad s_S^2=\frac{1}{R-1}\sum_{r=1}^{R}(S_r-\bar S)^2.

The simulated mean estimates $E[S]$; the simulated variance estimates $\operatorname{Var}(S)$. For independent replicates, the standard error of $\bar S$ is $\sqrt{\operatorname{Var}(S)/R}$. Thus, a useful mean check is to divide the observed difference $\bar S-E[S]$ by this standard error. A difference of about one standard error is unsurprising; a difference several standard errors from zero is more concerning, especially if a rerun with a larger $R$ gives a similar result.

This standard-error calculation assumes the theoretical aggregate variance is finite and that the replicates are independent. A rough check is not a proof that the code is correct: different errors can offset one another in the mean, and matching two moments does not establish that the entire distribution is correct. Variance estimates are often noisier than mean estimates, particularly for skewed or heavy-tailed aggregate losses. Do not treat a modest variance discrepancy as definitive evidence of a bug without considering the sample size and tail behavior.

Worked Example: A Small Poisson Simulation Check

A hypothetical model has $N\sim\operatorname{Poisson}(2)$, and every claim payment is a fixed $\$10$. Ten independent simulated aggregate totals, in dollars, are $0,10,10,20,20,20,30,30,40,20$. Compare their mean and sample variance with the compound-model moments.

Method. First calculate the theoretical compound moments; then compute the summaries of the ten replicate totals. This check avoids expecting a short simulation run to reproduce the theoretical variance exactly.

For the frequency, $E[N]=2$ and $\operatorname{Var}(N)=2$. For the fixed severity, $E[X]=10$ and $\operatorname{Var}(X)=0$. Therefore,

E[S]=2(10)=20, \qquad \operatorname{Var}(S)=2(0)+2(10^2)=200.

The ten simulated totals sum to $200$, so $\bar S=200/10=\$20$. Their squared deviations from $\$20$ are $400,100,100,0,0,0,100,100,400,0$, with total $1{,}200$. Thus,

s_S^2=\frac{1{,}200}{10-1}=133.33\text{ dollars}^2

The simulated mean matches its theoretical value, while the sample variance is below the theoretical variance of $200$ dollars squared. With only ten replicates, that difference is not by itself a persuasive diagnosis of a coding error. As a mean check, the standard error is $\sqrt{200/10}=\$4.47$, and the observed mean is exactly at the theoretical mean. Check: The simulated total is $20$ dollars on average, consistent with the theoretical mean; the variance comparison remains imprecise because the run is small.

Worked Comparison for a Binomial Frequency

For each simulated run, calculate the aggregate first; then summarize the resulting list of aggregate totals. Do not confuse variability in individual claim counts within one run with the sample variance across independent runs. The latter is the statistic being compared with the model variance.

Worked Example: Checking a Binomial Claim Count

A hypothetical portfolio has $N\sim\operatorname{Binomial}(3,0.20)$. Each claim pays a fixed $\$50$. In $R=100$ independent simulated runs, the aggregate totals occur with these counts: $\$0$, 51 times; $\$50$, 39 times; $\$100$, 9 times; and $\$150$, once. Compare the simulated and theoretical moments.

Method. Use the binomial frequency moments and the fixed-severity compound formulas. The main trap is using the Poisson identity $\operatorname{Var}(N)=E[N]$, which does not hold for this binomial frequency.

The frequency moments are $E[N]=3(0.20)=0.60$ and $\operatorname{Var}(N)=3(0.20)(0.80)=0.48$. Since $E[X]=50$ and $\operatorname{Var}(X)=0$,

E[S]=0.60(50)=\$30, \qquad \operatorname{Var}(S)=0.60(0)+0.48(50^2)=1{,}200\text{ dollars}^2.

From the simulated frequency table, the total of all aggregates is $39(50)+9(100)+1(150)=\$3{,}000$, giving $\bar S=3{,}000/100=\$30$. The sum of squared aggregate totals is $39(50^2)+9(100^2)+1(150^2)=210{,}000$. Using $\sum(S_r-\bar S)^2=\sum S_r^2-R\bar S^2$, the centered sum of squares is $210{,}000-100(30^2)=120{,}000$. Hence,

s_S^2=\frac{120{,}000}{99}=1{,}212.12\text{ dollars}^2.

The mean matches the model mean, and the sample variance is close to the model variance. The standard error of the simulated mean is $\sqrt{1{,}200/100}=\$3.46$; the observed mean is at the target. Check: The sample variance uses $R-1=99$ in its denominator and is $1{,}212.12$, not $1{,}200$. Its modest difference from the theoretical value is plausible for a finite run.

Using the Pattern of Discrepancies

A useful validation is more than a pass-or-fail comparison. Look at the direction and relative size of the mean and variance differences, confirm the units, and check the implementation against the model assumptions. For a mean discrepancy, the Monte Carlo standard error gives a direct scale for the expected fluctuation. For a variance discrepancy, its sampling variability depends on the fourth moment; skewness and rare large losses can make a sample variance unstable. When that fourth moment is unavailable or the loss is heavy-tailed, avoid claiming that a simple fixed tolerance has a universal justification.

Worked Example: Detecting a Severity-Scale Error

A hypothetical model specifies $N\sim\operatorname{Poisson}(1)$, independent of exponential claim payments with mean $\$100$. A simulation with $R=10{,}000$ runs reports a mean aggregate of $\$200$ and sample variance $80{,}000$ dollars squared. Assess whether these summaries are consistent with the specified model, and identify a plausible error to investigate.

Method. Compare the simulated summaries with both theoretical moments and their sampling scales. This avoids dismissing a large discrepancy as ordinary Monte Carlo variation.

For an exponential severity with mean $\$100$, $E[X]=\$100$ and $\operatorname{Var}(X)=10{,}000$ dollars squared. The Poisson frequency has mean and variance $1$. Thus,

E[S]=1(100)=\$100, \qquad \operatorname{Var}(S)=1(10{,}000)+1(100^2)=20{,}000\text{ dollars}^2.

Under the specified model, the standard error of the simulated mean is $\sqrt{20{,}000/10{,}000}=\$1.41$. The reported mean is $\$100$ above its target, or about $100/1.41=70.7$ standard errors. That is not a credible ordinary fluctuation under the stated model.

The variance discrepancy is also substantial. For this compound Poisson model, the cumulants satisfy $\kappa_r(S)=\lambda E[X^r]$. The exponential moments give $\kappa_2=2(100^2)=20{,}000$ and $\kappa_4=24(100^4)=2{,}400{,}000{,}000$. The fourth central moment is $\mu_4=\kappa_4+3\kappa_2^2=3{,}600{,}000{,}000$ dollars to the fourth. For independent replicates, the exact variance of the unbiased sample variance is

\operatorname{Var}(s_S^2) =\frac{1}{R}\left[\mu_4-\frac{R-3}{R-1}\operatorname{Var}(S)^2\right].

Substituting $R=10{,}000$ and $\operatorname{Var}(S)=20{,}000$ gives a standard deviation for $s_S^2$ of approximately $566$ dollars squared. The reported variance exceeds its target by $60{,}000$, over 100 such standard deviations. Both reported summaries are instead consistent with doubling each severity: that would double the aggregate mean to $\$200$ and multiply the aggregate variance by four to $80{,}000$ dollars squared. A severity multiplier applied twice is therefore a plausible error to investigate, not a conclusion proven by the summaries alone. Check: The discrepancy is far too large to dismiss as routine simulation noise, and the mean/variance pattern points to a possible severity-scale error.

Common Traps in Moment Checks

Dropping frequency variation from the aggregate variance. The incorrect calculation $\operatorname{Var}(S)=E[N]\operatorname{Var}(X)$ omits $\operatorname{Var}(N)(E[X])^2$. It is only correct when the frequency variance is zero or the severity mean is zero.
Using the wrong frequency distribution. A binomial count has variance $mp(1-p)$; a Poisson count has variance equal to its mean. Replacing one with the other changes the theoretical aggregate variance.
Comparing unlike quantities. A variance is in squared currency units, not dollars. Check that severity amounts, simulated totals, and theoretical moments use the same currency scale.
Expecting a short run to match exactly. Finite-run summaries fluctuate. Compare a mean difference with its standard error and treat a variance discrepancy cautiously, particularly for skewed losses.
Assuming moment agreement proves correctness. A simulation can match the first two moments while still having the wrong tail or other distributional features. Moment checks are useful diagnostics, not a complete validation.

Key takeaway: Calculate the theoretical aggregate mean and variance from the specified frequency and severity moments, then compare them with summaries of independent simulated aggregates. Use sampling error to judge discrepancies, and treat a large, patterned mismatch as a reason to audit model parameters, units, frequency generation, severity scaling, and aggregation.

Check Your Understanding

For each problem, compare the requested simulated or theoretical quantity with the compound-model formulas. Answers are given for checking.

A compound Poisson model has frequency parameter $\lambda=3$, with severity mean $4$ and severity variance $9$. Find the aggregate mean and variance. (Answer: mean $12$; variance $75$.)
A model has $N\sim\operatorname{Binomial}(5,0.10)$ and independent severity with mean $20$ and variance $100$. Find the aggregate mean and variance. (Answer: mean $10$; variance $230$.)
Independent simulated aggregates have theoretical mean $50$, theoretical variance $900$, and run size $R=400$. Find the standard error of the simulated mean. (Answer: $1.5$.)
A compound Poisson model has $\lambda=0.5$ and a fixed payment of $\$200$ per claim. Find the aggregate mean and variance. (Answer: mean $\$100$; variance $20{,}000$ dollars squared.)
A model's theoretical aggregate mean is $40$ and variance is $1{,}600$. A run of $400$ independent aggregates has sample mean $42$. How many standard errors from the target is this, and how would you describe the difference? (Answer: one standard error; a difference of this size is not by itself concerning.)
Four simulated aggregate totals are $0,0,10,10$. Find their sample mean and unbiased sample variance. (Answer: mean $5$; sample variance $100/3$, or approximately $33.33$.)

What You'll Learn