Checking Simulated Frequencies And Probabilities

Why Simulated Frequencies Vary

A simulation is not expected to reproduce theoretical probabilities exactly in a finite run. Even when the model and random-number generation are correct, independently generated outcomes vary from one run to another. A useful check therefore asks not only whether a simulated frequency differs from its theoretical value, but also whether the difference is large relative to the variation that ordinary sampling would produce.

For example, if a hypothetical portfolio model gives a probability of 0.20 that a policy has at least one claim, then 1,000 simulated policies are expected to include 200 such outcomes. The realized count need not be exactly 200. The relevant comparison is between the observed count and the distribution of that count under the assumed model.

Definition: Suppose a simulation produces $n$ independent outcomes, and an event has theoretical probability $p$ on each outcome. The event count $K$ has a binomial distribution, $K\sim\operatorname{Binomial}(n,p)$. Its expected count is $np$, its variance is $np(1-p)$, and its standard deviation is $\sqrt{np(1-p)}$. The simulated relative frequency is $\hat p=K/n$, with expected value $p$ and standard error $\sqrt{p(1-p)/n}$.

The binomial model applies when each simulated outcome either belongs to the event or does not, the event probability is the same for every outcome, and the outcomes are independent. These conditions are natural for independent replications of a fixed model. If simulated outcomes share random inputs or depend on one another, the binomial variance formula need not apply.

From an Event Count to a Standardized Deviation

The standard deviation gives a scale for judging the gap between an observed count and its expectation. A convenient summary is the standardized deviation, often called a z-score:

z=\frac{K-np}{\sqrt{np(1-p)}}.

A value near zero means the count is close to its expected value in standard-deviation units. A larger absolute value means a more unusual deviation under the binomial model. When $np$ and $n(1-p)$ are sufficiently large, the binomial distribution is often approximated by a normal distribution, so $z$ can be interpreted using the standard normal distribution. This is an approximation, not an exact probability calculation; for small expected counts or probabilities near zero or one, use an exact binomial calculation or a suitable method for the discrete distribution.

Derivation: Let $I_j$ equal 1 if simulated outcome $j$ is in the event and 0 otherwise. Then $K=\sum_{j=1}^n I_j$. Each indicator has $E[I_j]=p$ and $\operatorname{Var}(I_j)=p(1-p)$. Independence makes the covariances between distinct indicators zero, so $E[K]=np$ and $\operatorname{Var}(K)=np(1-p)$. Dividing $K$ by $n$ divides its variance by $n^2$, giving $\operatorname{Var}(\hat p)=p(1-p)/n$.

An approximate 95% range for the simulated relative frequency, when the normal approximation is suitable, is

p\ \pm\ 1.96\sqrt{\frac{p(1-p)}{n}}.

This is a reference range for the variability of repeated simulation runs under the specified model. It is not a guarantee that the observed frequency will fall inside it, and one run outside the range does not by itself prove a simulation error.

Worked Example: Checking the Zero-Claim Frequency

A hypothetical model simulates the claim count $N$ for each policy, with $N\sim\operatorname{Poisson}(1)$. In 400 independent policy simulations, 151 policies have zero claims. Check this frequency against the model. Use probabilities to four decimal places and the standardized deviation to two decimal places.

Method and model assumptions. For a Poisson random variable with mean 1, the probability of zero claims is $P(N=0)=e^{-1}$. The 400 policy outcomes are assumed independent and generated from the same Poisson model. Thus the number $K$ with zero claims is binomial with $n=400$ and $p=e^{-1}$.

Step 1: Find the theoretical event probability.

p=P(N=0)=e^{-1}=0.3679.

Step 2: Calculate the expected count and its standard deviation. Using the unrounded probability for the calculations,

E[K]=400e^{-1}=147.1516,

\operatorname{SD}(K) =\sqrt{400e^{-1}(1-e^{-1})} =\sqrt{93.0176} =9.64.

Step 3: Standardize the observed count.

z=\frac{151-147.1516}{9.6446}=0.40.

The observed relative frequency is $151/400=0.3775$, which is 0.0096 above the theoretical probability. Its standardized deviation is only 0.40, so this difference is consistent with ordinary simulation variation.

Common mistake avoided. It would be misleading to call 151 a poor result merely because it is not the expected count 147.15. The standard deviation is about 9.64 policies, so a difference of about four policies is small on the relevant scale. This check is a plausibility assessment, not proof that every part of the simulation is correct.

Checking Several Mutually Exclusive Frequencies

A simulation often records a category for each outcome, such as the claim count being zero, one, or at least two. The category counts are related: if one category has more outcomes, fewer are available for the others. Their frequencies should not be treated as independent binomial counts.

Suppose there are $k$ mutually exclusive categories with probabilities $p_1,\ldots,p_k$, where $\sum_{j=1}^{k}p_j=1$. For $n$ independent simulations, let $K_j$ be the count in category $j$. Then the vector of counts has a multinomial distribution. Each expected count is $np_j$, and

\operatorname{Var}(K_j)=np_j(1-p_j), \qquad \operatorname{Cov}(K_j,K_l)=-np_jp_l\quad (j\ne l).

The negative covariance follows because the total count is fixed at $n$. An outcome assigned to one category cannot also be assigned to another. When expected category counts are adequate for a chi-square approximation, an overall discrepancy can be summarized by Pearson’s statistic:

X^2=\sum_{j=1}^{k}\frac{(K_j-np_j)^2}{np_j}.

If all category probabilities are specified in advance and the expected counts are sufficiently large, this statistic is approximately chi-square with $k-1$ degrees of freedom. If probabilities have been estimated from the same data, the degrees of freedom generally need adjustment for the fitted parameters. Small expected counts make the usual approximation unreliable; consider combining substantively compatible categories or using an appropriate exact approach instead.

Worked Example: Checking Simulated Claim-Count Categories

A hypothetical model has a claim count $N\sim\operatorname{Binomial}(2,0.5)$. In 400 independent simulations, the counts of outcomes $N=0$, $N=1$, and $N=2$ are 112, 183, and 105. Assess the category frequencies using Pearson’s statistic. The model probabilities are treated as specified, not estimated from these results.

Method and model assumptions. Use the binomial probability formula to obtain the three mutually exclusive category probabilities. The simulation outcomes are independent, and the category counts therefore have a multinomial distribution.

Step 1: Calculate the category probabilities.

P(N=0)=\binom{2}{0}(0.5)^0(0.5)^2=0.25, \quad P(N=1)=\binom{2}{1}(0.5)^1(0.5)^1=0.50, \quad P(N=2)=\binom{2}{2}(0.5)^2(0.5)^0=0.25.

Step 2: Find expected counts. Multiplying each probability by 400 gives expected counts of 100, 200, and 100. All are large enough for the usual chi-square approximation.

Step 3: Calculate the statistic.

X^2= \frac{(112-100)^2}{100} +\frac{(183-200)^2}{200} +\frac{(105-100)^2}{100} =1.440+1.445+0.250 =3.135.

There are three categories and no fitted parameters, so the approximate degrees of freedom are $3-1=2$. The upper-tail probability for a chi-square variable with 2 degrees of freedom at 3.135 is approximately $e^{-3.135/2}=0.2085$. The statistic is not unusually large at a 5% reference level; these counts are reasonably consistent with the specified probabilities.

Common mistake avoided. Do not treat the three category counts as independent binomial observations and add separate conclusions as if their total could vary. They must sum to 400, and the multinomial check accounts for that constraint.

Checking a Selected Event Probability

The same approach applies when the event is defined by a condition on a simulated loss, such as exceeding a threshold. First calculate the theoretical event probability from the assumed loss distribution. Then compare the observed number of exceedances with the binomial expectation and variability. Be careful that the event definition matches the model exactly: exceeding a threshold is not the same as equaling or exceeding it, although the distinction has probability zero for a continuous distribution.

Worked Example: Simulated Losses Above a Threshold

A hypothetical severity model has exponential losses $X$ with mean \$1,000, so its rate parameter is $1/1000$ per dollar. A simulation generates 1,000 independent losses, of which 246 exceed \$1,500. Assess the exceedance frequency using a 95% approximate range. Give probabilities to four decimal places.

Method and model assumptions. For an exponential loss with mean $\theta$, the survival probability is $P(X>x)=e^{-x/\theta}$ for $x\geq 0$. Here $\theta=\$1{,}000$, and each simulated loss is assumed independent and generated from this same distribution.

Step 1: Calculate the theoretical exceedance probability.

p=P(X>1500)=e^{-1500/1000}=e^{-1.5}=0.2231.

Step 2: Calculate the expected count and standard deviation.

E[K]=1000e^{-1.5}=223.1302,

\operatorname{SD}(K) =\sqrt{1000e^{-1.5}(1-e^{-1.5})} =13.17.

Step 3: Compare the observed count and frequency. The observed frequency is $246/1000=0.2460$. Its standardized deviation is

z=\frac{246-223.1302}{13.166\ldots}=1.74.

The approximate 95% range for the frequency is

0.2231\ \pm\ 1.96\sqrt{\frac{0.2231302(1-0.2231302)}{1000}} \ \approx\ [0.1973,\ 0.2489].

The observed frequency 0.2460 lies inside this range. It is above the theoretical probability, but the difference is not especially large relative to the expected variation in 1,000 trials.

Common mistake avoided. Comparing 0.2460 with 0.2231 without accounting for sample size can make a difference of 0.0229 appear more decisive than it is. With 1,000 simulations, the standard error is about 0.0132, making the difference about 1.74 standard errors.

Interpreting a Simulation Check

A frequency check is evidence about consistency with a model, not a complete validation of a simulation. A result within an approximate range does not establish that the model, implementation, or random-number generation is correct. Conversely, a result outside a nominal 95% range can occur by chance. If many events or categories are checked, the chance that at least one looks unusual increases; isolated 5% flags should therefore be interpreted with the number of checks in mind.

When checking results, keep these distinctions clear:

Expected value is not a target that every run must hit. It is the long-run average count or frequency over repeated runs.
Use the right scale for the comparison. A count deviation is judged against $\sqrt{np(1-p)}$; a frequency deviation is judged against $\sqrt{p(1-p)/n}$.
Respect the structure of the outcomes. A single event count is binomial under the stated assumptions; counts across mutually exclusive categories are multinomial and dependent.
Check the approximation conditions. Normal and chi-square approximations can be poor when expected counts are small.
Keep the event definition and model aligned. A threshold, unit, or category boundary interpreted differently from the model changes the theoretical probability being checked.

These checks are useful when reviewing simulated claim counts, loss exceedances, or other event probabilities in actuarial models. The central question is always whether the observed difference is substantial relative to the sampling variation expected from the number of independent simulations. This approach is aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.

Check Your Understanding

For each problem, calculate using the stated model and assumptions. Round standardized deviations to two decimal places where needed.

A model gives a probability of 0.08 that a simulated policy has a claim. Among 250 independent policies, 27 have claims. Find the expected count, standard deviation, and standardized deviation. (Answer: Expected count $=20$; standard deviation $=\sqrt{18.4}=4.29$; $z=1.63$.)
For $N\sim\operatorname{Poisson}(2)$, find the probability that a policy has at least 3 claims. In 500 independent simulations, 149 policies have at least 3 claims. Find the expected count, its standard deviation, and the standardized deviation. (Answer: $p=1-5e^{-2}=0.3233$; expected count $=161.66$; standard deviation $\approx10.46$; $z\approx-1.21$.)
A model has three specified category probabilities 0.10, 0.30, and 0.60. In 200 simulations the observed counts are 17, 61, and 122. Calculate Pearson’s statistic and its degrees of freedom, assuming the approximation is appropriate. (Answer: Expected counts $20,60,120$; $X^2=0.50$; 2 degrees of freedom.)
Losses are independent exponential random variables with mean \$500. In 1,000 simulated losses, 112 exceed \$1,000. Find the theoretical exceedance probability, expected count, and standardized deviation. (Answer: $p=e^{-2}=0.1353$; expected count $=135.34$; standard deviation $\approx10.82$; $z\approx-2.16$.)
An event has theoretical probability 0.40, and 100 independent simulations produce 50 event occurrences. Find the expected relative frequency, its standard error, and the standardized deviation of the observed count. (Answer: Expected relative frequency $=0.40$; standard error $=\sqrt{0.4(0.6)/100}\approx0.0490$; $z=2.04$.)

What You'll Learn