Why Reproducibility Matters in Simulation
A simulation result can look unusual for two very different reasons: the random variation may be substantial, or the program may not implement the intended model. A fixed seed makes a pseudorandom sequence repeatable, so a particular input can be traced through the code and its output checked. It does not establish that the simulation is correct, nor does it make a small simulated sample representative.
For actuarial simulation, a useful diagnostic asks whether each step maps its inputs to outputs as specified. Check the random-number generator, the transformation from uniforms to model variates, and the construction of frequency-severity outcomes separately. This isolates a fault more effectively than inspecting only a final aggregate loss.
Reproducibility can depend on implementation details. Changing the random-number generator, software version, parallelization, or order in which random numbers are requested may change the outputs even when the seed is unchanged. For a strong audit trail, record the generator and its settings as well as the seed. When debugging, it is often useful to pass a short, explicit list of uniform inputs into the transformation being tested; this separates transformation logic from the random-number generator itself.
A Staged Diagnostic Method
Begin with the smallest test that can expose the suspected failure. For a transformation, use hand-selected input values with known outputs. For a distributional check, compare simulated results with properties of the target model, while allowing for sampling variation. For a compound model, retain the policy or trial identifiers so that frequency and severity inputs can be traced to the correct outcome.
Record the distribution, parameter convention, support, and intended independence assumptions. For example, distinguish an exponential rate from its mean or scale.
Use a documented seed for a reproducible run, or supply known uniforms directly to test a transformation. Keep a trace of which inputs feed each output.
Verify a few input-output pairs, category boundaries, nonnegative values, and the number of severity observations generated for each frequency.
Compare empirical frequencies, means, variances, or dependence patterns with the target model. Treat a discrepancy as a clue to investigate, not automatic proof of a defect.
Change one component at a time. If a correction changes the output in the predicted way while other inputs stay fixed, the source of the discrepancy becomes clearer.
For a random variable \(X\), its support is the set of points for which every neighborhood has positive probability. For a purely discrete distribution, the support is the closure of the set of points with positive probability mass. It equals that set when there are no additional limit points outside it, as for a finite-valued distribution. This distinction matters when testing whether simulated values can occur: checking only the values with positive point mass can be insufficient for a discrete distribution with infinitely many such values.
Worked Examples
Worked Example: Detecting an Exponential Parameterization Error
A hypothetical claim severity generator is intended to produce an exponential loss \(X\) with rate \(\lambda=0.5\) per dollar, so its mean is \(1/\lambda=2\) dollars. The intended inverse transform is \(X=-\ln(1-U)/\lambda\), where \(U\) is uniform on \((0,1)\). Test the transformation at \(U=0.25, 0.50,\) and \(0.75\). A programmer may have entered the mean as though it were the rate.
Method. Use fixed test inputs and compare the outputs with the inverse-transform formula. This checks the mapping itself rather than relying on a sample average.
For the intended rate \(\lambda=0.5\), the three outputs are:
The outputs are nonnegative and increase with \(U\), as this inverse cumulative distribution function should. If the programmer instead uses the mean \(2\) as the rate, the outputs become \(0.1438, 0.3466,\) and \(0.6931\), respectively: each is one-quarter of the intended output because the denominator has changed from \(0.5\) to \(2\). The resulting model has rate \(2\) and mean \(0.5\), not rate \(0.5\) and mean \(2\).
A further check uses the target cumulative distribution function. At \(x=2\), the intended probability is \(F_X(2)=1-e^{-0.5(2)}=1-e^{-1}=0.6321\), to four decimal places. With the erroneous rate \(2\), it is \(1-e^{-2(2)}=0.9817\). This large difference confirms that the parameterization error changes the distribution. Common mistake avoided: treating a short-run sample mean as a definitive test. With only three simulated values, the mean need not be close to 2; the known-input mapping gives a more direct test of the suspected coding error.
Worked Example: Checking a Discrete Inverse Transform
A hypothetical payment variable \(X\), measured in dollars, has \(P(X=0)=0.5\), \(P(X=100)=0.3\), and \(P(X=500)=0.2\). The intended inverse-transform intervals are \([0,0.5)\) for 0, \([0.5,0.8)\) for 100, and \([0.8,1)\) for 500. Apply the rule to fixed inputs \(0.12,0.55,0.79,0.81,0.96\), and assess what a short run can establish.
Method. Trace each uniform to its interval, then compare the observed counts with the model probabilities without treating five observations as a precise goodness-of-fit test.
The mapping is \(0.12\mapsto0\), \(0.55\mapsto100\), \(0.79\mapsto100\), \(0.81\mapsto500\), and \(0.96\mapsto500\). Thus the output sequence is \(0,100,100,500,500\). Its counts are 1, 2, and 2, giving empirical proportions \(0.2,0.4,\) and \(0.4\), respectively. These do not equal the target probabilities \(0.5,0.3,\) and \(0.2\); with only five draws, that difference alone does not prove a bug.
The interval trace is more decisive for implementation. For example, an input exactly equal to \(0.5\) should map to 100 under the stated half-open intervals, and an input exactly equal to \(0.8\) should map to 500. Testing such boundaries can reveal use of the wrong inequality or a cumulative-probability cutoff. The outputs must also lie in \(\{0,100,500\}\), the support of this finite-valued model. Common mistake avoided: declaring the generator wrong merely because a tiny empirical proportion differs from its target. A deterministic boundary failure is evidence about the code; ordinary count variation is not.
Worked Example: Finding Dependence From a Reused Uniform Stream
For each hypothetical policy, the intended frequency \(N\) is Bernoulli with \(P(N=1)=P(N=0)=0.5\). Independently, a claim severity \(X\) is 100 dollars with probability 0.5 and 10 dollars with probability 0.5. The policy aggregate is \(L=NX\). Use the four frequency uniforms \(0.1,0.3,0.7,0.9\) and four severity uniforms \(0.8,0.2,0.6,0.4\). For this test, set \(N=1\) when its uniform is below 0.5 and \(X=100\) when its uniform is below 0.5.
Method. Trace two intended streams separately, calculate the policy aggregates, and compare with what happens if the frequency uniform is mistakenly reused for severity.
Using separate streams gives frequencies \(N=(1,1,0,0)\) and severities \(X=(10,100,10,100)\). Therefore the aggregates are \(L=(10,100,0,0)\). Their mean is \((10+100+0+0)/4=27.5\) dollars per policy.
If the frequency uniforms are mistakenly reused, the severities become \(X=(100,100,10,10)\). The aggregates are then \(L=(100,100,0,0)\), with mean \((100+100+0+0)/4=50\) dollars per policy. The paired values now display a clear dependence pattern: every policy with a claim has severity 100, while every policy without a claim has severity 10.
The model-level reason is that independence requires \(E[NX]=E[N]E[X]\). Here \(E[N]=0.5\) and \(E[X]=0.5(100)+0.5(10)=55\), so the intended \(E[L]=E[NX]=0.5(55)=27.5\) dollars per policy. Reusing one uniform makes \(N=1\) exactly when \(X=100\), giving \(E[NX]=0.5(100)+0.5(0)=50\). Common mistake avoided: assuming that each marginal distribution is correct, so the joint simulation must be correct. Both marginal distributions can pass checks while an unintended dependence changes aggregate losses.
Interpreting Diagnostic Evidence
A fixed seed is especially useful when a defect appears intermittently: it lets the same trace be examined repeatedly while code is changed. It is also useful to replace a large run with a small one whose inputs and expected outputs can be calculated by hand. If an output violates a deterministic requirement, such as a nonnegative loss generator returning a negative value, investigate the transformation or data handling before increasing the number of simulations.
Statistical checks answer a different question. If \(N\) is Bernoulli with probability \(p\), the sample proportion \(\hat p\) from \(m\) independent simulations has expected value \(p\) and variance \(p(1-p)/m\). To see why, write \(\hat p=m^{-1}\sum_{j=1}^{m}I_j\), where each \(I_j\) is 1 when the event occurs and 0 otherwise. Each indicator has mean \(p\) and variance \(p(1-p)\); independence makes the variance of the sum \(m p(1-p)\), and dividing by \(m^2\) gives the stated result. Therefore, a sample proportion need not equal \(p\), particularly when \(m\) is small.
Repeated runs with different seeds can show whether a discrepancy persists, but they are not a substitute for checking the implementation. If the same incorrect parameterization is used in every run, changing the seed will not repair it. Conversely, one unusual sample may disappear with another seed without any code change. Record the test, its expected behavior, the observed behavior, and the code change that resolves the discrepancy.
| Diagnostic evidence | What it can indicate | What it does not establish by itself |
|---|---|---|
| A fixed input maps to the wrong value | An incorrect transformation, interval, or parameter convention | That every other part of the simulation is correct |
| Values occur outside the target support | A coding, truncation, or transformation error | The exact location of that error |
| Empirical proportions differ from target probabilities | A possible distribution or sampling issue worth checking | A bug, especially in a small run |
| Frequency and severity appear associated | Possible shared-stream dependence or incorrect pairing | A defect solely from one small observed correlation |
The most reliable diagnosis combines local and aggregate checks. Test exact mappings and boundaries first; then check support and model summaries; finally examine dependence and the alignment of policy-level records. In frequency-severity simulation, retain enough identifiers to verify that each simulated count is paired with the severity observations generated for that same policy. A plausible total can conceal incorrect pairing, and an unusual total can arise from a correct random model.
Check Your Understanding
For each problem, distinguish a deterministic implementation check from evidence that can vary by chance.
- An exponential severity has intended rate \(\lambda=0.25\). For input \(U=0.5\), find the correct inverse-transform output and the output if the programmer mistakenly uses \(\lambda=4\). (Answer: Correct \(X=2.7726\); incorrect \(X=0.1733\), to four decimal places.)
- A discrete variable takes values 2 and 8 with probabilities 0.7 and 0.3. Its inverse-transform intervals are \([0,0.7)\) and \([0.7,1)\). What outputs should inputs 0.69 and 0.70 produce? (Answer: 2 and 8, respectively.)
- A Bernoulli event has probability 0.2 and is simulated independently 10 times. Must the observed proportion be exactly 0.2 for the generator to be correct? State the expected count and whether a count of 4 alone proves a bug. (Answer: No; expected count is 2, and a count of 4 alone does not prove a bug.)
- Frequency \(N\) and severity \(X\) are intended to be independent, with \(E[N]=0.4\) and \(E[X]=75\) dollars. Find the intended expected aggregate \(E[NX]\). What risk does reusing one uniform stream create? (Answer: \(30\) dollars; reuse can create unintended dependence and change \(E[NX]\).)
- A discrete generator assigns positive probability to each value \(1/n\) for \(n=1,2,\ldots\), and zero probability to 0. Is 0 in the support under the neighborhood definition? (Answer: Yes. Every neighborhood of 0 contains some \(1/n\) with positive probability, even though \(P(X=0)=0\).)