Why Reproducibility Is a Model-Control Requirement
A capital-model result is useful only if its calculation can be reconstructed. A reviewer should be able to identify the inputs used, follow each transformation, establish which calculation logic ran, and obtain the same result under the same controlled conditions. Without that evidence, a change in reported capital could reflect a genuine input or model change—or an unnoticed file, code, or execution difference.
Reproducibility is not the same as proving that a model is conceptually appropriate. It is a controlled way to establish what calculation was performed and whether its output follows from the recorded inputs and methods. The material here is aligned with the CAS Exam 9 (Risk Management for Actuaries) syllabus.
Specify the Run Before Calculating
A reproducible calculation begins with an explicit run specification, not merely a saved output workbook. At a minimum, the run record should identify the model purpose and measurement basis, the exposure and parameter versions, the code or calculation-engine version, the run date, and the output definition. These items make it possible to distinguish a change in the modeled portfolio from a change in the calculation itself.
For each input, record its source, effective date, units, and version or immutable identifier. A file name alone is weak evidence: a file called “current_inputs” can be overwritten without preserving what a previous run used. A checksum or hash can help identify the exact file contents. Similarly, preserve the actual calculation code and configuration, rather than relying only on a version label that may not uniquely identify them.
Transformations should be documented as ordered, testable steps. State, for example, how records are mapped to model segments, how units or currencies are converted, how missing values are treated, and at what stage values are rounded. A transformation specification should make clear both the intended operation and the resulting data that feed the next step. Where possible, retain intermediate totals so that a reviewer can reconcile the final result without guessing which operation was applied.
Record versioned exposure, parameter, and configuration data, along with units and effective dates. Do not allow the run to depend on an unrecorded file that can change during execution.
Apply documented schema checks and transformations in a defined order. Record counts, totals, mappings, and any permitted exclusions or adjustments.
Record the code or engine version, configuration, relevant software environment, and, for stochastic calculations, the random-number algorithm and seed or stored draws.
Compare key intermediate values and final outputs with the run record. Store the output alongside the run identifiers, warnings, exceptions, and completion status.
Deterministic Reruns and Randomness
A deterministic calculation returns the same output when the same defined inputs and calculation logic are applied in the same way. In abstract form, write a calculation as \(C=g(D,\theta,\mathcal{M})\), where \(D\) denotes data, \(\theta\) denotes parameters, \(\mathcal{M}\) denotes the specified model and calculation logic, and \(C\) is the output. If all of these are fixed and the implementation is deterministic, rerunning \(g\) produces the same \(C\).
This statement has a practical qualification: the recorded inputs must include anything that can affect the calculation. Examples include the order in which records are processed, software settings that change numerical precision, or an external data lookup. A seed alone may not reproduce a simulation if the random-number algorithm, software implementation, or order of random draws has changed. Strong controls either preserve the draw sequence or identify the algorithm, seed, and execution conditions needed to regenerate it.
The implication follows directly from treating the model as a specified function: identical arguments to the same deterministic function give the same result. If two runs differ, at least one relevant argument, calculation step, or implementation detail must differ—or an error has occurred. This is why run identifiers and intermediate values matter: they help locate the difference instead of merely showing that the final outputs disagree.
Worked Example: Reconciling Two Versioned Runs
A hypothetical model reports capital, in thousands of dollars, using \(C=k\sum_j E_jr_j+A\), where \(E_j\) is exposure, \(r_j\) is a rate, \(k\) is a portfolio factor, and \(A\) is an allowance. The controlled input set contains exposures \(E_1=200\), \(E_2=100\), rates \(r_1=0.010\), \(r_2=0.040\), and allowance \(A=2\). Run A uses factor \(k=1.50\); a later run uses \(k=1.30\). Reconcile both outputs and identify a common reporting trap.
Method. Recompute the common exposure-rate subtotal first, then apply the factor and allowance specified for each version. This separates a parameter change from a data change.
Step 1: Calculate the common subtotal. The first contribution is \(200(0.010)=2\), and the second is \(100(0.040)=4\). Thus the exposure-rate subtotal is \(2+4=6\) thousand dollars.
Step 2: Calculate Run A. Apply its factor and allowance: \(6(1.50)=9\), then \(9+2=11\). Run A’s output is \(11.00\) thousand dollars.
Step 3: Calculate the later run. Its factor is \(1.30\), so \(6(1.30)=7.8\), and \(7.8+2=9.8\). The later output is \(9.80\) thousand dollars.
Check. The exposure data and allowance are unchanged. The factor falls by \(0.20\), reducing the output by \(6(0.20)=1.2\). Indeed, \(11.0-9.8=1.2\).
Common mistake avoided. Applying the first run’s factor to the second run’s logged inputs, or reporting \(10.6\) for the later run, would not reconcile. The calculation from the stated later-run values is \(6(1.30)+2=9.8\), not \(10.6\).
Documenting Transformations and Handling Exceptions
A transformation is part of the calculation, not administrative housekeeping. Suppose an engine converts exposure from dollars to units of one thousand dollars before multiplying by a rate stated per thousand dollars. The conversion and the rate basis must both be explicit. Otherwise, a value can be numerically processed without being meaningful. The same principle applies to segment mappings, aggregation rules, and rounding.
Validation should occur before a calculation produces an apparently complete result. Examples of exceptions include a missing rate for a required segment, an invalid identifier, an exposure outside an allowed range, or a failed unit check. A controlled process defines whether each condition stops the run, permits a documented exclusion, or generates a warning that must be reviewed. It should not silently replace a missing value with zero or carry forward a stale value unless that treatment is explicitly approved, documented, and appropriate to the model.
For a hard failure, the system should identify the affected record or field, explain the failed rule, and mark the run as incomplete. It should not publish a partial total as if it were a valid final output. For an allowed warning, retain the warning and its disposition in the run record. This makes an exception visible to the person interpreting the output and allows the treatment to be reconstructed.
Worked Example: Reproducing a Small Stochastic Run
A hypothetical simulation has four equally weighted trials. The controlled run record contains the following pairs of uniform draws, generated using a stated random-number algorithm and seed: \((0.10,0.20)\), \((0.40,0.80)\), \((0.70,0.10)\), and \((0.20,0.60)\). In each trial, a claim occurs if the first draw is less than \(0.50\). If a claim occurs, its severity is 10 thousand dollars when the second draw is less than \(0.50\), and 20 thousand dollars otherwise. Calculate the trial losses and their average.
Method. Apply the occurrence rule first, then apply the severity rule only to trials with a claim. Preserve the draw-to-trial ordering in the run record.
Trial 1. Since \(0.10<0.50\), a claim occurs. Since \(0.20<0.50\), its loss is 10.
Trial 2. Since \(0.40<0.50\), a claim occurs. Since \(0.80\) is not less than \(0.50\), its loss is 20.
Trial 3. Since \(0.70\) is not less than \(0.50\), no claim occurs. Its loss is 0; the severity draw does not affect this trial.
Trial 4. Since \(0.20<0.50\), a claim occurs. Since \(0.60\) is not less than \(0.50\), its loss is 20.
The aggregate is 50 thousand dollars and the mean is 12.5 thousand dollars per trial. Check: three trials have claims, with losses 10, 20, and 20; their sum is 50. A common mistake is to treat the seed as a complete reproducibility record by itself. Here, the algorithm and mapping of draws to trials are also specified; changing the algorithm or draw order can change the losses even if a seed label is retained.
Diagnosing a Reproducibility Failure
When a reported output fails to reconcile, first separate three questions: are the recorded inputs complete and identical, were the documented transformations applied, and did the executed calculation use the documented logic and settings? Compare run identifiers and intermediate values in that order. A final-output discrepancy identifies a control problem, but does not by itself establish which step caused it.
Worked Example: Finding an Execution-Record Mismatch
A hypothetical model uses the same formula and units as the first example. Its exposure and rate records are \(E_1=120\), \(r_1=0.020\), \(E_2=80\), \(r_2=0.050\), with allowance \(A=3\), all in thousands as applicable. The run manifest says the factor is \(1.10\), but the reported output is \(10.68\). Reconcile the logged calculation and diagnose what the evidence supports.
Method. Recalculate from the manifest, then compare the reported output with the result implied by the logged inputs. Do not change an input merely to force agreement.
Step 1: Recompute the exposure-rate subtotal. The first contribution is \(120(0.020)=2.4\); the second is \(80(0.050)=4.0\). The subtotal is \(2.4+4.0=6.4\).
Step 2: Apply the manifest factor. The documented factor gives \(6.4(1.10)=7.04\). Adding the allowance gives \(7.04+3=10.04\). Thus the manifest implies \(10.04\), not the reported \(10.68\).
Step 3: Check the reported value against a possible execution factor. If the executed factor were \(1.20\), the calculation would be \(6.4(1.20)+3=7.68+3=10.68\), exactly the reported amount. This is evidence consistent with an execution-factor mismatch.
Conclusion and control response. The report cannot be certified as a run using the stated \(1.10\) factor. The run should be flagged, its execution configuration and retained logs checked, and a controlled rerun performed after the actual factor is established. The arithmetic alone does not prove how the mismatch occurred; it shows that the reported result does not follow from the manifest. Common mistake avoided: silently replacing \(1.10\) with \(1.20\) in the manifest after seeing the output would destroy the evidence of the discrepancy.
Common Reproducibility Traps
- Overwriting source files. A familiar file name does not establish that the same contents were used. Retain version identifiers or immutable file fingerprints.
- Recording only final results. Without intermediate totals, an error in mapping, conversion, or aggregation can be difficult to locate.
- Using undocumented defaults. Defaults for missing values, units, or rounding can change an output while leaving the calculation apparently complete.
- Assuming a seed is sufficient. For stochastic calculations, the algorithm, draw ordering, and relevant execution settings can also matter.
- Publishing a partial run as complete. A failed validation should result in a visible exception or incomplete status, not an unexplained total.
Check Your Understanding
Work each problem by stating the calculation or control check you apply. Amounts are in thousands of dollars unless stated otherwise.
- A model uses \(C=k\sum_j E_jr_j+A\). It has exposures 150 and 50, rates 0.020 and 0.060, factor 1.25, and allowance 1.5. Calculate the output. (Answer: 9.00 thousand dollars)
- A run records a subtotal of 4.9, factor 1.40, and allowance 0.5, but reports 7.50. Recalculate the logged result and state whether it reconciles. (Answer: \(4.9(1.40)+0.5=7.36\) thousand dollars; it does not reconcile.)
- A source exposure is 2,400,000 dollars. The documented transformation converts it to thousands of dollars, then applies a rate of 0.003 per thousand dollars. Find the contribution and identify the relevant unit check. (Answer: 2,400 thousand dollars times 0.003 = 7.20 thousand dollars; confirm the rate is per thousand dollars, not per dollar.)
- In a four-trial simulation, a claim occurs when \(U_1<0.40\); conditional severity is 5 when \(U_2<0.50\), otherwise 15. Draw pairs are \((0.20,0.10)\), \((0.60,0.20)\), \((0.30,0.80)\), and \((0.90,0.90)\). Find the aggregate and mean trial loss. (Answer: losses are 5, 0, 15, and 0; aggregate 20 thousand dollars and mean 5 thousand dollars.)
- A required segment has exposure 40 but no mapped rate. The engine’s documented rule says missing rates are fatal errors. What should the run do, and why is assigning a zero rate without review unacceptable? (Answer: Stop and record an incomplete run with the segment and failed rule identified; silently using zero changes the model input and can understate the output.)