Benchmarking a Simulator Against Known Results

Why Benchmark Before Comparing Designs?

A simulator can produce smooth, plausible-looking summaries while implementing the wrong model, parameterization, or random mechanism. That is especially consequential in design comparisons: a programming error shared unevenly across designs can alter their apparent ranking, while an error shared by every design can make all the results misleading. Before using simulation output to compare sampling plans, check the implementation in cases where the correct answer is already known.

Benchmarking is a comparison between an output from the simulator and an independently established reference result. The reference may be an exact calculation, a theoretical distribution, or a limiting case in which random variation disappears. A benchmark is strongest when its answer is derived independently of the code being checked; comparing two routines that use the same mistaken formula may simply reproduce the same error.

A useful sequence is to check deterministic calculations first, then random generation against known distributions, and finally end-to-end operating characteristics such as estimator bias or interval coverage. State the assumptions behind each reference before interpreting the comparison. A disagreement may indicate a coding defect, an incorrect assumption in the benchmark, or ordinary simulation variability; these possibilities should not be conflated.

Start With a Limiting Case

A limiting case removes one or more sources of complexity while preserving a result that the implementation should still satisfy. For example, set random-effects variances and residual-error variance to zero in a model that supports those components. Subject-level quantities then reduce to their fixed-effect predictions. This does not validate every stochastic feature, but it isolates the deterministic model calculation and can expose errors in units, parameterization, or time dependence.

For a deterministic benchmark, choose a numerical tolerance based on the calculation and the implementation’s numerical precision. Do not use a broad tolerance simply because the output looks close. Conversely, do not demand exact bit-for-bit equality when two correct calculations can differ slightly because of floating-point arithmetic.

Use Analytic Distributions to Check Random Output

A stochastic output will generally not equal its theoretical expectation in a finite run. The appropriate question is whether the discrepancy is plausible under the known sampling distribution. Suppose \(R\) independent simulation replicates produce an indicator \(I_r\), equal to 1 when a specified event occurs and 0 otherwise. If the event probability is \(p\), then the count \(K=\sum_{r=1}^{R} I_r\) has a binomial distribution with parameters \(R\) and \(p\). The simulated event fraction \(\hat{p}=K/R\) has expectation \(p\) and standard error \(\sqrt{p(1-p)/R}\).

A standardized discrepancy is a diagnostic, not a pass/fail rule that makes all other checks unnecessary. Values near zero are compatible with the reference, while a large absolute value warrants investigation. If many probabilities, outputs, or scenarios are checked, some will look discrepant by chance; interpret the full set of checks rather than treating one isolated threshold crossing as conclusive.

Benchmark the Statistical Procedure, Not Just the Model Output

A simulator used to evaluate designs often generates data and then applies an estimator or interval procedure. In that setting, benchmark the complete procedure where an analytic result is available. Bias, variance, and coverage each assess different behavior: an estimator can have the correct average but the wrong variability, and intervals can have the wrong coverage even when point estimates appear reasonable.

For a simple reference, let each replicate contain \(n\) independent observations \(X_i\sim N(\mu,\sigma^2)\), with known \(\mu\) and \(\sigma\). The sample mean \(\bar{x}\) has expectation \(\mu\) and variance \(\sigma^2/n\). Across \(R\) independent simulated data sets, the average of the \(R\) sample means has variance \(\sigma^2/(nR)\). This follows because variances of independent means add:

Thus the Monte Carlo standard error of the average simulated estimate is \(\sigma/\sqrt{nR}\), under these assumptions. This quantifies how much the reported benchmark summary itself is expected to fluctuate across repeated simulation runs. It is distinct from the sampling standard deviation of one replicate’s estimate, \(\sigma/\sqrt{n}\).

Choosing and Interpreting Benchmarks

A good benchmark matches the layer of the simulator that needs scrutiny. An exact model calculation checks deterministic equations. A known probability distribution checks random generation and aggregation. A tractable estimation problem checks the combined data-generation and analysis steps. When no exact result is available for the full model, reduce complexity until a useful reference is available, or compare a component with an independently implemented reference calculation. Agreement between independent implementations is informative but weaker than agreement with a mathematical result, because both implementations can share the same misunderstanding.

For stochastic comparisons, calculate the relevant Monte Carlo standard error from the benchmark assumptions, then report the simulated result, theoretical target, and discrepancy in context. A large standardized discrepancy is a reason to investigate, not an automatic diagnosis of a coding defect. Check inputs, units, parameter transformations, replicate independence, and whether the analytic assumptions actually match the simulation. If the theoretical target was estimated rather than known exactly, its uncertainty also matters; treating an uncertain reference as exact can overstate disagreement.

Increasing \(R\) reduces Monte Carlo standard error at a rate proportional to \(1/\sqrt{R}\), but it does not repair a biased implementation. A result that is consistently wrong becomes more precisely wrong as the number of replicates grows. Conversely, a modest discrepancy in a small run can be ordinary sampling variation. Report enough information for another analyst to distinguish those cases: the target, observed summary, replicate count, assumptions, and uncertainty calculation.

Common Pitfalls and Limitations

• Using a visually plausible result as validation. Smooth plots and reasonable magnitudes do not establish that the intended model was implemented. Compare with a stated reference and quantify the discrepancy.
• Confusing Monte Carlo error with model-sampling variability. The standard deviation among replicate estimates describes variation of estimates across data sets; the Monte Carlo standard error describes uncertainty in a summary calculated from a finite number of those replicates.
• Demanding exact equality from random output. A finite simulation is not expected to equal its theoretical expectation. Use the reference distribution or Monte Carlo standard error to assess plausible variation.
• Assuming one successful benchmark validates every feature. A fixed-parameter concentration check does not test random-effects distributions, residual errors, adaptive decisions, or estimation algorithms that were not exercised.
• Choosing a tolerance after seeing the result. Define meaningful numerical tolerances and statistical comparisons from the model and calculation, not to make a surprising output appear acceptable.

Benchmarks are deliberately targeted, so their conclusions are local: they support confidence in the tested inputs, equations, and procedures, not every possible model configuration. They also do not replace checks of the implementation’s broader behavior. Their value is that they create clear reference points before computational results are used to support design decisions.