Testing Simulation Code With Invariants and Unit Tests

Why Test Simulation Code at the Point of Construction?

A simulation can run to completion and still be wrong. A dose may be assigned to the wrong time, a generated-data array may silently drop a subject, or a metric may use the wrong denominator. Such errors can produce plausible-looking summaries, so inspecting final results alone is not a reliable way to find them.

Software tests make intended behavior explicit and check it with small, controlled inputs. In this tutorial, an invariant is a condition that must hold for every valid object of a stated type, while a unit test checks the behavior of one small function or component using an input whose expected result is known. Invariants are useful for broad structural checks; unit tests are useful for checking specific calculations and edge cases. Neither replaces checking that the scientific model and estimand are appropriate.

The checks below assume that the simulation has a declared data schema, a declared meaning and unit for each parameter, and explicitly defined metrics. For example, a field called “time” is not testable in a meaningful way until its units and reference point are specified. Assertions should encode the intended design, not merely whatever the current code happens to produce.

Invariants for Dose-Time Records

A dose-time record commonly contains at least a subject identifier, a dose time, and an administered amount. Depending on the model, it may also contain route, formulation, compartment, or infusion duration. The allowed values depend on the simulation specification, so state those rules before writing the tests. For a simple hypothetical oral-dose schedule with times in hours from the first dose and amounts in milligrams, possible invariants are: each identifier refers to a simulated subject; times are finite and nonnegative; amounts are finite and positive; and records are ordered by subject and then by time. If the design allows simultaneous doses, ordering does not imply that times must be strictly increasing.

These checks detect malformed records, but cannot by themselves show that a valid-looking schedule is the schedule intended by the design. A test should therefore include both general invariants and specific expected schedule values. For floating-point time values, comparisons should allow a documented tolerance when the computation can introduce rounding.

Useful failure cases include a negative time, a missing amount, a non-finite value, an unknown subject identifier, an incorrect unit conversion, and duplicate records when duplicates are prohibited. Do not prohibit duplicate times automatically: two distinct administrations may intentionally occur at the same nominal time. The test must follow the design’s stated rules.

Dimensions and Parameter Transformations

Generated-data shape is part of the simulation contract. If a fixed design generates \(n\) subjects at \(m\) observation times per subject, a rectangular concentration array may be required to have \(n\) rows and \(m\) columns. If the simulator instead returns a long table, the expected number of rows is \(nm\), provided every subject has all \(m\) observations and no rows are intentionally omitted. Test the representation actually used; a correct row count does not establish that subject and time labels are correctly aligned.

Parameter transformations need similarly direct checks. A common example maps an unconstrained parameter \(\eta\) to a positive parameter \(\theta\) using \(\theta=\exp(\eta)\). The inverse is \(\eta=\log(\theta)\) for \(\theta>0\). These formulas imply testable properties: the transformed value is positive, and applying the inverse after the forward transformation returns the original value up to numerical precision. The test should also cover a known input-output pair, because two mutually incorrect functions could still pass a round-trip check.

For bounded parameters, the transformation must match the specified bounds. For example, if \(a<\theta<b\), a logistic mapping is \(\theta=a+(b-a)/(1+\exp(-\eta))\). Tests can check that representative finite values of \(\eta\) produce values strictly between \(a\) and \(b\), and that the inverse mapping recovers \(\eta\) within tolerance. Extreme inputs deserve attention: numerical overflow or rounding may make a computed value equal to a boundary even when the mathematical mapping is strictly interior. Such behavior should be handled deliberately rather than concealed by an overbroad test tolerance.

Unit Tests for Performance Metrics

Metric tests should use a small data set for which the answer can be calculated by hand. Let \(\theta\) be a target parameter and let \(\hat{\theta}_r\) be its estimate in replicate \(r\), for \(r=1,\ldots,R\). The empirical bias is the mean estimation error; the root mean squared error (RMSE) is the square root of the mean squared estimation error. For interval estimates \([L_r,U_r]\), empirical coverage is the fraction of intervals containing the target. These are finite-simulation summaries, not guarantees about future performance.

Here \(I(\cdot)\) equals 1 when the condition inside it is true and 0 otherwise. Bias and RMSE have the same units as \(\theta\); coverage is a proportion with no measurement units. The denominator is \(R\) for these empirical averages. If some replicates fail and are excluded, the rule for exclusion and the effective denominator must be made explicit; silently dropping failures can distort the metric.

A short justification explains why hand-calculated tests are effective. Each replicate contributes its signed error \(e_r=\hat{\theta}_r-\theta\) to bias and its squared error \(e_r^2\) to the RMSE calculation. With the example errors, positive and negative errors cancel in the signed sum, while both contribute positively to the squared sum. Thus zero bias does not imply zero estimation error: the RMSE is positive. The interval indicator separately checks inclusion of the target, which is why coverage cannot be inferred from bias or RMSE alone.

Designing Reliable Tests and Diagnosing Failures

A practical test suite combines general invariants with a few deliberately chosen input-output tests. For each function, include an ordinary valid case, a boundary or edge case, and an invalid case when the function is responsible for rejecting invalid input. Keep tests small enough that a failure points to a likely source. For example, test the metric function directly with fixed estimates rather than running the entire stochastic simulator just to check one arithmetic formula.

Exact equality is appropriate for values that are exactly represented by the software, such as a small integer count or an array dimension. Floating-point calculations generally require a tolerance. A combined check can be written \(|x_{\mathrm{observed}}-x_{\mathrm{expected}}|\leq \varepsilon_{\mathrm{abs}}+\varepsilon_{\mathrm{rel}}|x_{\mathrm{expected}}|\), with absolute tolerance \(\varepsilon_{\mathrm{abs}}\) and relative tolerance \(\varepsilon_{\mathrm{rel}}\). Select tolerances based on numerical precision and scale, and document them so they cannot be adjusted merely to make a failing test pass.

Randomized code presents a special challenge. A test that expects one exact random draw can be fragile across software versions or random-number implementations. Where an earlier simulation setup fixes seeds for reproducibility, a seeded test can be useful for checking a stable, explicitly supported configuration. More generally, test deterministic properties of the output, such as dimensions, finite-value constraints, and adherence to specified ranges. Do not test that a random sample has exactly its theoretical mean in one run.

When a test fails, check the test expectation as well as the implementation. A failed shape check may reveal a coding error, or it may reveal that the design intentionally allows variable observation counts and the test assumed a rectangular structure. A passing test can also be weak: checking only that every dose is positive will not detect a positive dose assigned to the wrong subject. Tests should reflect the full data contract relevant to the component.

Comparison, Limits, and Common Mistakes

Invariants, unit tests, and benchmarking serve related but distinct purposes. Invariants check broad conditions that should hold across many inputs. Unit tests check a component against specific known cases. Benchmarking compares simulator behavior with known results, which can reveal discrepancies in the simulator as a whole. A robust validation strategy can use all three: invariants can catch malformed outputs early, unit tests can localize implementation mistakes, and benchmarks can assess broader behavior. None establishes correctness for every possible scenario.

These tests also do not establish that a simulation’s assumptions are scientifically suitable, that its estimand answers the intended question, or that a design performs well in untested scenarios. Passing a test suite means only that the tested contracts and cases were satisfied. Maintainability matters too: tests must be revised when the design specification changes, while expected values should be derived independently rather than copied from the function under test.

• Mistake: “The simulator ran, so its data must be valid.” A run can complete despite incorrect labels, dimensions, or units. Check structural invariants and specific expected records.
• Mistake: “A round-trip transformation test proves both functions are right.” Two wrong functions can cancel each other. Also test known input-output pairs and the parameter’s required range.
• Mistake: “A zero bias means accurate estimates.” Opposing errors can cancel. In the worked example, bias is zero while RMSE is 0.7071 mg/L.
• Mistake: “Every floating-point value should match exactly.” Rounding can create tiny differences. Use a justified tolerance for numerical results, but exact checks for dimensions and exact discrete outcomes.