Why Test Model Reproducibility?
A QSP model can change unintentionally when someone edits an equation, modifies an event, updates a dependency, or alters solver settings. These changes may leave the model executable while changing a simulated concentration, biomarker trajectory, or decision-relevant summary. A regression test provides an automated check: under a specified simulation protocol, do selected outputs remain sufficiently close to approved reference results?
A test failure is a prompt to investigate, not a diagnosis. The difference could indicate a coding error, a changed computational environment, an unstable numerical calculation, or an intentional model revision. The purpose of the test is to make differences visible and reproducible so they can be evaluated in context.
A useful test therefore makes its comparison reproducible. It identifies the model version, inputs, initial conditions, simulation protocol, outputs, units, comparison method, and tolerances. The earlier material on reproducible project layouts, input lineage, computational environments, and solver settings provides the relevant context for recording these elements.
What a Test Compares
A reference result is an approved output generated from a specified model and simulation setup. It may be a single value, such as the concentration at a defined time, or a vector of values, such as a time-course sampled at named times. Tests should focus on outputs that are meaningful for the model’s intended use: for example, a peak concentration, time to threshold, a pathway response at a prespecified time, or a trajectory over an interval.
Before comparing results, establish that the two outputs refer to the same quantity. Confirm matching units, species or compartment, time points, intervention, parameter set, initial conditions, and event definitions. A concentration at 24 hours cannot be meaningfully compared with a value at 23 hours merely because both are stored in the same position in an array. Likewise, values expressed in nM and µM need conversion before comparison.
For a deterministic simulation, the test assumes that the model, inputs, and protocol are sufficiently specified to produce comparable results. Differences from floating-point arithmetic may occur even when the scientific calculation is effectively unchanged, particularly across computing environments. Exact equality is therefore often unnecessarily strict.
Absolute and Relative Tolerances
A common comparison rule combines an absolute tolerance and a relative tolerance. For a new result \(y_{\mathrm{new}}\) and approved reference \(y_{\mathrm{ref}}\), declare the comparison to pass when:
Here, \(\mathrm{atol}\) is an absolute tolerance in the output’s units, and \(\mathrm{rtol}\) is a dimensionless relative tolerance. Both must be chosen before examining the result being tested. The rule allows a fixed amount of difference near zero while also allowing a difference that scales with the magnitude of larger reference values.
To see the role of the two terms, define the normalized discrepancy \(q\) as the absolute difference divided by the permitted difference. Then the test is equivalent to checking whether \(q\leq 1\), provided the denominator is positive:
This form is useful for reporting how close a result came to failing. If both tolerances are zero, the rule reduces to exact equality. If the reference is zero, the relative term contributes nothing, so the absolute tolerance determines whether a small nonzero result passes. That is why relative tolerance alone is poorly suited to outputs that can be zero or close to zero.
Worked Example: Testing a Single Concentration
In a hypothetical deterministic QSP simulation, the approved reference for a specified drug concentration at 12 hours is 100.0 nM. A test uses \(\mathrm{atol}=0.10\) nM and \(\mathrm{rtol}=0.010\). The new run reports 100.6 nM. Does the comparison pass?
The observed absolute difference is:
The allowed difference is:
Because \(0.6\ \mathrm{nM}\leq1.10\ \mathrm{nM}\), the test passes. This says only that this output is within the specified tolerance under this protocol; it does not prove that the model is valid or that every other output agrees.
Worked Example: A Near-Zero Output
A hypothetical simulation has an approved residual biomarker value of \(0.00\) µg/L. The test specifies \(\mathrm{atol}=0.05\) µg/L and \(\mathrm{rtol}=0.02\). The new result is \(0.02\) µg/L.
The absolute difference is \(0.02\) µg/L. Since the reference is zero, the relative contribution is \(0.02|0.00|=0\) µg/L, and the allowed difference is:
The comparison passes because \(0.02\ \mathrm{\mu g/L}\leq0.05\ \mathrm{\mu g/L}\). If this test used relative tolerance alone, the zero reference would provide no usable scale for the comparison. The absolute tolerance supplies that scale. Its value should still be scientifically and numerically justified, rather than chosen simply to permit this result.
Testing a Trajectory
For time-course outputs, compare values at explicitly matched times and apply the rule at each point, or use a separately justified trajectory-level metric. A pointwise test is transparent: it shows where a trajectory first exceeds tolerance. It also makes the sampling schedule part of the test definition. If event times or adaptive solver output grids differ, interpolate only under a defined, appropriate rule; do not compare array positions as if they necessarily represented the same time.
Worked Example: Finding a Trajectory Mismatch
A hypothetical model predicts a concentration in nM at three fixed times. The test uses \(\mathrm{atol}=0.20\) nM and \(\mathrm{rtol}=0.020\) at each time.
| Time (h) | Reference (nM) | New result (nM) |
|---|---|---|
| 0 | 0.0 | 0.0 |
| 1 | 50.0 | 50.4 |
| 2 | 20.0 | 21.0 |
At 0 hours, the difference is 0.0 nM, which is within the 0.20 nM absolute tolerance. At 1 hour, the difference is \(0.4\) nM and the allowed difference is \(0.20+0.020(50.0)=1.20\) nM, so that point passes. At 2 hours, the difference is \(1.0\) nM, while the allowed difference is \(0.20+0.020(20.0)=0.60\) nM. That point fails. The trajectory test therefore fails even though two of its three points pass. The failure identifies a discrepancy at 2 hours; further investigation is needed to determine its cause and importance.
Building Reliable Tests and Investigating Failures
A robust test suite uses a small set of purposeful outputs rather than attempting to store every value produced by every simulation. Include checks that cover important model behavior, such as a baseline scenario, a key intervention, an event boundary, or an output used in a downstream analysis. A test should also check that outputs are finite and correctly labelled: a missing value, a non-finite number, or an unexpected unit should not pass merely because a numerical comparison was skipped.
Record the model and inputs, initial conditions, simulation protocol, output definition, units, and sampling times.
Confirm that the simulation completed as expected and that required values are present, finite, and aligned with the specified times.
Apply the predeclared comparison rule and report the reference, new value, difference, permitted difference, and pass or fail status.
Trace failures to their source. Preserve the prior reference and document any reviewed, intentional update with its rationale.
When a test fails, first verify that it ran the intended model and scenario. Check parameter and input versions, units, output labels, event timing, solver configuration, and computational environment. Then inspect the numerical pattern: a sudden change at an event may suggest an event-handling difference, while a gradual divergence may point to altered dynamics or numerical behavior. Compare other outputs and, where useful, rerun the same case to assess whether the result is stable.
The test can detect a changed result, but it cannot identify whether the cause is a coding defect or a scientifically intended change. If an equation or parameter was deliberately revised, a difference from the old reference may be expected. Review the change, its consequences, and the test case; then update the reference only when the new result is approved for that specified model version. Keep enough information to distinguish the old and new references and explain why the update was warranted. Replacing a reference simply to make a failing test pass defeats the test’s purpose.
Alternatives, Diagnostics, and Limits
| Comparison approach | Useful when | Main caution |
|---|---|---|
| Exact equality | Outputs are expected to be identical under a tightly controlled, deterministic computation. | Small floating-point differences can cause failures across otherwise equivalent environments. |
| Absolute and relative tolerances | Outputs vary in scale or can approach zero. | Tolerances must be justified; generous values can hide important changes. |
| Summary or trajectory metrics | A question concerns an overall feature such as peak value or integrated exposure. | Similar summaries can conceal differences in timing or trajectory shape. |
Use exact equality only when the computation and representation make it appropriate. Conversely, a tolerance-based pass does not guarantee close agreement in every scientific sense: an error in a low-magnitude but critical output may matter, and a poorly chosen tolerance can conceal it. Pair summary metrics with selected pointwise checks when both overall behavior and local features matter.
For stochastic simulations, a fixed random seed may make a run repeatable within a particular implementation, but it does not guarantee identical results across software versions or environments. If the intended output is stochastic, comparing one realization to a fixed reference may be misleading. Tests may instead check prespecified distributional summaries or other properties with appropriate methods, while accounting for simulation uncertainty. The comparison rule must match what the model is intended to reproduce.
Regression tests are also limited to the cases and outputs they cover. A suite can pass while an untested scenario has changed substantially. It does not replace code review, model qualification, or evaluation against relevant evidence. Its value is narrower and practical: it provides a repeatable early warning when selected outputs change under specified conditions.
Common Pitfall
Wrong reasoning: “The test failed, so the model is invalid; I will widen the tolerance or replace the reference until it passes.” This confuses a numerical alert with a scientific conclusion and can erase evidence of an unintended change.
Correction: Treat failure as a discrepancy to explain. Check the setup and computational conditions, identify whether the change was intended, and evaluate its implications. Change a tolerance only with a documented justification, and update an approved reference only after reviewing the changed model behavior. A passing result is evidence of agreement for the tested outputs and conditions, not a general certificate of validity.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A reference is 40.0 mg/L, a new result is 40.8 mg/L, \(\mathrm{atol}=0.10\) mg/L, and \(\mathrm{rtol}=0.015\). Does the comparison pass?
- Why is relative tolerance alone unsuitable when a reference value is zero?
- A trajectory test passes at every sampled time, but the event time differs between runs. What should be checked before concluding that the trajectories are reproducible?
- A model change was intentional, and a test now fails against its former reference. Does the failure establish that the change is a defect?
- Give one limitation of a regression test that checks only peak concentration.
Solutions. (1) The difference is \(|40.8-40.0|=0.8\) mg/L. The allowed difference is \(0.10+0.015(40.0)=0.70\) mg/L, so the comparison fails. (2) With a zero reference, the relative allowance is zero regardless of the new value; an absolute tolerance provides a meaningful allowance near zero. (3) Confirm that event definitions and event timing are specified and aligned, and check whether the compared outputs represent the same times and protocol. (4) No. The failure shows a difference from the old approved result; review whether the change is expected and appropriate before deciding whether to update the reference. (5) Different trajectories can have the same peak, so a peak-only test may miss changes in timing, duration, or values at other times.