Regression Tests for QSP Model Changes

Why Regression Tests Need an Explicit Comparison Plan

A code change can alter a QSP simulation without causing a crash or violating an obvious physical constraint. A small difference in a state trajectory may shift a threshold-crossing time; a changed interpolation routine may affect a derived exposure measure; and an unintended parameter-indexing error may alter one treatment arm but not another. Regression tests make such changes visible by comparing selected, interpretable model results before and after a change.

A regression test is not simply a demand that every output remain numerically identical. Some changes are intended: for example, a corrected implementation may deliberately change a previously incorrect event time. A useful test plan therefore identifies what should remain stable, what is expected to change, and how much numerical variation is acceptable. It treats a difference as evidence to investigate, not automatic proof of a scientific error.

The comparison is meaningful only when the reference and candidate runs represent the same test case. For a deterministic simulation, record the model version or source revision, inputs, initial conditions, parameter values, solver settings, output times, units, and relevant software environment. Keep the reference results unchanged during testing; update them only through a documented review when a change is justified. If random variation is part of the implementation, one run against one stored trajectory is generally not a suitable test of distributional behavior. That situation requires a test designed for stochastic implementations rather than an exact snapshot comparison.

The tested quantities also need stable definitions. A quantity called “response at day 14” must specify which state or derived output is measured, its unit, and how the value at day 14 is obtained. A threshold event must specify the threshold, crossing direction, and rule for a crossing that never occurs. Without these details, apparent disagreement may come from comparing different observables rather than detecting a code regression.

Choosing Comparison Rules

Exact equality is appropriate only when identical inputs and execution are expected to produce bit-for-bit identical values and the output is not affected by floating-point or platform variation. For most numerical simulation outputs, a tolerance is more useful. Set it from the quantity’s scale, units, numerical precision, and the decision the test is intended to protect. Do not choose a tolerance merely because it makes a failing comparison pass.

Here \(\mathrm{atol}\) is an absolute tolerance in the units of the quantity, and \(\mathrm{rtol}\) is a dimensionless relative tolerance. The absolute term protects comparisons near zero, where a relative difference alone can be uninformative or undefined. The relative term allows a larger absolute difference for a larger reference value. Apply the rule component by component to a vector of selected outputs; a good match in one component should not conceal a failure in another.

For a derived quantity, apply the comparison to the final quantity as well as, where useful, to its inputs or trajectory. This can reveal errors that a sparse set of state checks misses. Conversely, checking only a derived quantity can conceal compensating errors in different parts of the trajectory. Choose observables that reflect the intended use of the model rather than storing every computed value without a reason.

A Practical Test Design

Events and Derived Quantities

Event time is not interchangeable with a state value at a fixed time. To test an event, define what counts as an event and compare both its status and, if it occurs, its time. For example, define the event as the first downward crossing of a biomarker threshold. The test specification should say whether a trajectory with no crossing is expected, and how the crossing time is extracted. If one run has a crossing and the other does not, a numeric time comparison cannot resolve the disagreement: the event-status change itself is a failure unless it was planned.

Derived quantities require similarly explicit definitions. An area under a concentration-time curve (AUC), for instance, depends on the time interval and the numerical integration rule. A comparison is interpretable only if those are held fixed, or if a change in them is the intentional subject of the test. Units provide a practical check: if concentration is in mg/L and time is in hours, AUC is in mg·h/L.

Separating Planned Changes from Regressions

When a code change is intended to alter scientific behavior, record the expectation before examining the new results. For example, a corrected parameter mapping might be expected to change exposure in one treatment arm while leaving the control arm unchanged. Test both claims: verify that the targeted exposure changes in the justified way, and retain regression checks on unaffected outputs. A test suite that merely replaces old values with new ones can erase evidence of accidental changes.

Diagnostics, Alternatives, and Limitations

A failed comparison is a diagnostic signal. First confirm that the same test case and observable were used. Then inspect the size, direction, and location of the difference: a failure limited to one output time suggests a different pattern from a shift across all treatment arms. Check units, output indexing, event definitions, derived-quantity calculations, and the reference record. If a difference is numerical, report it in context, including the reference, candidate, observed difference, tolerance, and affected scenario. Do not silently widen the tolerance before identifying its cause.

Regression checks have a different role from other verification tests. Analytical reference solutions can compare a simulation with a known solution for a tractable case; invariant and limiting-case tests check properties or expected boundary behavior; metamorphic tests examine whether related inputs and outputs obey a known relationship. A regression test instead asks whether selected results changed relative to a trusted prior implementation. These methods complement one another: agreement with an old result cannot establish that the old result was correct, and a property test may not detect a change in a clinically important endpoint.

Regression tests also have coverage limits. A finite set of scenarios cannot guarantee correctness for every parameter combination, treatment regimen, or virtual individual. Sparse output checks can miss a transient discrepancy between recorded times; summary quantities can hide offsetting trajectory errors. A passing regression suite is evidence that the selected checks meet their criteria, not proof of model validity, biological truth, or suitability for every intended use.

Common Pitfall

Wrong reasoning: “The candidate differs from the reference, so the code is defective; I will loosen the tolerance until the test passes.” This confuses difference with defect and makes the pass criterion depend on the observed result. The correction is to establish the comparison conditions and tolerance before testing, classify expected changes in advance, and investigate unexpected differences. If a justified scientific correction changes an output, preserve that explanation and update the approved reference through review rather than hiding the change with a broader tolerance.