Cross-Implementation Checks for Independent Code

Why Compare Independent Implementations?

When two separately coded versions of a mechanistic model receive the same inputs, their outputs should agree closely if they implement the same equations correctly and solve them adequately. A disagreement can reveal a coding error, a unit conversion problem, inconsistent event handling, or a numerical configuration that needs investigation. Agreement is useful evidence that the implementation is reproducible across code paths; it does not establish that the model’s equations describe biology correctly.

A cross-implementation check is informative only when the comparison is controlled. The implementations must represent the same model, receive equivalent inputs, and report comparable quantities at aligned times. Their numerical settings must be documented, and any difference in state representation must be resolved before comparing outputs. The discrepancy criterion should be chosen before looking at the results.

Conditions for a Meaningful Comparison

Write the model state reported by implementation A as \(x_A(t)\), and the state reported by implementation B as \(x_B(t)\). The same symbol need not mean the same stored quantity in both codes. For example, one implementation might store drug amount and another might store drug concentration. Define a mapping from each implementation’s reported state to a common comparison quantity, \(y_A(t)=g_A(x_A(t))\) and \(y_B(t)=g_B(x_B(t))\). Compare \(y_A\) with \(y_B\), not raw state vectors whose components may have different meanings or units.

Before running the comparison, document the conditions it assumes:

• Same model specification. Both codes implement the same equations, parameter meanings, initial conditions, and boundary conditions. Equivalent equations written in different forms are acceptable, but the correspondence must be explicit.
• Equivalent inputs. Parameter values, dose amounts, units, input functions, and intervention schedules agree after any necessary conversion.
• Comparable numerical settings. Record the solver, tolerances, step-size controls, and interpolation choices. For a controlled first comparison, match these settings as closely as the solvers permit. Different solver algorithms can still be compared, but their settings and consequences must be examined rather than assumed equivalent.
• Aligned outputs. Compare the same requested times and the same event convention, such as a value immediately before or immediately after a dose. If one code reports at different times, use a justified interpolation procedure or rerun it at the common times.
• Valid mappings and tolerances. State transformations must preserve the intended quantity, and each component must have a prespecified discrepancy tolerance appropriate to its scale and units.

For stochastic models, matching a random-number seed may not make implementations comparable: the codes may consume random draws in different orders or use different random-number generators. The earlier material on statistical tests for stochastic QSP implementations addresses comparisons that account for simulation variability. The examples here focus on deterministic trajectories and outputs.

A Scale-Aware Discrepancy Rule

For mapped output component \(j\) at comparison time \(t_k\), denote the results by \(y_{A,j}(t_k)\) and \(y_{B,j}(t_k)\). A common prespecified rule combines an absolute tolerance \(a_j\) and a relative tolerance \(r_j\):

Here \(a_j\) has the units of component \(j\), while \(r_j\) is dimensionless. The absolute term gives a meaningful allowance near zero; the relative term scales the allowance when values are large. A comparison can also be summarized by the scaled discrepancy

Provided the denominator is positive, the component passes at that time when \(q_{j,k}\leq 1\). A trajectory-level rule might require this condition at every prespecified time for every monitored component. The rule should identify which components and times are tested; reporting only an average discrepancy can conceal a localized failure.

Why the Tolerance Has Two Parts

The rule follows directly from combining an absolute error allowance with an allowance proportional to the scale of the result. Let \(\Delta_{j,k}=|y_{A,j}(t_k)-y_{B,j}(t_k)|\), and let \(M_{j,k}=\max(|y_{A,j}(t_k)|,|y_{B,j}(t_k)|)\). If the allowed discrepancy is the sum of a baseline allowance \(a_j\) and a scale-dependent allowance \(r_jM_{j,k}\), then passing means \(\Delta_{j,k}\leq a_j+r_jM_{j,k}\). Dividing by that positive allowance gives \(q_{j,k}\leq1\). Near zero, the relative contribution is small, so \(a_j\) prevents an unrealistically strict requirement based solely on relative error. At larger magnitudes, \(r_jM_{j,k}\) allows the criterion to reflect scale.

This tolerance is an acceptance rule for comparing outputs, not a solver setting and not a universal guarantee of accuracy. The numerical solver’s internal error controls should be sufficiently tight for the check’s purpose, but a solver tolerance does not automatically determine a scientifically appropriate \(a_j\) or \(r_j\). Prespecify comparison tolerances using the output’s units, scale, and intended computational use.

Aligning Events and Comparison Times

Discontinuous inputs require particular care. At a bolus dose, the state may jump; a reported value at the event time must therefore be identified as pre-dose or post-dose. The two implementations must use the same convention. A mismatch can produce a large discrepancy even when both codes implement the intended dose correctly.

A Practical Comparison Sequence

A disciplined sequence makes the evidence easier to reproduce and disagreements easier to diagnose. Keep the test conditions, mappings, and tolerances in the record alongside the output comparisons.

Useful diagnostics include plots of both trajectories, plots of their signed difference, and tables of the largest scaled discrepancies \(q_{j,k}\). A sudden difference near a dose suggests checking event alignment; a steadily growing difference may point to parameter, equation, or numerical discrepancies. These patterns guide investigation but do not identify a cause on their own. If the disagreement changes materially when numerical accuracy is tightened, solver convergence deserves further examination.

What Agreement Does—and Does Not—Show

Cross-implementation checks complement other verification approaches. An analytical reference solution can test a code against a known result in a case where such a solution exists. Invariant, limiting-case, and metamorphic tests examine properties or transformations expected of the model. Regression tests check whether a code change alters outputs from established cases. Independent implementations offer a different kind of evidence: agreement across separately coded paths for shared test cases. None of these checks alone establishes that the model is biologically adequate.

The word independent also needs care. If both implementations reuse the same equation-generation tool, parameter-processing module, or copied code, they may share the same defect. Record shared dependencies and avoid claiming independence beyond what the development process supports. Even genuinely separate implementations can agree because they share a mistaken interpretation of the specification.

A common mistake is to reason, “The two codes agree within tolerance, so the model is correct.” The comparison supports only the narrower conclusion that the mapped outputs agreed within the chosen tolerance under the tested conditions. It cannot show that the equations, assumptions, parameter values, or predictions are correct for a biological use. The opposite mistake is to compare raw states without checking units or meanings: numerical similarity between unlike quantities is not agreement.