Why Compare Predictions Across Software?
When the same pharmacokinetic model is implemented in NONMEM, Monolix, nlmixr2, and mrgsolve, the concentration predictions should agree when the model, inputs, and calculation conditions match. Agreement is not necessarily exact to the last displayed digit: different numerical solvers and floating-point calculations can produce small differences even when the implementations represent the same model. The practical task is therefore to define what counts as agreement, compare predictions under controlled conditions, and investigate differences that exceed the declared tolerance.
This is a deterministic comparison, not a statistical hypothesis test. There is no sampling distribution, p-value, or probability that the implementations are equivalent. The conclusion applies to the model inputs and prediction times actually checked; it does not prove agreement for every possible covariate, parameter value, or event history.
Conditions for a Meaningful Comparison
Before comparing numbers, establish that the implementations are being asked the same deterministic question. Use identical model equations and parameter values, consistent parameter mappings and units, the same covariate values, and the same observation times. The initial conditions and dose or other input history must also match. These conditions matter because even a perfectly functioning implementation will return different predictions if, for example, clearance differs or a concentration is reported in different units.
For a comparison of deterministic predictions, disable or hold fixed sources of stochastic variation, such as random effects or residual-error draws. Compare the same prediction type on the same scale: for instance, concentration in mg/L to concentration in mg/L, not one tool's concentration to another tool's log concentration. A matched time grid is essential; values at different times are not a pointwise comparison.
A useful first check is a simple model whose prediction can also be calculated by hand. For a one-compartment intravenous bolus model with first-order elimination, amount \(A(t)\) satisfies \(dA/dt=-(CL/V)A(t)\), where \(CL\) is clearance and \(V\) is volume. With dose \(D\), \(A(0)=D\), and \(C(t)=A(t)/V\). Separating variables and applying the initial condition gives:
This expression is a compact independent reference for checking the prediction comparison. Its assumptions are a single well-mixed compartment, an instantaneous intravenous bolus, and first-order elimination, with no additional concentration input after the bolus.
Worked Example: Comparing Four Tools on an Analytic Prediction
Suppose four hypothetical implementations use \(D=100\) mg, \(V=20\) L, and \(CL=2\) L/h for the model above. They are evaluated at \(t=0, 2, 5,\) and \(10\) h. Assume the dose, initial condition, parameter values, output units, and times match in all four tools. The elimination rate constant is:
The concentration at 5 h, for example, is:
The complete comparison is shown below. Values are hypothetical and displayed to six decimal places.
| Time (h) | NONMEM (mg/L) | Monolix (mg/L) | nlmixr2 (mg/L) | mrgsolve (mg/L) |
|---|---|---|---|---|
| 0 | 5.000000 | 5.000000 | 5.000000 | 5.000000 |
| 2 | 4.093654 | 4.093653 | 4.093654 | 4.093653 |
| 5 | 3.032653 | 3.032654 | 3.032653 | 3.032653 |
| 10 | 1.839397 | 1.839397 | 1.839398 | 1.839397 |
At 5 h, the largest pairwise difference is \(3.032654-3.032653=0.000001\) mg/L. The small displayed differences are consistent with rounding or numerical calculation. To make a formal operational decision, however, the analyst still needs to specify a tolerance. Visual similarity alone is not a criterion.
Choose and Apply an Equivalence Rule
An absolute tolerance is useful when a difference has a direct practical meaning in the units of the output. A relative tolerance is useful when the relevant scale changes across predictions. A combined rule accommodates both: it uses an absolute floor near zero and a relative limit for larger values. For two predictions \(C_a(t)\) and \(C_b(t)\), one common rule is:
Here \(\epsilon_{\mathrm{abs}}\) has concentration units, and \(\epsilon_{\mathrm{rel}}\) is dimensionless. The maximum of the two limits avoids demanding an arbitrarily tiny absolute difference near zero while still imposing a scale-sensitive bound at larger concentrations. The tolerances should be chosen for the intended use before inspecting the comparison results; they are not universal constants.
For several tools, decide whether the requirement is that each tool agree with a designated reference or that every pair agree. These are not the same. If each of two predictions is within \(\epsilon\) of a reference, the triangle inequality guarantees only that their mutual difference is at most \(2\epsilon\):
Thus, if the requirement is pairwise equivalence, compare all pairs or directly compute the maximum and minimum prediction at each time. A reference-based workflow can still be appropriate, but its conclusion must be stated as agreement with that reference under its specified bound.
Worked Example: Applying a Combined Tolerance Pairwise
Consider hypothetical predictions at three matched times, from a comparison with four tools. The concentrations below are in mg/L. Set \(\epsilon_{\mathrm{abs}}=0.001\) mg/L and \(\epsilon_{\mathrm{rel}}=0.001\), equivalent to 0.1%. The values are:
| Time | NONMEM | Monolix | nlmixr2 | mrgsolve |
|---|---|---|---|---|
| 1 h | 5.0000 | 5.0002 | 4.9998 | 5.0001 |
| 4 h | 2.0000 | 2.0010 | 1.9990 | 2.0025 |
| 12 h | 0.2000 | 0.2003 | 0.1998 | 0.2006 |
At 1 h, the largest difference is \(5.0002-4.9998=0.0004\) mg/L. The pair setting that difference has a relative limit of \(0.001(5.0002)=0.0050002\) mg/L, so the combined tolerance is \(\max(0.001,0.0050002)=0.0050002\) mg/L. The observed maximum is below the limit; the predictions pass at this time.
At 4 h, the largest difference is \(2.0025-1.9990=0.0035\) mg/L. The relative part for that pair is \(0.001(2.0025)=0.0020025\) mg/L, so the combined tolerance is \(0.0020025\) mg/L. Since \(0.0035>0.0020025\), the predictions fail the pairwise rule at 4 h.
At 12 h, the largest difference is \(0.2006-0.1998=0.0008\) mg/L. The relative part is \(0.001(0.2006)=0.0002006\) mg/L, so the absolute floor controls: the tolerance is \(0.001\) mg/L. Since \(0.0008<0.001\), the predictions pass at 12 h. Overall, the four implementations do not meet the stated equivalence requirement on this grid because they fail at 4 h.
Use Discrepancy Patterns to Guide Investigation
A failed tolerance check establishes that the chosen numerical agreement rule was not met; it does not identify the cause. First verify that compared rows correspond to the same times and that units, parameter values, covariates, and output definitions really match. Then examine the differences by time and tool. A roughly constant offset, a difference that grows with time, and a discrepancy restricted to one output region suggest different questions to investigate, but no pattern alone proves a cause.
Worked Example: A Clearance Mismatch Produces a Time-Dependent Difference
Use the one-compartment model and common inputs \(D=100\) mg and \(V=20\) L. Three hypothetical tools use \(CL=2\) L/h, but a fourth uses \(CL=2.2\) L/h. At 5 h, the three matching implementations predict \(3.032653\) mg/L. The fourth predicts:
The absolute difference is \(3.032653-2.884749=0.147904\) mg/L. Relative to the matching prediction, this is approximately \(0.147904/3.032653=0.04877\), or 4.877%. This is far larger than the differences caused by rounding in the earlier example. An audit of the parameter inputs finds that clearance is not matched, so this comparison does not test equivalence of implementations under identical parameters.
The model also explains why the gap grows over time. From \(C(t)=(D/V)\exp[-(CL/V)t]\), the sensitivity of log concentration to clearance is:
The magnitude increases with \(t\), so a clearance discrepancy can produce an increasingly large relative prediction difference at later times. This pattern is a useful clue, not a substitute for checking the supplied parameter values.
Alternatives, Diagnostics, and Limits
Exact equality is a poor general criterion for floating-point output: harmless numerical rounding can make the last digits differ. An absolute-only tolerance is interpretable in concentration units but may be too strict at high concentrations or too permissive relative to very small predictions. A relative-only tolerance scales with predictions but becomes unstable near zero. The combined rule addresses these limitations, though its thresholds still require justification for the application.
Diagnostics should include more than a single summary number. Check that each tool returned a finite prediction at every requested time, that times were aligned before subtraction, and that the reported concentration units and scale are consistent. Plotting paired differences against time or concentration can expose where failures occur. Report the tolerance, whether comparisons were reference-based or pairwise, and the maximum discrepancy with its time and units. If a prediction is zero or extremely close to zero, explicitly inspect the absolute difference because a relative difference alone can be misleading or undefined.
A successful check means only that the tested outputs satisfy the stated rule for the selected scenario and grid. It does not establish that all model components are translated correctly in every setting, nor does it show that the model itself is scientifically appropriate. A model can be implemented consistently across tools and still be misspecified. Conversely, discrepancies may arise from numerical settings rather than a different scientific model. The relevant conclusion is bounded: state what was compared, under which matched conditions, and against which tolerance.
Name the prediction scale, tools, times, model inputs, and whether the criterion is reference-based or pairwise.
Choose absolute and relative limits with units and a rationale tied to the intended use.
Apply the rule at each matched time, identify failures, and investigate the inputs and time-specific patterns.
Report whether the tested predictions meet the criterion without extending the claim beyond the scenarios and times checked.
Check Your Understanding
Answer each question using the comparison definitions above.
- Why is exact equality of every printed digit not generally required for a deterministic cross-software comparison?
- Two tools each differ from a reference by at most 0.002 mg/L. What upper bound does the triangle inequality give for their mutual difference?
- At a time where two predictions are 0.40 and 0.41 mg/L, calculate their absolute difference. Under \(\epsilon_{\mathrm{abs}}=0.001\) mg/L and \(\epsilon_{\mathrm{rel}}=0.01\), do they pass the combined rule?
- What does a successful check on a finite observation-time grid establish, and what does it not establish?
Solutions. (1) Different numerical calculations and display rounding can create small differences even when the same deterministic model is implemented; a prespecified tolerance gives a meaningful acceptance rule. (2) The mutual difference is at most \(0.002+0.002=0.004\) mg/L. This bound alone does not show that the pair meets a 0.002 mg/L pairwise limit. (3) The absolute difference is \(|0.41-0.40|=0.01\) mg/L. The relative limit is \(0.01(0.41)=0.0041\) mg/L, so the combined tolerance is \(\max(0.001,0.0041)=0.0041\) mg/L. Because \(0.01>0.0041\), they fail. (4) It establishes agreement under the stated rule for the checked inputs and times only; it does not establish agreement at untested times or scenarios, or validate the scientific model.