Why Compare a Reviewed Model With a Benchmark?
A model can change unintentionally when code is edited, software is updated, or data-processing steps are revised. The change may leave a plausible-looking output while altering an estimate or prediction that matters. A benchmark run gives reviewers a documented reference point: rerun the reviewed model under controlled conditions, then compare selected outputs with outputs from a trusted run.
The comparison is useful only if it is designed deliberately. A difference does not automatically mean the model is wrong: it may reflect different inputs, a change in numerical environment, or ordinary simulation variation. Conversely, an unchanged output does not prove the model is correct. Benchmarking detects departures from a reference; it does not replace independent test cases, model review, or assessment of whether the model is scientifically appropriate.
Conditions for a Meaningful Comparison
The benchmark and reviewed run must represent the same comparison target. At a minimum, identify the model version, input data, data-processing steps, estimation or simulation settings, software environment, and outputs to be compared. If any of these differ, record the difference and decide whether the runs remain comparable. A result from a different dataset or evaluation grid should not be treated as a like-for-like benchmark simply because the output files have similar names.
For an estimation run, relevant settings can include the estimation method, initial estimates, convergence criteria, and options that affect computation. For a simulation, they can include the design, random seed, and replicate settings. Earlier material on random-seed and replicate variability explains why stochastic variation matters: when the same fixed seed and settings do not reproduce the same random stream, a difference may reflect simulation variation rather than a code change. In that case, document and control the stochastic component or define a comparison based on an appropriate summary rather than interpreting a single noisy replicate as a regression.
The reference must also be trustworthy. It should be reviewed, versioned, and retained with enough information to reproduce it. A flawed or obsolete benchmark can faithfully flag departures from itself while pointing reviewers toward the wrong target. State which outputs are important before looking at the new differences; choosing comparison metrics only after seeing the results risks overlooking inconvenient changes.
Absolute and Relative Tolerances
An absolute tolerance is useful when the size of a discrepancy is meaningful in the quantity’s units. A relative tolerance expresses discrepancy as a fraction of the benchmark value and is useful when the acceptable scale of error grows with the magnitude being compared. Relative discrepancy alone is unstable when a benchmark value is zero or very close to zero, so a robust policy often uses both.
Let \(b_i\) be benchmark output \(i\), \(r_i\) the corresponding reviewed output, \(\tau_{\mathrm{abs},i}\) its absolute tolerance, and \(\tau_{\mathrm{rel},i}\) its relative tolerance. For a nonzero benchmark value, define the comparison limit as the larger of the absolute limit and the relative limit based on the benchmark. The normalized discrepancy gives a convenient pass/fail score.
The comparison passes this numerical rule when \(q_i \leq 1\), and fails it when \(q_i > 1\). If both tolerances are zero, only exact numerical equality passes; handle that case explicitly rather than dividing by zero. A tolerance should reflect the purpose and scale of the output, not be selected merely to make a particular run pass. For parameter estimates, specify units and meaningful precision. For predictions, specify the prediction type, units, evaluation points, and any relevant subgroup or individual indexing.
This rule uses the benchmark value in the relative term, not the reviewed value. That makes the reference the fixed comparison scale and prevents the denominator from changing with the result being assessed. The maximum allows the absolute limit to protect comparisons near zero and the relative limit to govern when it is larger.
This equivalence follows by multiplying both sides by the positive comparison limit \(L_i\). It justifies using either the normalized score or the direct difference rule, provided \(L_i>0\). Report the signed difference \(r_i-b_i\) as well as its magnitude when direction is informative: a positive difference means the reviewed value is larger than the benchmark, but does not establish why it changed.
Worked Example: Comparing Parameter Estimates
Suppose a hypothetical benchmark reports clearance \(CL=5.00\) L/h and volume \(V=40.0\) L. The reviewed run reports \(CL=5.02\) L/h and \(V=40.8\) L. For each parameter, use an absolute tolerance of 0.05 in that parameter’s units and a relative tolerance of 1.0%.
For clearance, the absolute difference is \( |5.02-5.00|=0.02\) L/h. The relative limit is \(0.010(5.00)=0.050\) L/h, so \(L=\max(0.05,0.050)=0.050\) L/h. Thus \(q=0.02/0.050=0.400\). Clearance passes the stated rule; the reviewed estimate is 0.02 L/h, or 0.40%, higher than the benchmark.
For volume, the absolute difference is \( |40.8-40.0|=0.8\) L. The relative limit is \(0.010(40.0)=0.40\) L, so \(L=\max(0.05,0.40)=0.40\) L. Thus \(q=0.8/0.40=2.00\). Volume fails: its 0.8 L difference is twice the allowed 0.40 L limit. This is a detected departure requiring review, not proof that the new estimate is scientifically inferior.
Compare Predictions on a Defined Grid
Parameter agreement alone may not reveal a consequential change in model behavior. Conversely, different parameter estimates can sometimes produce similar predictions over a particular range. Include predictions relevant to the intended use, evaluated at fixed, documented conditions. A grid might specify dose, time after dose, covariate values, and whether predictions are population or individual predictions. Use the same rows and labels in both runs so that each prediction is paired with its intended benchmark counterpart.
A prediction comparison should not be reduced to an overall average difference if local changes matter. Check differences by time, dose, and relevant subgroup, and examine both the largest discrepancy and its location. A mean discrepancy near zero can conceal large positive and negative differences that cancel. The evaluation grid should be sufficiently informative for the intended review, but it remains a finite check: agreement at selected points does not establish agreement at every possible input.
Worked Example: Checking Predictions Near Zero
A hypothetical benchmark and reviewed run give concentrations at two prespecified times. Use an absolute tolerance of 0.020 mg/L and a relative tolerance of 1.0%.
| Time | Benchmark (mg/L) | Reviewed (mg/L) |
|---|---|---|
| 2 h | 12.00 | 12.06 |
| 24 h | 0.020 | 0.045 |
At 2 h, \(d=|12.06-12.00|=0.06\) mg/L. The relative limit is \(0.010(12.00)=0.12\) mg/L, so \(L=\max(0.020,0.12)=0.12\) mg/L and \(q=0.06/0.12=0.500\). The prediction passes.
At 24 h, \(d=|0.045-0.020|=0.025\) mg/L. The relative limit is \(0.010(0.020)=0.00020\) mg/L, so \(L=\max(0.020,0.00020)=0.020\) mg/L and \(q=0.025/0.020=1.25\). The prediction fails. Reporting only a relative discrepancy would give \(0.025/0.020=125\%\), which is hard to interpret as a useful acceptance rule near zero. The specified absolute tolerance supplies a stable, unit-based limit; it does not make this particular difference acceptable.
Run the Comparison Reproducibly
A practical comparison is a controlled sequence, not just a visual inspection of two output files. Preserve the benchmark outputs unchanged, record the reviewed run’s inputs and settings, align corresponding outputs, apply the pre-specified rules, and retain a report of discrepancies. The comparison should make it possible for another reviewer to identify exactly which quantity failed and reproduce the check.
Record the benchmark version, model and data identifiers, run settings, software information, and the set of outputs to compare.
Check that the reviewed run uses the intended inputs, settings, units, and evaluation grid. Investigate mismatches before interpreting numerical differences.
Match outputs by stable labels such as parameter name, time, dose, and subject identifier. Calculate signed differences, absolute differences, and tolerance scores.
Report pass/fail status, failed quantities and locations, the tolerance rules, and run identifiers. Retain the comparison report and the files needed to reproduce it.
Worked Example: Interpreting a Prediction Grid Summary
Suppose a hypothetical prediction check compares three times using an absolute tolerance of 0.10 mg/L and a relative tolerance of 2.0%. The benchmark predictions are 10.0, 2.00, and 0.20 mg/L; the reviewed predictions are 10.1, 1.94, and 0.31 mg/L.
At the first time, \(d=0.10\) mg/L and \(L=\max(0.10,0.020(10.0))=0.20\) mg/L, giving \(q=0.10/0.20=0.500\): pass. At the second time, \(d=|1.94-2.00|=0.06\) mg/L and \(L=\max(0.10,0.040)=0.10\) mg/L, giving \(q=0.600\): pass. At the third time, \(d=|0.31-0.20|=0.11\) mg/L and \(L=\max(0.10,0.004)=0.10\) mg/L, giving \(q=1.10\): fail.
The maximum score is 1.10, at the third time point, so the grid fails the defined rule. The first two passes do not cancel that failure. The useful review record states the specific time, benchmark and reviewed predictions, units, tolerance, and score; it does not merely report that the average difference was small.
Diagnostics, Alternatives, and Limitations
A failed comparison is a prompt to investigate. First verify that outputs are aligned correctly: a row-order change can produce apparent discrepancies even when values are unchanged. Check units, labels, missing values, rounding, and whether the expected model and data versions were used. Then inspect where differences occur. A cluster at one time point or for one parameter may suggest a localized change; a broad shift may suggest a wider change in inputs, settings, or implementation. These patterns guide review but do not identify the cause by themselves.
Other warning signs include a tolerance so wide that material changes pass, a relative-only rule applied near zero, and a tolerance so narrow that routine numerical variation repeatedly fails the check. If rounding occurs before comparison, the reported values may conceal a discrepancy; when available, compare full-precision values and report rounded values for readability. Tolerances should be justified in context and applied consistently. A pass means only that the selected outputs met the specified numerical rule under the checked conditions.
Benchmark comparison complements, rather than replaces, independent test cases. Test cases can check whether selected components behave as expected, including cases not represented by the benchmark. A benchmark is especially useful for detecting changes across a known, reviewed end-to-end run. Exact reproducibility is stricter than tolerance-based agreement and may not be realistic across computational environments; a tolerant comparison makes permitted numerical variation explicit. Neither approach establishes that the model’s assumptions or scientific interpretation are correct. The earlier material on testing model components with independent test cases addresses a distinct form of checking.
Check Your Understanding
Answer each question using the stated comparison rules. Solutions follow the questions.
- A benchmark estimate is 8.0 mg/h and the reviewed estimate is 8.1 mg/h. With absolute tolerance 0.05 mg/h and relative tolerance 1.0%, does the comparison pass?
- A benchmark prediction is 0.010 mg/L and the reviewed prediction is 0.018 mg/L. With absolute tolerance 0.005 mg/L and relative tolerance 2.0%, does it pass?
- Why should a benchmark prediction grid specify times and covariate values rather than compare only an overall mean prediction?
- A comparison reports no failed values, but the reviewed run used a different input dataset. What can be concluded?
Solutions. (1) The difference is \(0.1\) mg/h. The relative limit is \(0.010(8.0)=0.08\) mg/h, so the comparison limit is \(\max(0.05,0.08)=0.08\) mg/h. Since \(0.1>0.08\), it fails. (2) The difference is \(0.008\) mg/L. The relative limit is \(0.020(0.010)=0.00020\) mg/L, so the absolute tolerance governs and the limit is \(0.005\) mg/L. Since \(0.008>0.005\), it fails. (3) An overall mean can hide changes at specific times or covariate values, including differences that cancel when averaged; fixed points make the comparison interpretable and reproducible. (4) The runs are not a like-for-like comparison. The result cannot establish agreement for the intended fixed-input benchmark; first establish why the dataset differs and whether a valid comparison is possible.