Why Compare Outputs Numerically?
Reproducing a submitted run is useful only if its results can be compared in a disciplined way. Two reports may display slightly different last digits because of rounding or numerical computation, while larger differences may indicate that the rerun did not reproduce the submitted analysis. The task is to define what counts as acceptable variation for the outputs under review, then apply that rule consistently.
This comparison assumes that the submitted run and rerun are intended to represent the same analysis, and that the relevant run records and output definitions have been aligned. Establishing those facts is a prerequisite, not something that numerical agreement can establish. A comparison of outputs also does not replace review of the model, data, or implementation.
A Scale-Aware Tolerance Rule
One comparison rule combines an absolute tolerance with a relative tolerance. The absolute part protects comparisons near zero; the relative part allows a discrepancy to scale with the magnitude of the quantity. Specify both tolerances before judging the results, and set them for the output type and its context.
Here \(a\) is a nonnegative absolute tolerance in the same units as the output, and \(r\) is a nonnegative, dimensionless relative tolerance. The comparison fails the prespecified rule if \(|R-S|>T(S,R)\). Record \(a\) and \(r\), including their units or lack of units, so that another reviewer can reproduce the decision. Different quantities may need different tolerances: a tolerance appropriate for a concentration in mg/L is not automatically appropriate for a clearance in L/h or an objective-function value.
The combined rule has a simple justification. Suppose an output is considered acceptable if its numerical discrepancy is no greater than an absolute allowance \(a\) plus an allowance proportional to its scale. Using the larger of \(|S|\) and \(|R|\) as that scale gives the displayed threshold. When both values are close to zero, the relative contribution is small and \(a\) prevents a tiny denominator from making the comparison unstable. For larger values, the relative contribution grows with the output. This is a practical decision rule, not a confidence interval or a claim that the two underlying analyses are identical.
A tolerance is meaningful only if its basis is documented. It might reflect the precision with which values are reported, known harmless numerical variation, or the size of a difference that matters for the review. These are not interchangeable justifications. In particular, selecting a generous tolerance after seeing an inconvenient discrepancy risks concealing a material difference.
Worked Example: Comparing Parameter Estimates at Different Scales
Suppose a hypothetical submitted run reports clearance \(CL=5.000\) L/h and volume \(V=0.0200\) L. The rerun reports \(CL=5.040\) L/h and \(V=0.0230\) L. For this comparison, set \(a=0.010\) in the relevant parameter units and \(r=0.010\) for clearance; for volume, set \(a=0.0010\) L and \(r=0.020\). These are illustrative review tolerances, not universal standards.
For clearance, the absolute difference is \(|5.040-5.000|=0.040\) L/h. The threshold is \(0.010+0.010\max(5.000,5.040)=0.010+0.0504=0.0604\) L/h. Because \(0.040\leq0.0604\), the clearance estimates are within the specified tolerance.
For volume, the absolute difference is \(|0.0230-0.0200|=0.0030\) L. Its threshold is \(0.0010+0.020\max(0.0200,0.0230)=0.0010+0.00046=0.00146\) L. Since \(0.0030>0.00146\), the volume estimates exceed the specified tolerance and warrant review. The small absolute size of the difference alone does not make it negligible; it is large relative to this parameter's scale and the stated rule.
Compare Like With Like
Before calculating differences, align each reported quantity. Parameter names alone may not be enough: confirm that estimates represent the same parameter, parameterization, and units. If one output reports a logarithm of a parameter and the other reports the parameter itself, their numerical values cannot be compared directly. Likewise, a parameter reported in one unit must be converted before comparison with the same parameter in another unit. Do not infer agreement from similarly named fields until their meanings are aligned.
For each aligned estimate, report the submitted value, reproduced value, signed difference, absolute difference, tolerance, and decision. Preserve enough digits for the calculation. If only rounded display values are available, say so: the apparent discrepancy may reflect display rounding rather than the underlying stored values, and the comparison has limited precision.
An objective function also requires a like-for-like comparison. Confirm that both values use the same objective definition and convention, including whether a reported value is, for example, a negative log-likelihood or a scaled version of one. An objective value without this context may not be comparable. Objective-function values are usually dimensionless; use a tolerance appropriate to that definition and document it. A small difference under the tolerance rule supports numerical agreement for that output, but does not demonstrate that the same optimum was reached or that all other outputs agree.
For predictions, select outputs that can be matched unambiguously: the same prediction type, individual or summary, dose and covariate conditions, and time point or event. Compare predictions in their stated units. A difference at unmatched times or under different conditions does not measure reproducibility. Prespecify which predictions will be checked; choosing only those that happen to agree can give a misleading picture.
Worked Example: Comparing an Objective-Function Value
In a hypothetical review, the submitted run reports an objective-function value of 1248.60 and the reproduced run reports 1248.74. Assume the reviewer has confirmed that both values use the same objective definition and convention. Use \(a=0.50\) objective-function units and \(r=0.00010\).
The absolute difference is \(|1248.74-1248.60|=0.14\). The tolerance is \(0.50+0.00010\max(1248.60,1248.74)=0.50+0.124874=0.624874\), or about 0.6249. Since \(0.14\leq0.6249\), the values are within the specified tolerance. This result supports agreement for the objective-function value under this rule; it does not show that parameter estimates or predictions agree, nor does it prove that the two runs followed identical computational paths.
Worked Example: Checking Selected Concentration Predictions
Consider hypothetical submitted and reproduced concentration predictions at three matched times for one prespecified profile. All values are in mg/L. Use \(a=0.020\) mg/L and \(r=0.030\).
| Time | Submitted (mg/L) | Reproduced (mg/L) |
|---|---|---|
| 1 h | 8.00 | 8.15 |
| 6 h | 2.00 | 2.10 |
| 12 h | 0.40 | 0.41 |
At 1 h, the difference is \(|8.15-8.00|=0.15\) mg/L and the threshold is \(0.020+0.030(8.15)=0.2645\) mg/L. This prediction is within tolerance. At 6 h, the difference is \(|2.10-2.00|=0.10\) mg/L and the threshold is \(0.020+0.030(2.10)=0.0830\) mg/L, so this prediction exceeds tolerance. At 12 h, the difference is \(|0.41-0.40|=0.01\) mg/L and the threshold is \(0.020+0.030(0.41)=0.0323\) mg/L, so it is within tolerance.
The comparison therefore identifies one prediction that needs attention, even though the other two pass. Reporting only an average difference could obscure that pointwise discrepancy. Here the conclusion is limited to these selected predictions; it is not a statement that every prediction from either run agrees or disagrees.
Make the Comparison Reproducible
A clear output comparison is an auditable calculation rather than a visual impression. Use the same rule for each value in a comparison set, and retain a table of inputs to the calculation and its result. A practical sequence is:
Match parameter definitions, objective-function conventions, or prediction conditions, and confirm the units.
Record the absolute and relative tolerances and why they are suitable before classifying discrepancies.
Compute the signed and absolute differences, calculate the threshold, and mark each output as within or beyond tolerance.
Check whether a flagged result reflects rounding, a mismatch in definitions or units, or an unexplained difference. Record unresolved discrepancies for follow-up.
The comparison table should include the tolerance values as well as the decision. A label such as “matches” without the underlying values and rule cannot be independently checked. It is also useful to keep parameter estimates, objective-function values, and predictions as separate comparison groups, because passing one group does not compensate for a failure in another.
Diagnostics, Alternatives, and Limitations
A failed comparison is a signal to investigate, not by itself an explanation of the cause. First recheck alignment, units, and the exact fields being compared. Then confirm that the calculation used unrounded values when available and that the tolerance was applied as written. If the discrepancy remains, report its magnitude and direction and identify the affected outputs. Do not silently widen a tolerance or substitute a more favorable summary.
Byte-for-byte equality is a stricter alternative: every compared character or stored value must match exactly. It is useful when exact identity is required and outputs are expected to be deterministically identical, but it can flag harmless changes in formatting or final digits. Numerical tolerance rules are more informative for numerical outputs, but depend on defensible, prespecified thresholds. Visual inspection is quick but subjective; it should not replace a documented calculation when a review decision depends on the result.
There are also limits to what numerical agreement can show. Similar parameter estimates can coexist with a prediction difference at a particular time, and similar predictions do not prove that parameter estimates agree. A close objective-function value alone does not establish agreement elsewhere. Conversely, a tolerance pass means only that the compared values meet the stated numerical rule. It is not a statistical test, proof of model correctness, or guarantee that all relevant outputs have been examined.
Common Pitfalls
- Using only relative difference near zero. Dividing by a value close to zero can produce an unstable or undefined comparison. Include a justified absolute tolerance.
- Comparing unmatched quantities. Values with different units, definitions, parameterizations, or prediction conditions are not made comparable by applying a tolerance formula.
- Relying on displayed rounding. Printed values can hide differences in stored results. Use full-precision outputs when available and report when only rounded values can be assessed.
- Treating a pass as proof of identity. A tolerance pass establishes agreement only for the checked values under the stated rule.
- Changing the threshold after seeing the results. This weakens the meaning of the decision. Document the tolerance rationale before applying it, and explain any necessary revision transparently.
Check Your Understanding
Try each question, then compare your reasoning with the solutions below.
- A submitted estimate is 10.00 mg/L and a reproduced estimate is 10.30 mg/L. With \(a=0.05\) mg/L and \(r=0.02\), are they within tolerance?
- Why should an objective-function comparison confirm that both runs use the same objective definition and convention?
- A prediction comparison passes at two matched times but fails at a third. Is it accurate to report that all selected predictions agree? Explain.
- What is one advantage and one limitation of exact, byte-for-byte comparison relative to a numerical tolerance rule?
Solutions. 1. The difference is \(0.30\) mg/L. The threshold is \(0.05+0.02(10.30)=0.256\) mg/L. Since \(0.30>0.256\), the estimates exceed tolerance. 2. Different objective definitions or scaling conventions can produce values that are not meaningfully comparable, even when both are labelled as objective-function values. 3. No. The third prediction exceeds the rule, so report the pointwise results and identify that discrepancy. 4. Exact comparison can establish character-for-character identity when that is required, but can flag harmless formatting or numerical last-digit variation that a documented tolerance would allow.