Why a Mismatch Needs an Investigation
A difference between two model outputs is a finding, not yet an explanation. The discrepancy might arise because the runs used different records, different code, different settings, or different software environments. It may also reflect expected variation from random-number generation rather than a defect. A useful investigation therefore does more than locate two unequal numbers: it isolates a cause and gathers evidence that distinguishes that cause from alternatives.
The earlier material on benchmark runs and random-seed variability provides useful context: comparisons need a defined reference, and stochastic outputs can differ even when a model is implemented correctly. Here the emphasis is the investigation itself. Each comparison should preserve the other relevant conditions so that a changed result can be attributed to the factor deliberately varied.
Before investigating, specify the quantity being compared, its units, the reference result, and a tolerance for agreement. Exact equality may be appropriate for a record count or a deterministic text output; floating-point calculations may require a numerical tolerance. The tolerance should be justified by the intended use and numerical precision, not selected after seeing the mismatch to make it disappear.
A Controlled Route from Symptom to Cause
A practical investigation proceeds from observable inputs toward less visible execution details. Preserve the original run before changing anything. Then compare data, code, settings, and environment systematically. Within each category, inspect the smallest relevant difference rather than making several corrections at once.
Record the compared outputs, units, tolerance, run identifiers, and the exact conditions under which the difference occurs. Re-run if possible to determine whether it is repeatable or stochastic.
Check file identity and version, record counts, identifiers, units, missingness, sorting, and inclusion or exclusion rules. Compare the actual analysis inputs, not merely filenames or high-level summaries.
Check the executable code or model specification, then run controls such as seeds, replicate counts, solver options, estimation settings, and requested outputs. Identify a specific difference that could affect the result.
When the earlier factors match, inspect relevant software and library versions, operating system, hardware or numerical backend, and dependency configuration. Record versions precisely.
Change only the suspected factor while holding other inputs fixed. If feasible, restore the original condition afterward. A cause is more convincing when the targeted change removes the discrepancy and reversing it restores the discrepancy.
This logic is a controlled comparison. If \(Y\) denotes the output, \(A\) is the suspected factor, and \(Z\) represents all other conditions, compare \(Y(A_1,Z)\) with \(Y(A_2,Z)\). Because \(Z\) is held fixed, an output change supports an effect of \(A\) for that test case. If several factors change together, the contrast is \(Y(A_2,Z_2)-Y(A_1,Z_1)\); the observed difference cannot be attributed uniquely to \(A\), because changes in \(Z\) are also possible causes. Interactions can complicate diagnosis, so a single-factor test is evidence for a specific controlled case, not proof that no other factor matters in other cases.
Worked Examples of Root-Cause Evidence
Worked Example: A Dose-Unit Mismatch
Hypothetical case. Two implementations calculate concentration after an intravenous bolus using \(C(t)=\frac{D}{V}\exp(-kt)\). The specified dose is 0.1 g, the apparent volume is 20 L, the elimination rate constant is 0.1 h\(^{-1}\), and the comparison time is 2 h. One run reports 0.004094 mg/L; the other reports 4.094 mg/L.
First put the dose into the units required by the formula. Since 1 g = 1000 mg, the intended dose is \(0.1\ \mathrm{g}\times1000\ \mathrm{mg/g}=100\ \mathrm{mg}\). The expected concentration is
Rounded to four significant figures, this is 4.094 mg/L. If a run instead reads the numeric value 0.1 as milligrams, its calculation is
The ratio \(4.093654/0.004093654\) is approximately 1000, matching the g-to-mg conversion factor. That pattern makes a unit mismatch plausible, but the ratio alone is not enough to establish it. The investigation checks the stored dose and unit metadata, confirms that one path treats 0.1 as mg while the intended input is 0.1 g, and reruns the same code and settings after converting the input to 100 mg. If that targeted change gives approximately 4.094 mg/L, while restoring 0.1 mg returns approximately 0.004094 mg/L, the controlled rerun supports the unit interpretation as the cause. Preserve both input values and the corresponding outputs as evidence.
Worked Example: An Unexpected Difference in the Analysis Records
Hypothetical case. A review compares the mean concentration at one nominal time in two analysis files. The reference file contains six observations, in mg/L: 1, 2, 3, 4, 5, and 6. The run file contains only the first four. The analysis code and settings appear identical.
For the reference file, the mean is \(\bar{x}_{\mathrm{ref}}=(1+2+3+4+5+6)/6=21/6=3.5\ \mathrm{mg/L}\). For the run file, it is \(\bar{x}_{\mathrm{run}}=(1+2+3+4)/4=10/4=2.5\ \mathrm{mg/L}\). The discrepancy is \(2.5-3.5=-1.0\ \mathrm{mg/L}\). This arithmetic shows that the observed record difference is sufficient to produce the difference in these means; it does not yet explain why two records are absent.
The investigator compares identifiers and finds that observations 5 and 6 are present in the reference file but absent from the file actually passed to the run. The recorded analysis rule requires all six eligible observations, and both missing records meet that rule. Re-running the unchanged calculation with the six-record file gives 3.5 mg/L. Re-running with the four-record file gives 2.5 mg/L. These checks support the input-record discrepancy as the direct cause of this mean difference. If the two missing observations had instead been correctly excluded by a prespecified rule, the issue would be a question about the intended analysis, not automatically a data-processing error.
Worked Example: Separating an Environment Difference from Code
Hypothetical case. Identical deterministic input and code produce 1.23456 mg/L in environment A and 1.23451 mg/L in environment B. Their absolute difference is \(|1.23456-1.23451|=0.00005\ \mathrm{mg/L}\). Relative to the environment A result, the percentage difference is \((0.00005/1.23456)\times100\%\approx0.00405\%\). Whether this is acceptable depends on the prespecified tolerance and the intended use.
The code, input file, and model settings are checked and match. A controlled cross-over then runs the same code and data under both environments: environment A produces 1.23456 mg/L on both machines or installations, and environment B produces 1.23451 mg/L on both. The relevant numerical library versions differ between environments. This pattern supports an environment-associated difference, rather than a difference in the checked data or code. It does not, by itself, prove which internal library operation creates the difference. A stronger diagnosis would identify the relevant operation or dependency and reproduce the shift with a minimal test, while continuing to treat the conclusion as limited to the tested configurations.
Interpreting Evidence and Knowing When to Stop
Evidence is strongest when it forms a short, reproducible chain: the mismatch is repeatable under documented conditions; a specific difference is observed; a targeted change to that difference changes the output as predicted; and restoring the original condition restores the original discrepancy. Keep the original run intact. Record input identities or checksums, code revision, settings, software versions, output values, units, tolerances, and the commands or steps used for each comparison. This makes the reasoning inspectable without relying on memory or an informal description.
Not every discrepancy has one isolated cause. Data and code differences can interact, and a software-version change may alter a numerical calculation only for particular inputs or settings. When a one-factor test fails to explain the symptom, use a small factorial comparison of the plausible factors: test the combinations while keeping all other conditions fixed. A pattern in which the effect of one factor depends on another is evidence of an interaction, not a reason to force a single-factor explanation.
For stochastic simulation, different random seeds can create different replicate outputs even when data, code, settings, and environment otherwise match. Repeat the comparison with the same seed and the same replicate configuration before interpreting a difference as an implementation defect. Conversely, matching seeds alone may not guarantee matching streams across different software or random-number-generator versions. Distinguish expected replicate variability from a persistent discrepancy by documenting the randomization conditions and checking whether the difference is within the intended simulation variability.
A useful comparison may be a direct reproduction, a targeted input correction, a code or settings swap, or an environment cross-over. These approaches answer different questions. A direct reproduction establishes that the symptom is repeatable. A targeted change tests a proposed cause. A cross-over helps separate environment from code or data. None substitutes for defining the expected result and comparison tolerance. If the output is within a justified tolerance, the numerical difference may be immaterial for the stated purpose, but that conclusion should not be confused with identifying why the values differ.
Common Pitfalls
- Changing several things at once. A new dataset, revised code, and updated software may make the output agree, but they do not reveal which change mattered. Change one factor per test where feasible.
- Inferring cause from a suggestive ratio. A 1000-fold difference is consistent with a gram-to-milligram mismatch, but verify the actual stored units and demonstrate the effect with a controlled rerun.
- Comparing labels instead of contents. Matching filenames do not guarantee matching records. Compare file versions, identifiers, values, and relevant processing rules.
- Calling every small difference a bug. Floating-point precision, solver controls, and stochastic variation can produce differences. Judge them against a tolerance and an understanding of the calculation.
- Claiming more than the test supports. A cross-over can implicate an environment difference without identifying the precise software operation responsible. State the scope of the evidence and any unresolved alternatives.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A run differs from its reference after the analyst updates the dataset, changes a solver tolerance, and installs a new software version. Why can the mismatch not yet be assigned to any one of these changes?
- A concentration calculation uses 0.2 g, but the implementation treats the numeric value as 0.2 mg. By what factor do the resulting concentrations differ if all other terms are unchanged? What evidence would support this as the cause?
- A deterministic output is 5.0000 mg/L in one environment and 4.9998 mg/L in another. State the absolute difference and explain why its acceptability cannot be decided from these numbers alone.
- A stochastic simulation differs when run with different seeds. What should be held fixed or checked before concluding that the implementation is defective?
Solutions. (1) Three factors changed together, so their effects are confounded. Reproduce the comparison and vary one factor at a time, or use a controlled combination of runs that separates their effects. (2) The factor is 1000, because \(0.2\ \mathrm{g}=200\ \mathrm{mg}\) and \(200/0.2=1000\). Verify the actual input unit and show that converting the input to 200 mg, with code and settings fixed, produces the expected change; restoring 0.2 mg should restore the smaller result. (3) The absolute difference is \(|5.0000-4.9998|=0.0002\ \mathrm{mg/L}\). Its acceptability depends on the previously specified tolerance and the use of the result; the environment comparison may help diagnose its origin but does not define whether it matters. (4) Use the same data, code, settings, seed, and replicate configuration where possible, and check whether the discrepancy persists across repeated runs. If random-number streams can differ by environment or generator version, document and account for those conditions before attributing the difference to defective implementation.