Investigating Run-To-Run Discrepancies

Why Similar Runs Can Disagree

Two model runs can be described as “the same analysis” while differing in a detail that changes their inputs, execution, or reported results. A changed data file, an unrecorded configuration option, a different software library, or a run that stopped at a different point can all make a numerical comparison misleading. The useful question is not merely whether two numbers differ, but which run attributes are demonstrably the same, which differ, and whether the evidence supports a cause.

This tutorial focuses on investigating discrepancies between nominally equivalent runs. It uses the run records and common-input comparisons established in the earlier material on recording execution environments, preserving immutable run artifacts, and comparing runs on common inputs. A checksum, manifest, environment record, or log is evidence about a run; none is automatically a complete explanation of its results.

Definition: A run-to-run discrepancy is a difference between recorded attributes or reported results of two runs that were intended to be comparable. A discrepancy is explained only when the available evidence identifies a credible difference that accounts for it; otherwise, it remains unexplained. Runs described as nominally equivalent should not be treated as identical until their relevant inputs and execution conditions have been checked.

Establish What Is Being Compared

Before investigating a difference, specify the two runs and the quantity or artifact that differs. A useful comparison distinguishes at least four layers:

Inputs: data, model and configuration files, initial values, seeds, and other settings that can affect execution.
Run records: manifests that identify the files and settings associated with each run.
Execution: software versions, dependencies, relevant environment details, and log evidence about what happened.
Reported results: status, objective values, parameter estimates, precision summaries, and other outputs being compared.

The intended comparison matters. If the question is whether the same input artifacts were used, compare their recorded identities and checksums. If the question is whether two output values are numerically close, apply a stated comparison rule to those values. Similar outputs do not establish that inputs matched, and matching inputs do not guarantee bit-for-bit identical results across all execution environments.

Definition: A checksum is a digest computed from the bytes of a file using a specified algorithm. Matching checksums provide strong practical evidence that the files have identical bytes, assuming the same algorithm and reliable records. Different checksums establish that the bytes differ, but do not reveal whether the difference is scientifically meaningful. A checksum does not establish that a file is correct, complete, or the intended file.

A manifest makes checksums and related file details useful in context. It can record resolved file names or paths, file sizes, checksums, model and configuration identifiers, and the execution settings associated with a run. Compare the manifest entries for the files that actually affect the analysis, not just the names of the run directories. A file can retain its name while its contents change. Conversely, two differently named files can contain identical bytes.

Byte identity is not the same as semantic identity. For example, line-ending changes can alter a text file’s checksum even when the parsed records are unchanged. Sorting rows, changing numeric formatting, or modifying metadata may also change bytes. Such differences require inspecting the files and considering how the application reads them. Do not infer from a checksum mismatch alone that a change affected the model result.

Worked Example: A Data Checksum Does Not Match

Two runs use the same model version and are intended to use the same 120-record dataset. Their manifests identify a data file named analysis.csv in each run. Run A records a size of 8,420 bytes and checksum H1; Run B records 8,427 bytes and checksum H2. The checksums differ.

What the evidence establishes. The recorded files do not have identical bytes. The matching file name is not evidence that the contents match, and the seven-byte size difference is consistent with a file-level change. Neither fact identifies which records or values differ.

Inspection. A hypothetical comparison shows that all 120 parsed records and values are the same, but Run B’s file uses a different line ending on each of seven lines. The file bytes therefore differ even though the parsed analysis data appear equivalent under the documented reader.

Conclusion. The checksum discrepancy is real, but this inspection does not support a changed data value as the cause of any result difference. Record the byte-level difference and the evidence about parsed contents; do not replace the original files or claim they were identical. If the reader, parsing settings, or treatment of line endings is uncertain, the input equivalence remains unverified.

Use Environment Records and Logs as Diagnostic Evidence

Once the input comparison is understood, compare the environment records. Check software and dependency versions, relevant configuration, and any recorded platform or numerical-library details. The aim is not to assume that every environment difference changes results. It is to identify plausible execution differences and test them against the rest of the evidence. A different version is a lead for investigation, not by itself proof of causation.

Logs help establish what occurred during execution. Compare start and end times, warnings, messages about input loading or optimization, termination status, and indications of retries or interrupted execution. A “completed” status alone may not show that both runs reached the same computational stage or met the same convergence conditions. Conversely, a warning may be informational rather than consequential. Interpret each log message in the context of the run and its reported outputs.

A useful investigation keeps observations separate from interpretations. “Run B used dependency version 2.4 rather than 2.3” is an observation if supported by its environment record. “That version caused the parameter shift” is a causal conclusion and needs more evidence, such as a controlled rerun or a reproducible connection between the version difference and the result. Preserve the original artifacts while investigating; otherwise, the evidence needed to explain the discrepancy may be overwritten.

Worked Example: A Log Reveals Different Execution Outcomes

Two hypothetical runs have matching checksums for the model, data, and configuration files. Run A reports a final objective value of 812.6. Run B reports 819.1. Their environment records match on the recorded software versions. However, Run A’s log reports normal termination after the configured optimization, while Run B’s log reports that execution stopped after a time limit and labels the estimates as an intermediate result.

Compare the reported values. The absolute difference is \( |819.1-812.6|=6.5 \) objective units. This arithmetic confirms a numerical discrepancy; it does not determine its cause.

Evaluate the execution evidence. The logs show that the runs did not report results from the same stage of execution. Run B’s value is explicitly intermediate, so it is not directly comparable as a final result to Run A’s completed value.

Conclusion. The time-limited execution is a credible explanation for why Run B lacks a final result comparable to Run A. It does not establish what Run B’s final objective would have been. The appropriate record is that the reported values differ and the logs identify a different termination outcome—not that the completed analyses produced final values differing by 6.5 under otherwise identical execution.

Compare Numerical Results With an Explicit Rule

Numerical outputs may differ slightly even when runs use the same nominal model and inputs. Before calling a difference negligible, define a comparison rule appropriate to the quantity, its units, and the precision needed for the task. One useful form combines an absolute tolerance with a relative tolerance. For values \(a\) and \(b\), declare them close under the chosen rule when:

|a-b| \leq \tau_{\mathrm{abs}}+\tau_{\mathrm{rel}}\max(|a|,|b|)

Here, \(\tau_{\mathrm{abs}}\) is an absolute tolerance in the units of the quantity, and \(\tau_{\mathrm{rel}}\) is a dimensionless relative tolerance. The absolute component prevents the criterion from becoming impractically strict near zero; the relative component scales the allowance for larger values. This is a comparison convention, not a universal scientific threshold. Choose and report tolerances based on the purpose and scale of the comparison, rather than selecting them after seeing which result passes.

Justification: A purely relative difference, \( |a-b|/\max(|a|,|b|) \), is undefined when both values are zero and can be unstable when both are near zero. An absolute tolerance alone, in contrast, applies the same numerical allowance at every scale. The combined bound \( \tau_{\mathrm{abs}}+\tau_{\mathrm{rel}}\max(|a|,|b|) \) supplies a nonzero allowance near zero and a scale-dependent allowance for larger values. It defines a transparent numerical rule, but does not prove that two results are scientifically interchangeable.

Worked Example: Applying a Mixed Tolerance to an Estimate

Suppose two runs report a clearance estimate in L/h: \(a=4.812\) L/h and \(b=4.817\) L/h. For this hypothetical comparison, the team has specified \(\tau_{\mathrm{abs}}=0.010\) L/h and \(\tau_{\mathrm{rel}}=0.001\), or 0.1%.

Compute the observed difference. \( |a-b|=|4.812-4.817|=0.005 \) L/h.

Compute the allowed difference. The larger magnitude is 4.817 L/h, so the bound is \(0.010+0.001(4.817)=0.014817\) L/h.

Apply the rule. Since \(0.005 \leq 0.014817\), the estimates are close under the stated tolerance rule. This conclusion is limited to the clearance values and the specified rule. It does not show that all parameters, objective values, inputs, or execution conditions agree, nor does it establish that the tolerance is suitable for every scientific purpose.

Some sources of numerical variation are intentional. If the model or procedure includes stochastic components, a different random seed can produce different results even when other records match. A seed is therefore a relevant input, not incidental metadata. For deterministic computations, differences in numerical libraries, hardware, parallel execution, or rounding can sometimes affect reported digits; whether they matter depends on the computation and the scale of the observed difference. If the run records do not capture the relevant conditions, the discrepancy may remain unresolved.

A Practical Investigation Sequence

Use a sequence that narrows the search without treating any single check as decisive. At each stage, preserve the evidence and record what was checked, what differed, and what remains uncertain.

Define the discrepancy.
Name the runs and the specific artifact, status, or numerical output that differs. Include units and reported precision for numerical quantities.

Compare manifests and input checksums.
Check the resolved model, data, configuration, and other relevant files. For mismatches, inspect the contents or parsing consequences rather than assuming scientific impact.

Compare execution records and logs.
Check recorded software and dependency versions, settings, termination outcomes, warnings, retries, and evidence that each run completed the intended computation.

Assess numerical differences explicitly.
Apply a pre-specified, scale-appropriate tolerance where numerical closeness is the question. Keep that conclusion separate from claims about input identity or scientific equivalence.

State the conclusion at the strength of the evidence.
Classify the discrepancy as explained, partially explained, or unresolved. Identify any missing records or controlled comparisons needed to go further.

Failure Modes and Limits

Several tempting shortcuts produce conclusions stronger than the evidence allows:

“The filenames match, so the inputs match.” Names do not establish file identity. Compare checksums and, where needed, inspect contents and parsing.
“The checksums differ, so the analysis data differ.” Checksums compare bytes, not parsed values or scientific meaning. Determine what changed and how the software reads it.
“The environments differ, so that explains the results.” A difference is a candidate cause, not proof. Establish a plausible link and, if feasible, test it in a controlled comparison.
“The results are close, so the runs are equivalent.” A tolerance applies only to the specified values. Other outputs, inputs, or run outcomes may still differ.
“Both runs completed, so they completed the same task.” Status labels may conceal different termination conditions or intermediate results. Check the logs and reported outcome details.

Investigation is limited by record quality. Missing manifests, incomplete logs, mutable files, or environment details that were never captured may prevent a definitive explanation after the fact. A rerun can test a hypothesis, but it cannot recreate an undocumented historical environment with certainty. Report what is known, distinguish it from what is inferred, and keep unexplained differences visible rather than forcing a single cause.

Key takeaway: Trace discrepancies through input identity, execution evidence, and reported outputs in that order. Checksums establish byte-level evidence, logs describe recorded execution, and tolerances define numerical closeness; none alone proves that two runs are scientifically equivalent.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Two manifests list the same data filename but different checksums. What can be concluded immediately, and what cannot?
For \(a=0.0004\) and \(b=0.0007\), use \(\tau_{\mathrm{abs}}=0.0005\) and \(\tau_{\mathrm{rel}}=0.01\). Are the values close under the mixed-tolerance rule?
Two runs have matching input checksums, but one log reports normal termination and the other reports a time-limit stop. Why should their reported final estimates not automatically be treated as directly comparable?
A different software version is recorded for one run, but no other evidence links that version to the changed result. How should the finding be described?

Solutions. (1) The recorded files have different bytes, assuming the checksums were computed and recorded reliably. The mismatch alone does not show whether parsed values or scientific inputs differ; inspect the files and their use. (2) The observed difference is \( |0.0007-0.0004|=0.0003 \). The bound is \(0.0005+0.01(0.0007)=0.000507\). Since \(0.0003 \leq 0.000507\), they are close under this rule. (3) The logs indicate different execution outcomes, so the estimates may represent different computational stages; the time-limited run may not have produced a comparable final result. (4) Report the version difference as an observed discrepancy and a possible lead, not as an established cause. Further evidence or a controlled comparison is needed to support causation.