Reproducible Model Execution Across Environments

Why Reproducibility Is an Execution Property

A dosing model can be scientifically sound and correctly packaged yet still produce different results when run in development, testing, and production. A library update may change a calculation; an implicit default may vary by environment; or two systems may interpret a timestamp differently. If the same intended request can lead to different predictions or recommendations, a clinician or reviewer cannot reliably tell whether a changed result reflects new patient information, a changed model, or an execution difference.

Reproducible execution means controlling the conditions under which a specified model receives specified inputs and produces outputs. It is not enough to copy the model source code between systems. The relevant execution includes dependencies, runtime settings, configuration, input interpretation, and any stochastic elements. Earlier material on packaging a dosing model addresses how to prepare a model for deployment; here the emphasis is on making its behavior consistent across the environments where it is built, tested, and used.

Definition: An execution is reproducible across environments when equivalent, explicitly specified model inputs and configuration, processed by the intended model under controlled execution conditions, yield outputs that agree according to a defined reproducibility criterion. Agreement may mean identical output representations, numerical agreement within a stated tolerance, or equivalent clinical decisions, depending on the model and use.

These criteria are not interchangeable. Bitwise reproducibility requires identical output bits. Numerical reproducibility allows small differences within a defined numerical tolerance. Decision-level reproducibility asks whether the outputs lead to the same intended clinical action under specified rules. A system should state which criterion applies to each output. For example, an audit identifier may need exact agreement, while a floating-point concentration prediction may reasonably use a numerical tolerance. A matching dose category alone is not sufficient if the underlying predictions differ substantially.

Control the Execution Conditions

A useful starting point is to describe the complete execution context, not just the model file. Dependencies include the programming language runtime, numerical libraries, and any packages that influence data handling or calculations. A dependency specification should identify fixed versions rather than broad ranges such as “latest” or “version 2 or higher.” Where practical, record a lockfile or an equivalent resolved dependency list, and build from that fixed set.

A fixed list is not always enough. Two installations of the same package version can differ if they use different builds, system libraries, or processor-specific implementations. Reproducible builds therefore aim to produce the same deployable artifact from controlled inputs, and to retain enough build information to investigate how it was made. An immutable artifact promoted from testing to production reduces the risk that production silently rebuilds a different application from nominally similar source.

Runtime conditions also matter. Relevant settings may include the language version, operating system image, time zone, locale, character encoding, numerical precision, and thread configuration. A stochastic algorithm should have its random-number generator and seed controlled when repeatable output is required. Parallel calculations can sometimes change the order of floating-point operations; since addition is not exactly associative in finite precision, a different operation order can yield a small numerical difference. These details should be controlled when material to the model’s output, rather than assuming that the same source code guarantees the same result.

(a+b)+c \ne a+(b+c)

This inequality can occur in floating-point arithmetic even though the corresponding real-number expressions are equal. It helps explain why exact bitwise agreement may be unrealistic across different hardware or numerical libraries. The appropriate response is to define and justify a numerical agreement criterion, then check that differences remain within it. Numerical stability itself requires a fuller analysis; the practical point here is to identify execution conditions that can affect the result.

Configuration must be controlled as carefully as dependencies. Model choice, estimation or forecasting options, units, dose timing conventions, and any thresholds that affect a recommendation should be explicit inputs or controlled configuration values. Avoid hidden defaults whose values depend on the environment. Separate non-sensitive model configuration from credentials and other secrets, and do not allow a production secret or local machine setting to alter the mathematical behavior of the model.

Key practice: Define an execution contract that states the model artifact, fixed dependencies, relevant runtime settings, configuration, input conventions, stochastic controls, and required output agreement. A deployment that cannot identify these conditions is difficult to reproduce or investigate.

Reference Cases as Cross-Environment Anchors

A reference case is a fixed, clearly specified input with an expected result or expected range. It gives teams a common anchor for checking that environments execute the intended computation. A compact set should cover ordinary use and important boundaries, not only a convenient average case. In a dosing application, cases might include a typical patient profile, a profile near a recommendation threshold, and a case exercising a supported dosing interval or covariate value.

Reference cases are useful only when their conditions are unambiguous. State units, timing, configuration, expected outputs, and the agreement rule. If a result is allowed to vary numerically, set a tolerance appropriate to that output and the model’s use; do not choose a tolerance merely because observed discrepancies happen to fit it. Keep the reference input and expected result separate from environment-specific setup so that each environment is evaluated against the same case.

The examples below illustrate how execution control and reference cases work together. Their model values are hypothetical and are included only to explain reproducibility practices; they are not clinical dosing guidance.

Worked Example: Pinning the Calculation Environment

Suppose a hypothetical linear model reports exposure over a dosing interval as \(AUC = D/CL\), where \(D\) is the administered dose in milligrams and \(CL\) is clearance in litres per hour. For an input dose of 800 mg and clearance of 4.00 L/h, the expected interval exposure is:

AUC = \frac{800\ \text{mg}}{4.00\ \text{L/h}} = 200\ \text{mg}\cdot\text{h/L}

A development environment reports 200, while a test environment reports 200.0. These printed strings differ, but the underlying numeric values may be equal. The team first specifies whether this output requires identical serialization or numerical agreement; it does not infer the criterion from the display format. It then fixes the runtime and calculation dependencies, uses the same model artifact, and checks both environments with the same dose and clearance inputs.

If the outputs differ beyond the stated numerical tolerance, the comparison points to an execution discrepancy to investigate. A package update, different precision, or different configuration could be responsible. The value 200 mg·h/L is the reference result for this hypothetical case; matching it does not by itself establish that every behavior of the model is correct.

Worked Example: Finding a Hidden Configuration Difference

Assume a hypothetical model calculates a recommendation from a target exposure of 180 mg·h/L and estimated clearance of 4.50 L/h, using \(D = AUC_{\text{target}}\times CL\). In development, the configuration explicitly uses the target of 180 mg·h/L:

D_{\text{dev}} = 180\ \text{mg}\cdot\text{h/L}\times 4.50\ \text{L/h} = 810\ \text{mg}

A test environment instead uses an undocumented default target of 200 mg·h/L:

D_{\text{test}} = 200\ \text{mg}\cdot\text{h/L}\times 4.50\ \text{L/h} = 900\ \text{mg}

The 90 mg difference is not a numerical rounding effect. The input clearance and model formula agree, but configuration does not. The correction is to make the target explicit, supply the same controlled value in both environments, and rerun the fixed case. This example shows why dependency pinning alone cannot ensure reproducibility: the same program can behave differently when a hidden or environment-specific setting changes.

Worked Example: Applying an Explicit Agreement Rule

Consider a hypothetical reference case whose expected concentration is 12.00 mg/L. Suppose the team has justified an absolute numerical tolerance of 0.05 mg/L for this output. The test result in one environment is 12.03 mg/L. Its absolute difference from the reference is:

|12.03 - 12.00| = 0.03\ \text{mg/L}

Since \(0.03\ \text{mg/L} \le 0.05\ \text{mg/L}\), this result meets the specified numerical agreement rule. A second environment returns 12.08 mg/L:

|12.08 - 12.00| = 0.08\ \text{mg/L} > 0.05\ \text{mg/L}

That result does not meet the rule and should be investigated rather than waved through because it looks “close.” The tolerance is specific to this hypothetical output and is not a general clinical threshold. If the model’s action depends on a decision boundary, teams should also consider whether a numerically small difference could change that action; numerical agreement and decision agreement answer different questions.

A Practical Reproducibility Workflow

A disciplined workflow makes differences easier to detect and explain. It separates creation of a controlled execution from comparison of its behavior, while keeping the same cases and criteria in view.

Specify the execution contract.
List the model artifact, fixed dependencies, runtime conditions, configuration, input conventions, and required output-agreement criteria.

Build from controlled inputs.
Resolve dependencies to fixed versions and create a deployable artifact from the declared build inputs. Record relevant build and runtime information.

Promote the same artifact.
Use the tested artifact in production where possible, rather than independently rebuilding an application that is intended to be equivalent.

Run shared reference cases.
Provide the same specified inputs and configuration in each environment, then compare outputs under the stated exact or numerical agreement rule.

Investigate and document discrepancies.
Check dependency resolution, configuration, runtime settings, input interpretation, and stochastic controls before attributing the difference to the model itself.

This sequence is most informative when the comparison records both the input and the environment used for each run. Otherwise, an apparent model discrepancy may simply be a changed input, a different setting, or an unrecognized runtime difference. Keep the record focused on reproducibility: what was run, under which conditions, and whether its output met the agreed criterion.

Diagnosing Failure and Choosing an Approach

Common warning signs include a recommendation changing after deployment without an intended model change, a test passing only on one developer’s machine, unexplained differences in decimal precision, and the same request producing different results on repeated runs. These signs should trigger checks for unpinned dependencies, implicit defaults, locale or time-zone effects, uncontrolled randomness, and differences between the tested artifact and the deployed one.

There are several ways to control environments. A dependency lockfile records resolved package versions but may not control operating-system libraries or hardware behavior. A container can package much of the runtime environment and improve portability, but it does not automatically make the build immutable, control every host-level numerical difference, or guarantee correct configuration. A virtual machine can isolate more of the operating system, at greater resource and maintenance cost. The best choice depends on which execution differences matter and what the operational setting can support; none replaces explicit configuration and shared reference cases.

Reproducibility also has limits. A fixed environment cannot compensate for incorrect model logic, inconsistent clinical inputs, or inappropriate interpretation of a prediction. Exact bitwise identity may be impractical across distinct hardware or numerical implementations. Nor does agreement on a finite set of reference cases prove that every possible input will behave identically. The aim is controlled, testable execution with clearly stated boundaries, not a claim that all sources of variation have vanished.

Common pitfall: “The same source code was deployed, so the outputs must match.” This reasoning ignores dependencies, configuration, runtime conditions, and input interpretation. The correction is to compare the actual execution contract and the same reference cases, using an agreement criterion defined before examining the results.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Explain the difference between bitwise reproducibility and numerical reproducibility.
A model produces different recommendations in testing and production, although both use the same source code. Name two execution conditions to check before concluding that the model algorithm differs.
A reference value is 8.40 mg/L, the allowed absolute tolerance is 0.10 mg/L, and a run returns 8.52 mg/L. Does it meet the criterion? Show the comparison.
Why is a dependency lockfile helpful but not, by itself, sufficient to guarantee reproducible execution?
What is one limitation of using a finite collection of reference cases?

Solutions. (1) Bitwise reproducibility requires identical output bits; numerical reproducibility permits differences within a prespecified numerical tolerance. (2) Check, for example, hidden configuration defaults, resolved dependency versions, runtime settings, time zone, or stochastic seeds. (3) The difference is \(|8.52-8.40|=0.12\ \text{mg/L}\), and \(0.12>0.10\), so it does not meet the stated criterion. (4) A lockfile fixes package versions but may not fix system libraries, runtime settings, hardware effects, configuration, or input interpretation. (5) Passing the selected cases does not establish agreement for every possible input or boundary condition.