Why Computational Provenance Matters
An NLME fit is produced by more than a model formula. It also depends on the data supplied to the program, the implementation and configuration used, the estimation algorithm, software and engine versions, random-number generation, and details of the computing environment. These factors can matter especially when estimation involves stochastic algorithms or numerically sensitive calculations. Capturing them makes a run traceable: a later analyst can determine what was executed and what conditions would need to be recreated for a meaningful rerun.
A saved model script alone is not a reproducibility record. It may call a different software release, load a changed data file, inherit an undocumented default, or run with a different number of threads. Conversely, a long log of machine details is not enough if it cannot be linked to the precise data, code, and settings for one run. Provenance is most useful when it connects these parts through a unique run identifier.
Reproducibility can mean different things. Bitwise reproducibility means a rerun produces identical output bits. Numerical reproducibility means results agree within a stated numerical tolerance. Scientific reproducibility means the rerun supports the same substantive conclusions, even if small numerical differences occur. These are not interchangeable claims. Differences in floating-point arithmetic, parallel execution, software libraries, or random-number streams can prevent bitwise identity while leaving estimates numerically close. A provenance record should make the intended target explicit rather than promise exact identity without qualification.
What a Run-Level Record Needs
Record facts at the level of the individual execution, not only at the project level. A practical manifest can be a structured text file stored beside the run outputs. It should distinguish fixed inputs from execution-specific information and avoid relying on undocumented defaults.
| Record group | Useful details | Why it matters |
|---|---|---|
| Inputs and implementation | Run identifier; data-file checksum; model-code and configuration revision; input transformations or generated files | Identifies exactly what was fit, not merely the project or filename |
| Software and engine | Operating system; NLME application and version; estimation engine and version; relevant package or library versions | Defaults, algorithms, and numerical routines can change across releases |
| Estimation configuration | Method; iteration or evaluation limits; tolerances; initial values; bounds; enabled options; explicitly stated defaults | Defines how the software was asked to compute the fit |
| Randomness | Seed; random-number generator and version; stream or substream settings; purpose of each random stream | A seed is meaningful only together with the generator and the way draws are consumed |
| Execution environment | Hardware or compute service; processor architecture; thread count; precision settings; relevant environment variables | Parallelism and numerical libraries can affect execution and rounding |
| Execution record | Start and end times; command or job identifier; exit status; interruption or restart details; log location | Shows what actually ran, including whether the run completed as configured |
Checksums help identify files whose names have stayed the same while their contents changed. For example, a SHA-256 checksum is a fixed-length digest computed from a file’s bytes. Store the digest alongside the filename and file size; the digest is an identifier for checking content equality, not a substitute for retaining the file. Similarly, a version-control commit identifies code only if the repository and relevant dependencies remain accessible. Preserve configuration files and generated inputs when they are not fully recoverable from the code revision.
Worked Examples
Worked Example: Creating a Manifest for One Hypothetical Fit
A hypothetical team fits a two-compartment NLME model to a locked dataset. The run uses a stochastic estimation procedure. The team wants to distinguish this execution from other candidate runs and to give a colleague enough information to recreate its computational conditions.
They assign the run identifier PK-042-R03. The manifest records the input file as analysis_data_v5.csv, with SHA-256 digest 7f2a…91c4 (abbreviated here for display), and the model code revision as commit a81d6e2. It records application version 6.2.1, estimation-engine version 4.8.0, and the explicitly selected estimation method. It also stores the initial values, parameter bounds, maximum iterations of 1,200, and convergence tolerance of \(10^{-6}\), rather than relying on defaults.
The stochastic settings are recorded as generator PCG64, seed 27183, and one stream for this run. The environment record states Linux x86-64, 8 threads, and double-precision arithmetic. Finally, the job identifier, start and finish timestamps, exit status, and log path are stored.
This is more informative than recording “seed 27183” alone: the seed is linked to a named generator, engine, configuration, and execution. The digest provides a check that a proposed rerun uses the same data bytes. If the data digest or code revision differs, the rerun is not an exact recreation of this run, even if its output happens to look similar.
Worked Example: Describing a Numerical Difference
Suppose two hypothetical executions use the same data and model configuration, but run on different processor environments. They report the same objective quantity to the displayed precision except for values 128.430 and 128.432. The team must describe the size of the discrepancy without treating it as automatically consequential or automatically negligible.
The absolute difference is
Using 128.430 as the reference, the relative difference is
The appropriate report is that the objective values differ by 0.002, or about 0.00156% relative to the first value. That arithmetic describes the discrepancy; it does not establish whether the estimates or scientific interpretation are equivalent. The team should retain both manifests and compare results against a tolerance chosen for the intended numerical-reproducibility claim. The environment difference is a plausible source to investigate, not proof of the cause.
Worked Example: Repeating a Stochastic Fit With Recorded Seeds
In a hypothetical reproducibility check, the same locked inputs and configuration are used for four runs, each with a separately recorded seed. The estimated typical clearance, in L/h, is 1.82, 1.79, 1.84, and 1.80. The goal is to summarize variation across these executions, not to claim that four runs establish a universal property of the estimator.
The arithmetic mean of these four estimates is
The range is \(1.84-1.79=0.05\) L/h. These values show that the recorded runs differed by as much as 0.05 L/h in this small check. To make the comparison interpretable, the team must also verify that data, code, engine, settings, and environment were held fixed apart from the intended seed change. If other conditions changed, observed differences cannot be attributed to random variation alone.
Why Seeds and Settings Need Context
A random-number seed initializes a sequence, but it does not by itself specify every random draw an application will use. The generator algorithm maps a seed into a stream; software may consume that stream differently after a code change, different execution path, or parallel scheduling change. Some engines also use multiple streams or substreams. Therefore, record the generator, its version where available, and stream configuration as well as the seed. If a run uses several random components, document which streams are assigned to which components when the software supports that distinction.
A useful way to understand the requirement is to represent the fitted result abstractly as
where \(R\) denotes the run result, \(D\) the data, \(C\) the model code and configuration, \(S\) the software and estimation engine, \(E\) the execution environment, and \(U\) the random-number state and stream behavior. Recording only one input, such as \(U\), cannot determine the output if the other arguments can change. This is why a seed is a component of provenance rather than a reproducibility guarantee.
A Practical Capture Process
Capture provenance as part of launching and completing a run. Retrospective reconstruction is less reliable: defaults may not be logged, temporary files may disappear, and package versions may be difficult to recover. A compact process also helps ensure that every candidate execution receives its own record.
Record the run identifier, data checksum, code revision, configuration, and any generated input files. Keep the referenced artifacts available.
Capture application and engine versions, estimation method, initial values, limits, tolerances, random generator and seed, and all settings that affect the calculation.
Store environment details, thread and precision settings, command or job identifier, timestamps, completion status, and logs. Mark restarts or interruptions explicitly.
Say whether the aim is bitwise identity, agreement within a numerical tolerance, or scientific agreement. Record the tolerance or comparison rule when one is used.
Store the manifest with the run’s artifacts and ensure the run identifier appears in logs or output metadata. A record that cannot be mapped to a specific execution is difficult to use.
Diagnostics, Alternatives, and Limits
A provenance record is suspect if its entries are incomplete, contradictory, or impossible to verify. Warning signs include a seed with no generator named, a version listed only as “current,” an input filename with no checksum or immutable revision, unexplained defaults, a log whose job identifier does not match the manifest, or a record that omits whether a run was restarted. Another warning sign is claiming exact reproducibility while allowing variable thread counts or unrecorded library versions.
When a rerun differs, first compare the recorded inputs and execution conditions. If they match, repeat the computation under the same settings to see whether the discrepancy recurs; for a stochastic method, use recorded seeds and, when needed, a planned set of seeds. Change one computational factor at a time when diagnosing a suspected source. This is a provenance check, not a substitute for evaluating whether a fitted model is adequate or interpreting its results.
There are alternatives to recording the full environment manually. A container or managed computing image can preserve many software dependencies; a lockfile can record package versions; a workflow system can capture commands and job metadata. These approaches reduce omissions, but none replaces the run manifest. Containers may still depend on hardware, operating-system interfaces, external services, or parallel execution. Lockfiles may omit system libraries. Workflow logs may not preserve the exact input data or random stream. Use these tools alongside a concise record of the specific run.
Provenance also has limits. It can show how a result was generated and support a controlled rerun, but it does not prove that the model is appropriate, that the data are correct, or that the numerical solution is scientifically valid. Nor can every environment be preserved indefinitely. Record enough detail to make the intended computation recoverable, state what cannot be guaranteed, and avoid describing a numerical match as proof of scientific correctness.
Common Mistake
Wrong reasoning: “The script and seed are saved, so anyone can reproduce the fit exactly.” This overlooks data changes, software and engine updates, generator differences, hidden defaults, and execution behavior such as parallel scheduling.
Correction: Treat the seed as one field in a run-specific provenance record. Link it to immutable or checksummed inputs, a code revision, explicit settings, software and engine versions, environment details, and execution logs. Then state whether the intended standard is bitwise, numerical, or scientific reproducibility.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A team records a seed but not the random-number generator. What part of the computational provenance is missing, and why does it matter?
- Two runs have the same model code and seed, but use different data files with the same filename. What additional record would reveal this difference?
- A rerun produces a slightly different objective value after a software update. Is that automatically a failure of scientific reproducibility? Explain what should be checked.
- Name two reasons a container does not, by itself, guarantee bitwise reproducibility.
Solutions. (1) The generator and, where relevant, its version and stream configuration are missing. A seed initializes a sequence for a particular generator; another generator or a changed pattern of consuming draws may produce different random values. (2) Record a checksum, such as a SHA-256 digest, for each data file and retain the referenced file. Different content will generally have a different digest even when the filename is unchanged. (3) No. First compare data, code, settings, engine and software versions, and environment details; then determine whether the difference is within a pre-established numerical tolerance and whether substantive conclusions remain consistent. The software update is a possible explanation, not a conclusion by itself. (4) A container may not fix processor architecture or floating-point behavior, and parallel scheduling or external system libraries may still differ. Thus it can improve environmental control without ensuring identical output bits.