Recording Provenance for Imputed Datasets

Why Provenance Belongs With Completed Datasets

A completed dataset may be separated from the script, input files, and computing environment that produced it. Months later, an analyst may have a table of estimates and several imputed datasets but no reliable way to establish which source data, code revision, or run settings were used. A filename such as imputed_final_v2.rds does not establish that relationship.

Provenance is the recorded lineage of an artifact: what it came from, which transformations produced it, and which computational inputs and choices were involved. For completed MICE datasets, useful provenance connects each dataset to its source data, code revision, software environment, and resolved run configuration. It should also make it possible to follow a later analysis output back to those records.

Provenance is related to, but different from, the logging covered in the earlier material on Logging MICE Workflows for Auditing. Logs describe events as a workflow executes. Provenance is the durable set of links that remains attached to, or stored alongside, the artifacts. A log can help explain what happened; a provenance record should let a reviewer identify which inputs and run produced a particular saved dataset.

What to Record and Where to Attach It

A useful design has two levels. A run-level record stores facts shared by the imputation run, such as the input data identifier, code revision, software environment, configuration, and run identifier. A dataset-level record identifies one completed dataset and links it to that run. If a run produces \(m\) completed datasets, it is usually clearer to store the shared facts once and associate each of the \(m\) dataset records with the run, rather than copy the full shared record without a way to tell whether copies disagree.

At minimum, each link should be unambiguous and durable. A practical record can include:

• A unique run identifier and a unique identifier for each completed dataset.
• An identifier for the exact source-data artifact, plus a checksum or other controlled integrity identifier where appropriate.
• The code revision, such as a version-control commit identifier, and whether the working tree had uncommitted changes.
• The software environment actually used, including relevant software and package versions and the environment specification or lockfile identifier.
• The resolved run configuration, including values that were filled in from defaults, and the random-number information needed to identify the run settings.
• The saved dataset’s location or artifact identifier, creation time, and a checksum for the saved file or a clearly defined logical representation.

These items identify computational context; they do not establish that the model was appropriate, that the data were valid, or that the imputation succeeded statistically. Those questions require appropriate checks and interpretation. Provenance helps a reviewer locate the exact artifact and context in which to assess them.

The record can be embedded in a dataset object, stored as a sidecar manifest, or held in a database or artifact registry. Whichever arrangement is chosen, the link must survive copying and serialization. A sidecar should have a stable dataset identifier that also appears in the dataset’s filename or registry entry. If metadata are embedded, verify that the chosen serialization preserves them. Keep sensitive data out of the record: a source-data path, row-level details, or an unsalted hash of a small, guessable dataset can disclose information or enable a guessing attack. Prefer access-controlled identifiers and follow the applicable data-governance rules.

Worked Examples of Provenance Records

Why the Links Need Explicit Justification

A provenance chain is useful only when its links are stable and refer to the intended objects. A file path alone is a weak identifier because files can move or be overwritten. A human-readable run name is convenient but may be reused. Combining a unique identifier with an integrity check gives different information: the identifier names the record, while the checksum can help detect changes to the artifact it describes.

The logic is a simple chain of references. Let \(D_j\) be a saved completed dataset, \(R\) its run record, and \(I\), \(C\), \(E\), and \(G\) the recorded source data, code, environment, and configuration. If the dataset record identifies \(R\), and \(R\) identifies each of \(I,C,E,G\), then a lookup from \(D_j\) can reach all four recorded inputs. If any link is absent or ambiguous, the chain stops there.

This is why recording only a run log or only a version-control commit is insufficient. The log may not be retained with the data; the commit does not identify the input file, environment, or resolved settings. Likewise, a configuration filename is not enough if the file can change. Record the resolved values or an immutable reference and integrity identifier for the exact configuration used.

Diagnostics, Alternatives, and Limitations

Before relying on a provenance system, inspect it with practical trace tests. Choose a saved completed dataset and attempt to resolve its dataset identifier, run identifier, source-data identifier, code revision, environment, and configuration without relying on memory or an active analyst’s workspace. Repeat for an analysis output. Missing links, reused identifiers, and records that point only to mutable paths are evidence that provenance is incomplete.

• Uncommitted code: a commit identifier alone does not describe uncommitted edits. Record a clean/dirty status and preserve an appropriate patch or snapshot under suitable access controls, or prevent the run from proceeding with an unrecorded working tree.
• Implicit defaults: software defaults may change across versions. Record resolved settings, not merely the name of a configuration file.
• Changed serialization: a changed file checksum may reflect a different byte representation rather than changed values. State whether the checksum refers to file bytes or canonicalized logical content.
• Missing or inconsistent attachments: metadata can be lost when an artifact is copied, renamed, or exported. Test that each dataset identifier resolves to exactly one record and that the record points to the artifact actually used.
• Privacy and access: provenance can expose paths, study identifiers, or information that should not be public. Use protected registries and nonrevealing identifiers where needed; provenance should not become an unintended disclosure channel.

The main alternative is to rely on filenames, analyst notes, or a general workflow log. These can be useful supplements, but they usually lack the durable, structured links needed to identify an exact artifact and its context. A structured manifest or registry is more searchable and testable, but it adds maintenance and storage obligations. Neither approach guarantees scientific validity, and a checksum cannot substitute for preserving inputs or recording the choices that produced the data.

A further limitation is that provenance can become detached from the artifact. Store it in a way that is backed up and access-controlled alongside the dataset, and include stable identifiers in both places. When an artifact is intentionally modified, create a new identifier and record the relationship to the earlier version; do not overwrite the old record as if nothing changed.

Common Mistake