Why Provenance Matters
An external-control analysis depends on more than the statistical model. The analyzed values came from particular records, extracted at a particular time, transformed by particular rules, and passed to particular analysis code. If those steps cannot be reconstructed, a result may be numerically reproducible only by accident. It may also be unclear what a variable actually measures in clinical terms.
A provenance audit asks where the data came from and what is known about its history. A lineage audit follows the data through the operations that created the analysis-ready variables and results. These audits complement work such as harmonizing baseline covariates: harmonization concerns the comparability and representation of variables, while provenance and lineage establish which source information and operations produced those representations.
A useful audit follows a variable in both directions. Starting from a reported result, trace back to the model input, the analysis-ready variable, each transformation, and the source records. Starting from the source, trace forward and establish which records and operations contributed to the result. The audit should preserve enough detail to repeat either journey.
What to Record Along the Data Path
For each variable that affects eligibility, baseline description, treatment assignment, follow-up, or an outcome, identify the source and the operations applied to it. A compact lineage record should be specific enough that another analyst can locate the input, reproduce the transformation, and understand what the resulting value represents.
| Lineage stage | Audit question | Useful evidence |
|---|---|---|
| Source | Which system, table, or file supplied the information, and what population and period does it cover? | Data dictionary, source description, access or extraction record |
| Extraction | When and how were records retrieved, and which version or snapshot was used? | Query, query parameters, timestamp, file version or checksum |
| Transformation | Which filters, recodings, unit conversions, derivations, or rounding rules were applied? | Versioned code, specifications, intermediate outputs, validation checks |
| Analysis | Which dataset and variables entered the analysis, under which code and settings? | Analysis script, software and package versions, input and output files |
| Reporting | Can each displayed estimate or table be traced to an analysis output? | Output logs, table-generation code, documented rounding rules |
Names such as “final,” “clean,” or “latest” are not adequate version identifiers: they do not establish which file was used or whether it later changed. Record stable identifiers where feasible, such as a dated snapshot, a version number, or a checksum. A checksum can help establish that two files are identical, but it does not explain what the file contains or whether it is clinically appropriate.
Worked Audits
Worked Example: Verifying a Unit-Conversion Lineage
In a hypothetical external-control dataset, a baseline glucose measurement is stored in milligrams per deciliter (mg/dL), while the analysis specification requires millimoles per liter (mmol/L). The documented conversion is to multiply by 0.0555. Three source values are 90, 108, and 126 mg/dL. Audit whether the derived values match the stated transformation.
Apply the conversion to each value. For the middle record:
The other values give \(90\times 0.0555=4.995\) mmol/L and \(126\times 0.0555=6.993\) mmol/L. Their mean on the source scale is \((90+108+126)/3=108\) mg/dL; their mean after conversion is \((4.995+5.994+6.993)/3=5.994\) mmol/L, as expected for a constant linear conversion.
The audit should confirm more than the arithmetic. It should verify that the source field is indeed measured in mg/dL, the analysis field is in mmol/L, the conversion factor appears in the executed code, and any rounding occurs at the documented stage. If a factor of 0.00555 were used instead, the 108 mg/dL value would become \(0.5994\) mmol/L, a tenfold discrepancy. This points to a transformation or unit-provenance problem, not to biological variation in the patients.
Worked Example: Tracing a Binary Analysis Input to a Source Revision
Suppose a hypothetical external-control analysis includes 60 patients and a prespecified binary outcome. Under one archived source extract, 18 patients have the outcome recorded. A later source revision changes three of those recorded flags to no, leaving 15 outcome records among the same 60 patients. Assume, solely to illustrate the effect of the changed input, a common-probability binomial model with a Beta(1,1) prior. Compare the posterior means under the two extracts.
For \(y\) events among \(n\) patients, this model gives a Beta(\(1+y,1+n-y\)) posterior. Its mean is the first shape parameter divided by the sum of the shape parameters. For the archived extract:
For the revised extract:
The posterior means differ by \(0.3065-0.2581=0.0484\), or about 4.84 percentage points. This calculation does not determine which source revision is clinically correct. It shows why the extract version and the origin of the changed flags must be traceable: the same stated model and prior yield different results from different analysis inputs. The Beta-binomial calculation also assumes that the observations share a common event probability and are conditionally independent; it is an illustration of input sensitivity, not a justification for that model in a particular study.
Worked Example: Measuring a Rerun Discrepancy
An analyst reruns a hypothetical continuous baseline-variable derivation on the 20 patients in a saved analysis file. The original run reports a mean of 7.42 units; the rerun reports 7.39 units. Comparing the two derived datasets, 18 patient-level values match exactly. Two differ: one rerun value is 0.4 units higher and another is 0.2 units higher than the original. Calculate the mismatch rate and check whether the differences explain the change in the mean.
The patient-level mismatch rate is the number of nonmatching values divided by the number compared:
The differences sum to \(0.4+0.2=0.6\) units. Across 20 patients, they would raise the rerun mean by \(0.6/20=0.03\) units. The observed difference is \(7.42-7.39=0.03\) units in the opposite direction, so the stated patient-level differences do not explain the reported means as described. That inconsistency is itself an audit finding: either the direction of the differences, the reported means, or the comparison has been recorded incorrectly.
If the differences instead were \(-0.4\) and \(-0.2\) units in the rerun, their sum would be \(-0.6\), lowering the mean by \(0.6/20=0.03\) units and agreeing with the reported change. A provenance record should not merely say that a rerun was performed; it should preserve the compared inputs, transformation version, and reconciliation result.
A Reproducible Audit and Its Logic
A practical audit can be organized as a chain of verifications. Each step should identify both the claim being checked and the evidence that could support or contradict it.
Record the analysis question, dataset snapshot, variable definitions, and reported outputs being audited. Without a fixed target, “reproduction” can silently refer to a different data version or analysis.
For each reported quantity, identify its model input, derived variable, transformation code, extracted source, and source version. Record gaps rather than filling them with assumptions.
Starting from the documented source snapshot, rerun the recorded extraction and transformation steps and compare intermediate and final outputs with the archived analysis files.
Classify each mismatch by stage, document its cause and consequence, and determine whether the analysis requires correction, qualification, or a sensitivity analysis.
The justification for this chain is simple but depends on explicit assumptions. Let \(D_0\) be a fixed source snapshot and let \(f_1,\ldots,f_k\) be the documented extraction and transformation operations. The analysis input is \(D_k=f_k(\cdots f_2(f_1(D_0))\cdots)\). If the source snapshot, operations, and relevant execution settings are fixed, and each operation is deterministic, then replaying the composition yields the same \(D_k\). Applying the same analysis procedure to that same input should then reproduce its output, up to any documented numerical tolerances or stochastic settings.
A failure to reproduce the input or output therefore signals that at least one required element is not fixed or not faithfully recorded. Possible causes include a changed source snapshot, undocumented manual edits, different software behavior, or stochastic computation without a recorded seed. Successful reproduction is evidence of procedural consistency; it does not prove that the source is accurate, the variable is clinically valid, or the external controls are comparable to trial participants.
Diagnostics, Alternatives, and Limits
Useful diagnostics include comparing record counts and summary values at successive stages, checking plausible units and value ranges, confirming that every analysis variable has an identifiable source and transformation, and rerunning the pipeline from a frozen input. Counts and summaries are checks, not proof: two different files can have identical counts and means. Where feasible, compare record-level derived values or stable identifiers as well as aggregate summaries. Document tolerances for numerical comparisons in advance rather than choosing them after seeing a discrepancy.
Different audit strategies answer different questions. A documentation review can identify missing source descriptions, version records, or specifications, but cannot establish that code executed as written. An independent rerun tests whether recorded inputs and operations reproduce the analysis, but can reproduce a consistently wrong transformation. Aggregate validation is fast and can detect gross changes, but may miss offsetting record-level errors. These approaches are complementary; no single check establishes data quality, clinical validity, and reproducibility at once.
A provenance gap is consequential when the missing information prevents an analyst from determining what was measured, which records contributed, or how the reported result was created. For example, an unexplained change in source version may alter the analysis population or event count. An undocumented unit conversion may change the clinical scale of a covariate. If a correction cannot be resolved from evidence, report the uncertainty in the data history and consider whether results from plausible alternatives should be shown. Do not present an undocumented choice as a verified fact.
The audit also has limits. Records may have been irreversibly de-identified or aggregated, source systems may not retain old versions, and proprietary extraction procedures may not be fully accessible. A complete technical trail cannot establish that an electronic record accurately reflects a patient’s clinical state. Nor does reproducibility establish the assumptions needed for a causal comparison. Those are distinct questions and should not be inferred from a clean lineage diagram.
Common Mistake
Check Your Understanding
Answer the questions, then compare your reasoning with the solutions.
- What is the difference between data provenance and data lineage?
- A variable’s documented conversion is \(x_{\text{analysis}}=0.2x_{\text{source}}\). A source value is 35 units. What derived value should appear, and what should be checked besides the arithmetic?
- A rerun matches 45 of 50 patient-level derived values. What is the mismatch rate, and does a matching aggregate mean establish that the remaining values are correct?
- What can an independent rerun establish, and what can it not establish about clinical validity?
Solutions. (1) Provenance documents the origin and history of data; lineage traces the inputs and operations that connect source records to derived variables and results. (2) The derived value is \(0.2\times35=7\) analysis units. Verify the source field, units, documented conversion, executed code, and rounding stage. (3) Five values mismatch, so the rate is \(5/50=0.10\), or 10%. A matching aggregate mean does not establish record-level correctness; differences can cancel. (4) A rerun can test whether recorded inputs and operations reproduce the analysis. It cannot establish that the source accurately represents clinical reality or that the variable is clinically valid.