What an End-to-End Audit Establishes
A MICE project can produce plausible-looking tables even when the wrong input file was used, an undocumented change altered the analysis data, or the reported table no longer matches the saved results. An end-to-end audit checks the chain connecting the source file to the final reported result. It asks whether the files and records support the claims made about that chain, not merely whether the imputation code ran without an error.
The audit should begin with a defined scope. For example, an audit might cover one release of a hypothetical clinical analysis, from the frozen source file through a specified table of pooled estimates. It should identify which files, transformations, imputation run, analysis, and outputs are in scope. The audit can establish traceability and reproducibility within that scope; it cannot, by itself, establish that a study design is unbiased, that a missing-data assumption is true, or that the imputation model is scientifically appropriate.
A useful audit relies on several conditions: the source file and relevant project records are available; the audit identifies the exact analysis being checked; transformations and exclusions can be inspected; and comparison rules are defined before results are compared. If files are unavailable, records are incomplete, or outputs are compared using an unstated tolerance, the appropriate conclusion is that the audit is limited—not that the workflow has been verified.
Build the Audit Trail in Order
Auditing in the order the analysis runs makes gaps easier to locate. The following sequence connects evidence without treating any single record as sufficient proof.
Record the source file, its version or release, its checksum where appropriate, and the date and method by which it was obtained. A checksum can detect a changed file; it does not demonstrate that the file is the correct or complete source.
Check record identifiers, exclusions, transformations, variable types, and missing-value counts against documented expectations. Trace discrepancies to a specific step rather than silently adjusting counts to match a report.
Check that metadata identify the input data and the actual MICE settings used, along with the software environment and run status. A settings file describes intended settings; compare it with recorded settings or code that was executed.
Use automated checks for structural and workflow invariants, such as unique identifiers, expected numbers of imputations, preserved observed values, and allowed categories. Review failures rather than suppressing them.
Run the documented analysis from the saved inputs and compare generated tables and figures with the released versions using a stated exact-match or numerical-tolerance rule.
This sequence draws on the earlier material on logging MICE workflows, recording provenance, reporting the implemented workflow and software environment, and generating tables and figures from saved results. Here the emphasis is on joining those records into a single audit: each transition should have evidence that allows the next stage to be traced back to the previous one.
Check Provenance and Data Reconciliation
A practical audit starts with a small inventory: the source file and checksum, the code or documented steps that create the analysis data, the saved imputation output, the analysis code, and the released results. Check that identifiers are unique where required, that record counts reconcile, and that exclusions have documented reasons. Compare missingness counts by variable at the stage where they are expected; counts can legitimately change after exclusions or transformations, so comparisons must refer to the same stage.
Worked Example: Reconciling the Analysis Sample
Suppose a hypothetical project begins with 240 eligible records. Its flow record lists 12 exclusions before imputation, so the expected analysis sample is:
The saved analysis file contains 228 rows and 228 distinct participant identifiers, which is consistent with the flow record. The imputation log, however, says that 226 records were used. The discrepancy is \(228-226=2\) records. It cannot be resolved by pointing to the matching row count in the analysis file: the log refers to a different stage or run, or it is incorrect.
The auditor checks the run input identifier and finds that it refers to an earlier file containing 226 records. The discrepancy is therefore explained, but the run does not match the intended analysis sample. The appropriate finding is a workflow mismatch, not a successful reconciliation. The project must either rerun using the documented 228-record input or revise its stated analysis sample and report, with a documented justification.
The same logic applies to variable-level missingness. If 31 values of a variable are missing in the 228-record analysis file, the expected proportion missing at that stage is \(31/228=0.13596\), or about 13.6%. A different count in an imputation input should trigger investigation of the transformation or input file, not an assumption that the difference is harmless.
Checksums strengthen this comparison by identifying whether two files are byte-for-byte identical. They do not validate data content, transformations, or scientific meaning. If a file changes format or metadata while preserving the same data values, its checksum can change; conversely, a matching checksum only establishes identity with the file whose checksum was recorded.
Verify the Recorded Run and Environment
Run metadata should identify the specific input and output, the execution date or run identifier, the MICE settings, the software and dependency versions, and whether the run completed or failed. Settings commonly worth checking include the number of imputations, iteration count, methods assigned to variables, predictor configuration, seed, and any options that affect execution. The audit asks whether these records correspond to the run that generated the saved imputations, not merely whether a configuration file contains reasonable values.
Worked Example: Detecting a Run-Record Mismatch
A hypothetical analysis report states that it used 20 imputations, 15 iterations, and seed 73104. The saved run record shows 20 imputations, 15 iterations, and seed 73104, but its input-data checksum differs from the checksum of the approved analysis file. A separate execution log records the approved file checksum, 20 imputations, 15 iterations, and seed 73104, and identifies the output directory used for the released analysis.
The matching settings alone are not enough to link the saved imputations to the approved input: two runs can use identical settings on different data. The auditor therefore checks whether the saved output identifier and completion record link it to the execution log. If they do, the evidence supports the approved run; if they do not, the identity of the run that produced the imputations remains unresolved.
Next, the environment record is compared with the documented environment. For instance, if the lock record specifies package version 3.2.1 but the execution log reports version 3.3.0, the mismatch must be explained. It may reflect a harmless metadata error, or a software change that affects behavior. The audit should not declare the two environments equivalent without evidence, such as a documented regression test relevant to the changed version.
A seed is useful run metadata, but it is not a guarantee that rerunning will reproduce identical files. Random-number generation can depend on software versions, parallel execution, platform, and the order of computations. An audit should record these conditions and choose comparison rules suited to the claim being checked.
Automated Validations and Regenerated Results
Automated validations are most useful when they test explicit invariants. Examples include checking that each imputed data set has the expected number of rows, that participant identifiers are unchanged, that observed values remain unchanged where the workflow requires this, and that completed variables contain only permitted levels and plausible ranges. These checks detect certain classes of implementation error. They do not establish that the imputation model is adequate or that its assumptions hold.
Before regenerating an output, specify what counts as agreement. Text files or deterministic tables may be expected to match exactly. Numeric results from a stochastic workflow may instead be compared using a predeclared tolerance, appropriate displayed precision, and checks of the underlying saved estimates. A tolerance should be tight enough to catch meaningful changes and justified by the computation; choosing one after seeing a discrepancy weakens the audit.
Worked Example: Verifying a Regenerated Pooled Estimate
Consider a hypothetical analysis with five imputation-specific estimates of a regression coefficient: 1.20, 1.10, 1.30, 1.15, and 1.25. The saved within-imputation variances are each 0.0400. To check the pooled point estimate and its standard error, first calculate the mean estimate:
The deviations from 1.20 are 0, -0.10, 0.10, -0.05, and 0.05. Their squared deviations sum to 0.0250, so the between-imputation variance is:
The mean within-imputation variance is \(\bar{U}=0.0400\). Using Rubin's total variance \(T=\bar{U}+(1+1/m)B\), with \(m=5\), gives:
Thus, a regenerated result should have a pooled estimate of 1.20 and a standard error of about 0.218, subject to the analysis's specified pooling and reporting rules. If the released table instead gives an estimate of 1.20 but a standard error of 0.200, the point estimate alone does not establish agreement: the saved imputation-specific variances and pooling calculation should be checked. This calculation audits the reported arithmetic for these saved inputs; it does not validate the model or its assumptions.
A useful justification for checking intermediate quantities is that an end result can agree by coincidence. Two workflows may yield the same rounded estimate but differ in input data, number of imputations, or variance calculation. Checking identifiers, settings, estimates, and variances localizes discrepancies and makes the audit conclusion more informative than a simple statement that “the table matched.”
Interpreting Findings, Alternatives, and Limitations
An audit can use several complementary approaches. Manual review is valuable for understanding unusual records and documentation, but it is slow and difficult to repeat consistently. Automated assertions are reproducible and efficient, but only test conditions that have been encoded. Regenerating outputs provides direct evidence about reproducibility, but depends on accessible inputs and a sufficiently specified environment. A sound audit combines them rather than treating one as a substitute for the others.
- Missing or inconsistent records: investigate the specific pipeline stage, input identifier, and exclusion reason. Do not infer that a matching total proves the same individuals were included.
- Validation failure: determine whether the rule is wrong, the data violate an intended invariant, or the workflow changed. Do not disable a check merely to make a run pass.
- Output differences: distinguish changes in displayed rounding from changes in underlying estimates or uncertainty. Compare using the rule specified before regeneration.
- Incomplete provenance: state which link in the chain cannot be verified. A missing log is not evidence that a run did not occur, but it limits what can be established about that run.
A common mistake is to treat a successful rerun as proof that the original result is valid. The reasoning is: “The code runs and regenerates the table, therefore the analysis is correct.” The correction is that regeneration supports consistency between the available inputs, code, and output under the recorded conditions. It does not prove that the source data are accurate, the model is appropriate, the missingness assumptions are credible, or the reported interpretation is justified. Those require separate statistical and substantive assessment.
Check Your Understanding
Answer each question, then compare with the solutions.
- A source file has 180 records, and 9 are documented as excluded before imputation. The run log says 170 records were used. What should the audit conclude first?
- Why does a matching checksum not establish that an input file is scientifically correct?
- A validation confirms that all completed values lie within allowed ranges. What important claim does that check not establish?
- A regenerated table matches the released table after rounding, but the unrounded standard errors differ by 0.04. Is the result verified? What is the next step?
Solutions. (1) The expected analysis count is \(180-9=171\), so the log is short by one record. The auditor should identify the input and stage used and trace the discrepancy before calling the sample reconciled. (2) A checksum identifies file contents relative to a recorded checksum; it does not establish that the file contains the intended population, accurate measurements, or justified exclusions. (3) Range checks do not establish that the imputation model is appropriate or that missing-data assumptions are credible. (4) Not yet. Rounded agreement can conceal a meaningful difference. Apply the predeclared comparison rule to unrounded quantities, inspect the saved estimates and variances, and report whether the difference is within the justified tolerance.