What a Traceability Review Establishes
A reported model result is useful only if a reviewer can determine which project materials and recorded run it refers to. A repository may contain scripts, configuration files, output tables, and a README, yet still leave it unclear which version of the code produced a particular table. Reviewing for traceability means checking whether those pieces form a consistent, inspectable chain of evidence.
Traceability is not the same as showing that an analysis is statistically appropriate, and it does not establish that the result can be reproduced by running the project. Those are separate questions. The task here is to assess the links among the reported result, its output artifact, its run record, and the project state identified by that record. Recording Run Provenance introduced the purpose of run-level records; here the emphasis is on reviewing whether the repository’s evidence actually connects.
A useful review starts with a specific reported claim, not with a general impression that the repository looks organized. For example, “The final model’s estimated clearance was 4.8 L/h” is more reviewable than “The model was updated.” The reviewer asks which report or output contains the estimate, which run produced that output, and which project state and inputs that run record identifies.
The Evidence Chain and Its Assumptions
A compact way to represent the review is as a sequence of links. Each link needs enough identifying information for a reviewer to resolve it to a specific artifact, rather than to a broad category such as “the latest results” or “the model code.”
The arrows represent documented associations, not mathematical operations. The report should identify the output or run; the run record should identify the materials used; and the repository should make those materials identifiable. For a review focused on traceability, “identifiable” does not mean that every input must be stored in the repository. Sensitive or controlled data may be referenced through an approved identifier, version, or checksum, provided that the reference is precise enough for an authorized reviewer to distinguish the intended input from other versions.
The review assumes that the repository and its recorded run information are the evidence being examined. It does not assume that labels such as “final,” “production,” or “approved” are accurate just because they appear in a filename or commit message. Nor should a reviewer assume that a Git commit identifies every file present during a run: files not committed to that revision, including untracked files, are not represented by the commit snapshot.
A Reviewer Checklist
Apply the checklist to one reported result or one clearly defined set of results. Record what evidence supports each link and what remains uncertain. A concise review is more useful when it distinguishes “present and consistent” from “not found” and “conflicting.”
| Review area | Evidence to inspect | Questions to ask |
|---|---|---|
| Reported result and output | Report table, figure, or summary; associated output file | Is the reported quantity and model or run label specific? Can the cited output be located, and does it contain the stated result? |
| Run identity | Run identifier, timestamp, command or entry point, status, output location | Can this record be distinguished from other runs? Does it identify the output under review? |
| Code revision | Commit identifier and relevant tracked files or paths | Is a particular revision identified? Does the record explain whether relevant code changes occurred after that revision? |
| Configuration | Configuration file, resolved settings, overrides, and run-specific values | Can the values used for this run be distinguished from defaults or later edits? |
| Inputs and environment | Input dataset identifier or checksum; software and package versions | Are the intended data version and execution environment named precisely enough to distinguish them? |
| History and documentation | Commit history, decision or change records, README, report notes | Do these explain relevant changes and connect the reported result to the recorded run, rather than only describing the project generally? |
For each row, note the evidence location and its identifier. A file path alone may not identify a stable version if the file can be edited in place. A commit identifier may identify a code snapshot but does not identify a run’s settings or input data unless the run record links them. A timestamp can help distinguish runs, but timestamps alone are weak identifiers if clocks, time zones, or repeated runs make the records ambiguous.
Write down the reported quantity, model or analysis label, units where applicable, and the report location. Avoid beginning with whichever output file looks most recent.
Find the table, figure, parameter summary, or other output cited for that claim. Check that the reported value and label agree with the artifact.
Determine whether a run identifier connects that artifact to a recorded invocation and gives a specific output location or filename.
Check the cited code revision, configuration values, input reference, and software environment. Note omissions, ambiguity, or conflicts explicitly.
Confirm that identifiers agree across report, output, run record, and documentation. State what the evidence supports and what it cannot establish.
Worked Reviews
Worked Example: A Result with a Connected Run Record
Hypothetical repository evidence. A draft report gives a final-model estimate of apparent clearance as 4.80 L/h. The report cites output file run-2026-04-17-03/parameter_summary.csv. That directory contains a run record identifying run 2026-04-17-03, the output path, commit 8f31c2a, configuration file config/final.yaml and its recorded checksum, input dataset version analysis-data-v12, and software environment file environment.lock. The summary reports apparent clearance as 4.80 L/h. A project note says that this run was used for the draft report.
Apply the review. First, the claim is specific: the reported quantity is apparent clearance, its value is 4.80, and its unit is L/h. Second, the cited output exists in the named run directory and contains the same quantity and value. Third, the run record links that output to one run identifier. Fourth, the record names a particular code revision, configuration version, data version, and environment file. Finally, the report note links the run to the report.
Conclusion. The available evidence supports a traceable association between the reported clearance estimate and the identified run and project materials. This conclusion does not establish that the estimate is statistically valid, that all necessary files were captured, or that rerunning the analysis would produce the same output. Those require other forms of review.
Worked Example: A Commit Message Is Not a Run Link
Hypothetical repository evidence. A report gives an estimated volume of distribution of 32.1 L. The repository contains a file named final_results.csv with that value and a commit titled “final model results.” The file has no run identifier or recorded output path. The commit history shows changes to a model script, but no run record identifies the commit used to generate the CSV. Two configuration files, config/model.yaml and config/model-final.yaml, are present, and neither is identified in the report.
Apply the review. The claim and a plausible output are present, but the critical link from output to run is missing. The commit title indicates a human description of a change; it does not establish which script or configuration generated the output. The presence of two possible configurations adds ambiguity rather than resolving it. A reviewer cannot select one based on the word “final.”
Conclusion. The result is not adequately traceable from the available evidence. The appropriate finding is not that the estimate is wrong, nor that the latest commit must have generated it. The finding is that the repository does not connect the reported value to a specific run and configuration. A run record or other contemporaneous evidence that identifies those links would be needed to resolve the gap.
Worked Example: Conflicting Identifiers in Documentation
Hypothetical repository evidence. A report states that its final parameter table comes from run R118. The run record for R118 identifies commit c71a09d, dataset PK-clean-v8, and output file R118/final_parameters.csv. The report appendix instead says the table was generated from dataset PK-clean-v7. The output file’s header records the value PK-clean-v8, and both dataset versions are listed in the project documentation.
Apply the review. The report and run record agree on the run identifier, and the output header agrees with the run record on the dataset version. However, the report appendix names a different version. This is not a harmless lack of detail: the documentation contains two incompatible input references. The reviewer can identify the conflict but cannot infer whether the appendix is stale or the run record is wrong without further evidence.
Conclusion. The traceability chain is partially present but inconsistent. Record the disagreement, identify the specific locations, and request clarification or correction supported by the project record. Do not silently choose the version that seems more plausible. Until the discrepancy is resolved, the report’s data lineage is uncertain.
Why the Chain Must Be Complete
The need for a connected chain follows from a simple identification problem. Suppose a report cites an output file but the repository contains several runs with that filename. The report alone does not identify which run generated the cited file. Suppose a run record identifies a commit but not the configuration, while the configuration can change independently of the code. Then the commit alone does not identify the settings used. In either case, multiple candidate project states remain compatible with the evidence.
This is a logical justification, not a claim that a complete record makes the result correct. A connected chain supports attribution: it lets a reviewer say which recorded run and materials the result is associated with. Statistical assessment, data-quality assessment, and successful execution are separate questions. Even a well-documented run could contain a modeling error; even a defensible result could be poorly documented.
Diagnostics, Alternatives, and Limits
Common warning signs include a report that says only “final run,” output files overwritten in a shared directory, records that name a branch but not a fixed commit, configuration values that differ between the run record and the checked-in file, and documentation that refers to “latest” without defining when or how that state was determined. Also check for records that identify only a project-level environment file when run-specific software changes or overrides are documented elsewhere.
A Git history is useful for understanding changes and locating a code revision, but it is not a substitute for run-level provenance. Conversely, a run record can identify an execution without explaining why the model or configuration changed. Decision logs and change history provide context for decisions; commit history records versioned changes; run records associate a particular execution with its recorded inputs. These sources complement one another rather than replace one another.
Hashes can help distinguish file contents, but a hash does not explain what a file means or prove that it was used. A path can be meaningful to a project team but may point to a mutable file. A branch name can move as commits are added. Prefer stable identifiers for the actual revision or file version, and use paths and explanatory text to make those identifiers interpretable.
A traceability review also has limits. It may reveal that a report and output disagree, but it cannot always determine which one is authoritative. It may find a dataset identifier without access to the controlled data itself. It can document that an environment file is cited, without proving that the software actually used matched it. Report these as bounded findings rather than extending the review beyond its evidence.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A report cites a parameter table and names a commit, but no run record connects the table to that commit. What can the reviewer conclude?
- Why is a dataset checksum useful, and what does it fail to establish by itself?
- A run record identifies a configuration file path, but the file was edited after the run and no version or checksum was recorded. What traceability issue remains?
- What is the difference between a traceability finding and a conclusion that a model result is statistically valid?
Solutions. (1) The output and a commit are present, but the evidence does not establish that the cited commit produced the table. The result is not fully linked to a run. (2) A checksum can distinguish file contents from other versions; by itself, it does not explain the file’s meaning or prove that those contents were used in the run. (3) The path identifies a location, not necessarily the historical contents used. Without a recorded version, checksum, or equivalent evidence, the configuration state at run time is ambiguous. (4) Traceability supports attribution of a result to identified project materials and a recorded run. Statistical validity concerns whether the modeling and inference are justified; identifying the materials does not answer that question.