What an Evidence Package Is For
An independent review depends not only on what an analysis concluded, but also on whether the reviewer can inspect the materials needed to assess it. A folder of unlabeled files may contain the right items and still leave basic questions unanswered: Which dataset was used? Which run produced these outputs? Where is the model specification? An evidence package addresses those practical questions by organizing the materials relevant to a defined review.
The package should be designed around the review scope established for the analysis. It is not a general archive of every file ever created, nor is it a substitute for the review itself. Its purpose is to provide a clear, usable set of evidence: source files, run records, model outputs, and explanatory materials, accompanied by enough organization to show what each item is and why it is included.
The package can support assessment, but its existence does not establish that the analysis is correct, reproducible, or fit for purpose. Those are conclusions of review, not properties conferred by a folder structure or a checklist. The scope of review and the intended recipient also affect what belongs in the package. For example, a reviewer assessing only a report may need a different set of supporting materials from a reviewer assessing the model implementation as well.
Start With Scope, Then Select Evidence
Before collecting files, translate the agreed review scope into evidence needs. For each review question, ask what the reviewer would need to inspect to assess it. A review of a reported result may require the submitted report, the output supporting that result, and the run record identifying the submitted execution. A review that includes model implementation also needs the relevant model source and written model specification. Include materials that answer a review question; avoid adding unrelated drafts merely because they are available.
A useful package has four connected layers. Source files are the analysis materials being reviewed, such as the model code, analysis dataset, configuration files, and written specification, as relevant to scope. Run records identify the submitted execution and its status, with references to the principal inputs and outputs. Model outputs are the results and diagnostics the reviewer is expected to assess. Explanatory materials give context: for example, a short readme, a data dictionary, a report, or a note defining file organization and known limitations.
This organization follows a simple dependency: the reviewer needs to know what analysis materials were submitted, which execution record identifies the run under review, what outputs are associated with that run, and how to interpret the materials. If any link is unclear, an output may be difficult to evaluate even when the file itself is present. An index and concise explanatory notes make those links explicit without requiring the reviewer to infer them from filenames.
Choose and Describe the Package Contents
The exact contents vary by scope, but the following questions help determine what to include:
- Source files: Are the model source and the analysis data or data reference included where the scope requires them? Are relevant specifications and configuration files present? Are drafts clearly distinguished from submitted files?
- Run records: Is there a record identifying the submitted run, its date or identifier, its execution status, and the principal input and output references? Can the reviewer tell which run is under review?
- Model outputs: Are the reported results, tables, and diagnostics needed for the review included? Are the files tied to the run record by clear names or references?
- Explanatory materials: Does the package explain its directory structure, abbreviations, units where needed, and any known omissions or access restrictions?
- Package index: Can a reviewer use one list to find each item, understand its role, and see whether it is required for the stated scope?
A package index is a practical navigation aid, not a conclusion about the evidence. It can list a stable filename or relative path, a brief description, the category of material, and its relevance to the review. When a required item is intentionally absent, state that explicitly and give the reason or the location from which it can be accessed under the agreed arrangements. Silent omissions are difficult to distinguish from accidental ones.
Worked Example: Selecting Evidence for a Scoped Review
A hypothetical review is limited to assessing the submitted model implementation and the results presented in a report. The handoff folder contains a model source file, an analysis dataset, a written model specification, a run log, a results table, a diagnostic report, three older draft reports, and an unrelated slide deck.
Apply the scope. The model source and specification support assessment of the implementation. The analysis dataset is relevant because it is an input to the analysis under review. The run log identifies the execution. The results table and diagnostic report support assessment of the reported results and model diagnostics.
Exclude or label unrelated material. The three older reports and slide deck do not answer the stated review questions. They should not be mixed into the main evidence set. If drafts are needed to explain a decision, identify them as contextual material and state why they are included; otherwise leave them out.
Make the selection inspectable. The package index lists the six relevant items, gives each a short description, and identifies which review purpose it supports. The handoff is now focused and navigable. This selection alone does not determine whether the implementation or results are correct; it equips the reviewer to assess them.
Organize for Navigation and Interpretation
A shallow, predictable directory structure is generally easier to review than a deeply nested one. For example, a package might use separate folders for source materials, run records, outputs, and explanatory materials, with a readme and index at the top level. The names should distinguish submitted items from drafts and should remain understandable outside the analyst's working environment. Avoid relying on private abbreviations or filenames such as “final2_revised.”
The run record should provide enough information to identify the execution being submitted: a run identifier, date or time, completion status, and clear references to relevant inputs and outputs. It may also state the command or options used when these are part of the review handoff. Keep the record focused on identification and navigation; detailed examination of the project’s history or the software environment is a separate assessment task.
Outputs should be labeled so the reviewer can tell what each contains and how it relates to the run record. A report can summarize outputs, but it should not silently replace important underlying tables or diagnostics that are within scope. Conversely, dumping every generated file into the package can make relevant evidence harder to find. Select outputs according to scope, then explain any selection or omission that could affect interpretation.
Explanatory materials should resolve likely questions without directing the reviewer toward a desired conclusion. A readme can describe the folder structure, the purpose of files, the meaning of abbreviations, and known limitations. It should not claim that an analysis has passed review simply because its files are present. Use concise, factual descriptions and clearly mark any assumptions or access conditions relevant to the handoff.
Worked Example: Checking an Index Against the Handoff
A hypothetical package index lists four required items: model_source, run_record, results_table, and diagnostics_report. The submitted folder contains all four, plus a file named results_table_old. The run record refers to results_table and diagnostics_report.
Check presence and references. The four listed items are present, and the run record points to the two named output files. The older table is not listed or identified as a draft. Its presence creates ambiguity: a reviewer could mistake it for another submitted result.
Resolve the ambiguity. If the older table is not relevant, remove it from the review package. If it is included as context, add it to the index with a clear draft label and explain that it is not the submitted output. The corrected package then distinguishes the intended evidence from incidental files. This check establishes navigational consistency, not the validity of the table’s contents.
Completeness Checks, Alternatives, and Limitations
A final package check should test whether the handoff is coherent from the reviewer’s perspective. Confirm that each item required by scope is present or has an explicit explanation; that the index agrees with the folder contents; that the run record points to the intended outputs; and that the explanatory materials use terms and file names consistently. Also confirm that the intended reviewer has appropriate access to the files. These are checks of package usability, not a technical verification of the analysis.
There are several ways to deliver evidence. A single PDF is easy to circulate and can be appropriate for a report-only review, but it may hide or omit source files and underlying outputs needed for a broader assessment. A shared folder is convenient for collaboration, but contents can change or become difficult to distinguish if versions and submitted status are not clear. A fixed archive can provide a stable handoff, but it still needs an index and understandable organization. Choose the delivery form that fits scope and access needs; do not treat any one format as inherently complete.
The package also has limits. It cannot make missing evidence available, explain undocumented analytical decisions, or establish that a file is the correct one merely by labeling it “submitted.” Nor can a polished report substitute for source materials when the review scope requires their assessment. When evidence is unavailable, restricted, or outside scope, record that fact so the reviewer can judge its implications rather than infer that the item was checked.
List the review questions and identify the source files, run records, outputs, and explanations needed to assess them.
Separate submitted evidence from drafts or unrelated working files, and explain any necessary exclusions.
Describe each item and ensure the run record, outputs, and package index refer to the intended files consistently.
Check access, navigation, labels, and stated omissions. Correct ambiguity before submission.
Common Pitfall: Treating Presence as Adequacy
A common mistake is to reason, “All expected file types are in the folder, so the evidence package is complete.” The correction is to test whether the files are the ones required by the agreed scope, whether their roles are clear, and whether their references agree. A report, a model file, and a run log can all be present while the run log points to a different output than the report presents. Package assembly should expose such ambiguity, not certify the underlying analysis.
Another mistake is to over-package: sending every intermediate and draft file in the hope that more material must help. Unlabeled excess can obscure the submitted evidence and increase the chance of reviewing the wrong item. A concise package with explicit references and candid notes about gaps is usually more useful than a large, unexplained directory.
Worked Example: Documenting a Known Omission
A hypothetical review requires a model output table and a diagnostic report. The table is available, but the diagnostic report was not generated for the submitted run. The analyst should not list a report from a different run as if it were the missing item.
The index can state that the diagnostic report required for the planned review is not included, identify the reason if known, and note that the available table is the submitted output. The reviewer can then decide whether the omission limits the review or whether additional material should be requested. This is more informative than leaving the report unmentioned or implying that the package is complete.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A review scope includes model implementation and reported results. Name one appropriate item from each of the four evidence categories.
- Why is an index useful if every file is already in the handoff folder?
- A run record points to one results table, but the report appears to contain a different table. What should the package assembler do before submission?
- A diagnostic report is missing, and a report from an earlier run is available. How should the omission be handled?
- Does a complete evidence package establish that the model is correctly implemented? Explain.
Solutions. (1) Examples include model source (source file), a record identifying the submitted execution (run record), a results table or diagnostic report (model output), and a readme or written specification (explanatory material). The specific items depend on the defined scope. (2) The index gives each file a clear role and helps the reviewer locate relevant evidence without guessing from filenames; it can also make omissions explicit. (3) Resolve the mismatch by checking which output is intended for submission, then align the run record, report references, and index. Do not leave conflicting references for the reviewer to interpret. (4) State clearly that the required report is missing, and do not present the earlier-run report as if it came from the submitted run. Explain the gap if its reason is known. (5) No. Completeness and navigability help the reviewer assess the analysis, but correctness requires substantive review of the evidence.