Defining a Standard Diagnostic Report Specification

Why a Diagnostic Report Needs a Specification

A diagnostic report is more than a collection of plots and tables. Someone reviewing a model needs to know which outputs are required, what each output represents, which run or scenario it belongs to, and what file formats must be delivered. If these decisions are left implicit, two analysts can produce different reports from the same analysis and both believe they have met the requirement.

A report specification makes those expectations explicit before files are generated. It is a contract for the intended deliverables, not a guarantee that a plot is scientifically adequate or that a table contains correct values. This distinction matters: a file can satisfy a presence-and-format check while still being misleading or wrong.

Definition: An artifact type describes a kind of report output, such as a fit overview figure or a parameter summary table. A required artifact instance is one required occurrence of that type for a defined scope and format. A delivery record describes one actual delivered file and must have a unique file identity, even when another file has the same name or logical purpose.

Specify the Deliverables Before Generating Them

A useful specification starts with an inventory. For each artifact type, state its purpose, the conditions under which it is required, the scope to which it applies, required formats, and its expected location or naming convention. A scope might identify a model version, analysis run, dose scenario, study, or subgroup. Include only scope dimensions that are needed to distinguish deliverables; an unnecessary dimension makes the inventory harder to review.

For example, “one fit overview per run” is more precise than “include fit plots.” “One scenario summary table per planned dose scenario, as CSV” tells the producer and reviewer how many outputs to expect and how to distinguish them. If a figure is required in both PDF and PNG, specify both. Those are two required file representations, not one file that can silently be delivered in either format.

Specification field	What it establishes
Artifact identifier	A stable, human-readable label for the required output, such as fit_overview.
Purpose and content	What the output is intended to show or summarize, stated without relying on an ambiguous filename.
Scope	Which run, scenario, study, subgroup, or other context the instance belongs to.
Condition	Whether the artifact is always required or is required only when a stated, testable condition holds.
Format	The required file representation, such as PDF, PNG, CSV, or JSON, plus relevant technical requirements.
Organization	The expected directory, file-naming pattern, and ordering or grouping in the report package.
Review status	Whether the item is mandatory, conditionally mandatory, or explicitly optional.

Identifiers should be stable across runs and informative enough to distinguish artifact types. Scope values and format complete the logical identity of a required output. A short, consistent naming pattern might use an artifact identifier, run or scenario label, and extension. Filenames are useful for people, but the specification should not rely on filenames alone to establish identity: paths can change, names can be reused, and two copies may share a name.

Organization rules should also be concrete. State whether outputs are grouped by run, scenario, or artifact type; whether a top-level report document is required; and where any supporting files belong. If a report has a fixed reading order, define it. Do not leave a reviewer to infer whether a missing file is absent or merely stored in an unexpected folder.

Worked Example: Counting a Required Report Package

A hypothetical analysis has two planned runs, R1 and R2, and three planned dose scenarios, A, B, and C. Its specification requires one fit overview figure in PDF for each run, one scenario summary table in CSV for each scenario, and one run-information file in JSON for each run. How many required files should the package contain?

First enumerate the instances by artifact type. The fit overview contributes \(2\) run-specific instances. The scenario summary contributes \(3\) scenario-specific instances. The run-information file contributes \(2\) run-specific instances. Since each instance has exactly one required format, the total is

N_{\mathrm{required}} = (2 \times 1) + (3 \times 1) + (2 \times 1) = 7.

Thus the package requires seven files: two PDFs, three CSVs, and two JSON files. The count does not depend on whether the files are placed in separate run folders or in one directory, provided the specified organization makes each scope unambiguous.

Formalizing Required Instances and Delivered Files

Let a logical requirement be identified by an artifact identifier, its scope, and a required format. Write a logical key as \(k=(a,s,f)\), where \(a\) is the artifact identifier, \(s\) is the scope, and \(f\) is the required format. The specification defines a collection of required keys. A conditionally required item is included in that collection when its stated condition is met; an optional item is not counted as mandatory.

The delivered package is not just a set of logical keys. It is a collection of file records, each with a unique file identity and a logical key. Two different records can have the same logical key. That happens, for example, when a file is copied twice or generated twice under different paths. Keeping the file records distinct preserves the information needed to flag duplicates.

Audit rule: For each required logical key \(k\), count the delivered file records whose logical key equals \(k\). A count of zero means the required output is missing; a count of one meets the presence requirement; a count greater than one indicates duplicate delivery unless multiple copies were explicitly required. A delivered record whose logical key is not required is unexpected and should be reviewed.

This count-based rule is a short but important justification for using records or a multiset rather than comparing ordinary sets. A set retains whether a key is present, but discards how many times it occurs. If two files both have key \(k\), their set representation still contains only \(k\); therefore, it cannot reveal that the second file is a duplicate. Counting records preserves multiplicity and supports separate checks for missing, duplicate, and unexpected deliveries.

c(k)=\#\{\text{delivered file records with logical key }k\}.

The expected count for a mandatory key is usually one. The audit classifies it as missing if \(c(k)=0\), present once if \(c(k)=1\), and duplicated if \(c(k)>1\). A separate comparison identifies delivered keys that do not appear among the required keys. If a specification intentionally requires multiple files with the same logical key, that multiplicity must itself be stated; otherwise, multiple records are not evidence of extra completeness.

Worked Example: Making Format and Scope Part of the Requirement

A hypothetical report covers two model runs, R1 and R2. A fit overview is required in both PDF and PNG for each run. A scenario summary is required as CSV for scenarios A, B, and C. Calculate the total number of required file instances and explain why the figure formats count separately.

For each run, the fit overview has two required representations. Across two runs, that is \(2 \times 2=4\) figure files. The scenario summaries contribute \(3 \times 1=3\) CSV files. Therefore,

N_{\mathrm{required}} = (2\text{ runs}\times 2\text{ formats})+ (3\text{ scenarios}\times 1\text{ format})=4+3=7.

The seven keys are: fit overview for R1 in PDF and PNG; fit overview for R2 in PDF and PNG; and scenario summary for A, B, and C in CSV. A PDF for R1 does not satisfy the PNG requirement for R1, because the format is part of the logical key. Likewise, an overview for R1 cannot satisfy the requirement for R2. Making these dimensions explicit prevents a vague “figure delivered” check from concealing a missing representation or scope.

Worked Audit of a Delivered Package

An audit compares the required inventory with actual file records. The specification should define which scope values and conditions apply to the package being checked. The person conducting the audit then checks each record’s logical identity and file properties, without treating two files as one merely because their names or contents appear similar.

Worked Example: Finding a Missing File, a Duplicate, and an Unexpected Delivery

A hypothetical package requires three keys: (fit overview, run R17, PNG), (scenario summary, scenario A, CSV), and (run information, run R17, JSON). The delivery contains four records: file F101 for the fit overview key; file F102 for the same fit overview key; file F103 for the scenario summary key; and file F104 for (fit overview, run R16, PNG). Each file has a distinct file identity. Classify the delivery.

For the fit overview key for R17, \(c(k)=2\), because both F101 and F102 have that key. It is a duplicate delivery: one file is needed, and one record is additional. For the scenario summary key for A, \(c(k)=1\), so that required item is present once. For the run-information key for R17, \(c(k)=0\), so it is missing. F104 has a run R16 key that is not among the requirements, so it is unexpected and must be reviewed.

The required inventory contains three logical keys, and two of those keys have at least one matching delivery. The package therefore has \(2/3\) required keys present, but it is not complete: one key is missing. The four delivered records should not be reduced to a set of three logical keys, because doing so would hide the duplicate F102. The extra R16 file is a separate unexpected item, not a substitute for the missing R17 run-information file.

Conditions, Diagnostics, and Limits

A specification is only auditable when its conditions can be resolved. “Include subgroup plots when useful” is subjective and cannot determine a required count. A better condition names the decision rule and the scope values to which it applies. If a planned scenario fails or is excluded, the specification should say whether its outputs remain required, are marked not applicable, or require a documented exception. Otherwise, a failed analysis may be confused with an overlooked deliverable.

Common checks include whether every required key has a delivery record, whether required formats are present, whether any logical key appears more often than permitted, and whether any delivered key is unexpected. Technical requirements can include image dimensions or resolution, character encoding, or whether a table must be machine-readable. State these only when they matter to downstream review or use. A file extension by itself does not prove that the file has the required internal format.

The specification also has limits. It can establish that the expected kinds of files exist under the expected scopes and formats; it cannot establish that plotted values are correct, that a model is appropriate, or that metadata claims are true. Those require separate scientific review and validation. A report should not be called validated merely because its inventory check passes.

Common Mistakes and Better Practice

Wrong reasoning: “The folder contains a figure with the right name, so the figure requirement is met.” The file may belong to the wrong run or format. Check the complete logical key, including scope and required format.
Wrong reasoning: “There is one matching key in the delivered set, so there cannot be a duplicate.” A set loses multiplicity. Preserve each delivered file as a uniquely identified record and count records per logical key.
Wrong reasoning: “A missing output is harmless because the analysis did not produce it.” The specification must state how failed or inapplicable scopes are handled. Record an explicit exception or status rather than silently treating the requirement as fulfilled.
Wrong reasoning: “A complete inventory proves the report is scientifically sound.” Presence and format checks do not assess scientific validity. Review the content and underlying analysis separately.

Key takeaway: Specify each required output by a stable artifact identifier, scope, and format; define its conditions and organization; and audit delivered files as uniquely identified records. Counting those records preserves duplicates as detectable events.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A report requires one table in CSV for each of four scenarios and one PDF figure for each of two runs. How many required file instances are there?
Why must format be part of a required logical key when a figure is required in both PDF and PNG?
A required key has no matching file record, while a different key has two matching records. Classify both findings.
What is wrong with using a set of filenames as the sole method for detecting duplicate deliveries?
Name one important conclusion that an inventory check cannot establish.

Solutions. (1) There are \(4+2=6\) required file instances. (2) PDF and PNG are distinct required representations; delivering one does not meet the requirement for the other. (3) The first key is missing, and the second is duplicated unless its required multiplicity is explicitly greater than one. (4) A set discards repeated elements, so two files that map to the same filename or logical key can collapse into one set entry; use uniquely identified file records and count them. (5) An inventory check cannot establish that the model or reported values are scientifically correct.