Capturing Run Metadata in Diagnostic Reports

Why diagnostic outputs need run metadata

A diagnostic plot or table can be numerically correct and still be difficult to interpret later if its origin is unclear. A filename such as residuals-final.png does not establish which model, dataset, estimation attempt, or analysis scenario produced it. In a reproducible workflow, provenance is not a guess made from the surrounding folder; it is recorded and checked as part of producing the report.

The earlier material on the standard diagnostic report specification, mapping model outputs to report artifacts, canonical diagnostic data schemas, and normalizing residual and prediction outputs establishes what report artifacts contain and how their data are represented. This tutorial focuses on a separate question: what information must travel with an artifact so that a reviewer can identify the run behind it?

Specify the metadata before generating artifacts

A useful run record contains the required fields named in the focus of this tutorial, represented consistently rather than inferred after the fact. The model identifier should identify the model definition used; where a model can change while retaining a familiar name, include a model version or source digest as well. The dataset version should identify an immutable data snapshot, such as a controlled release identifier or a checksum-backed version. A mutable label such as “latest” is not sufficient to reproduce which data were analyzed.

Record the run date as an unambiguous timestamp, preferably in UTC and a machine-readable form such as 2026-10-14T09:32:00Z. This is the time the run began, not necessarily the time a report file was rendered. If the rendering time matters, store it separately as an artifact-generation timestamp. Keeping those events distinct prevents a report regenerated later from appearing to describe a new estimation run.

Estimation status should describe the estimation attempt, not the visual quality of the report. Use a defined vocabulary, for example converged, terminated_nonconverged, failed, not_run, or unknown. The workflow must specify what each value means and retain the actual status even when estimation fails. An output created from a failed or nonconverged run must not be relabelled as successful merely because a plot rendered.

Analysis labels identify intended context that is not already conveyed by the model and dataset identifiers. Examples include an analysis name, population, study, scenario, or occasion. Store labels as explicit fields or as a structured set of label-value pairs. Use controlled, consistent values where possible: “high_dose” and “High dose” should not accidentally become distinct groups if they mean the same scenario.

In practice, the run record also needs its unique run identifier. The identifier connects the metadata record to each output and prevents two executions with the same model and dataset from being mistaken for one another. It should be stable once assigned and should not be recycled for a later execution.

Link records without creating ambiguity

A clean design separates one run-level record from the many artifacts it may produce. The run table has one row per run identifier. The artifact table has one row per artifact and includes that identifier as a reference. This avoids repeating and potentially changing run-level fields independently in every file. It also supports a validation rule: every artifact’s run identifier must match exactly one run record.

The arrow expresses a lookup, not a statistical relationship: the artifact’s run identifier selects its metadata record. If that identifier matches zero records, provenance is missing. If it matches more than one, provenance is ambiguous. A report consumer should be able to make the lookup without relying on file ordering, folder names, or assumptions about which run was “current.”

Check completeness and uniqueness

A metadata record is complete only when every required field is present and valid under the workflow’s conventions. Presence alone is not enough: an empty string for dataset version is not an identifiable version, and a date that cannot be parsed is not a usable run timestamp. Define the required fields before running validation, then test each record and artifact reference against them.

A simple field-level completeness rate is the number of valid required metadata cells divided by the total number of required cells examined. This summary can help monitor a batch, but it does not replace record-level checks. One completely missing record may be hidden by many complete records, so report the affected artifact and field as well as any aggregate rate.

A further check is uniqueness: each run identifier must occur on exactly one run-level record. This condition justifies the metadata lookup. If a key is duplicated, the same artifact could join to multiple records, multiplying rows or assigning conflicting metadata. The issue is structural, not cosmetic; a report that happens to display one of the matching records is not reliably traceable.

Why the one-to-one lookup condition matters

The metadata-linking rule follows directly from the structure of the records. Let \(a_i\) be an artifact with run identifier \(k_i\), and let \(R(k)\) be the set of run records whose identifier equals \(k\). For the artifact to have a single, unambiguous provenance record, the set selected by its key must contain exactly one record:

If \(|R(k_i)|=0\), no metadata can be retrieved. If \(|R(k_i)|>1\), more than one candidate exists. Only when the cardinality is 1 does the lookup return a unique record, so every run-level field has one defined value for that artifact. This is why both reference integrity (no unmatched artifacts) and uniqueness (no duplicated run records) are necessary; checking only that a run identifier is nonempty does not establish provenance.

Diagnostics, alternatives, and limitations

Useful workflow diagnostics include counts of artifacts with missing run identifiers, identifiers with no match or multiple matches, missing required fields, unparseable timestamps, and values outside the status vocabulary. Also compare the run-level fields across artifacts linked to the same run: they should resolve to the same record. These checks catch common problems such as metadata copied from a previous run, a changed dataset label paired with a stale checksum, and an output file written before its run record was registered.

Filenames and plot captions can make metadata visible to a human, but they are poor substitutes for a structured run record. Filenames are often shortened, manually edited, or duplicated; captions may omit fields or become detached from the underlying data. A sidecar manifest can provide structured metadata when the artifact format cannot embed it. The strongest practical approach is often to retain a machine-readable manifest keyed by artifact and run identifiers, while displaying a concise provenance summary in the report itself.

Embedding all fields in every artifact can make files self-describing, but repeated copies can drift when one copy is updated and others are not. A central manifest reduces duplication but depends on preserving the manifest and maintaining stable references to it. Either approach requires validation and durable storage. A checksum can help verify that a file has not changed, but by itself it does not establish which model or dataset produced that file. Similarly, a label such as “converged” records the workflow’s status classification; it does not prove that the model is scientifically adequate.

Metadata also cannot recover information that was never captured. If the dataset version points to a mutable location, or if analysis labels are vague, the record may be internally complete but practically insufficient. Define identifiers and allowed values in advance, preserve the referenced model and dataset versions, and treat unknown information as unknown rather than filling it by inference.

Common mistake

A common mistake is to use the report creation date as the run date and to infer the dataset version from the current input folder. The reasoning is that the report was generated from those inputs, so the surrounding context must still identify them. The correction is to record the run timestamp and immutable dataset version when the run occurs, then link later-generated artifacts to that run record. A report regenerated next week can then retain the original estimation provenance while recording its own generation time.