From Model Outputs to Report Artifacts
A diagnostic report is reliable only when each plot and table receives the right outputs at the right level of detail. A prediction-versus-observation plot needs paired values for the same record; a subject summary needs one row per subject; and a run summary needs values that describe the run as a whole. Mixing these levels can duplicate observations, hide missing records, or attach a run-wide result to every plotted point.
The earlier material on standard diagnostic report specifications establishes the purpose and intended audience of a report. Here, the practical task is to map each artifact in that specification to the fields needed to produce it. The map should say not only which variables to supply, but also what one row represents, how records are matched, and what units and labels must travel with the values.
The grain is part of the artifact definition, not an implementation detail. For example, a data set with one row per observation can support a prediction plot directly. It cannot support a subject-level table by simply relabelling its rows: repeated observations would still be repeated. That table requires an explicit subject-level summary or a clearly specified one-row-per-subject source.
Specify the Mapping Before Building the Plot
Start with the artifact’s purpose and write down its inputs. In a model diagnostic setting, common output fields include an observed response, a population prediction, an individual prediction, a residual value, a subject identifier, time, and run metadata. The name of a field alone is not enough: record what it represents, its units, and which records it applies to. In particular, distinguish population predictions from individual predictions, and state which one an artifact uses.
| Artifact | Grain and key | Required data fields | Documentation to retain |
|---|---|---|---|
| Observed versus population prediction | One row per matched observation; observation key, usually including subject and record identifiers | Observed response, population prediction | Response and prediction units; population-prediction definition; included observation types |
| Observed versus individual prediction | One row per matched observation; same observation key on both inputs | Observed response, individual prediction | Response and prediction units; individual-prediction definition; included observation types |
| Residual versus time or prediction | One row per residual record; observation key | Residual value plus time or the chosen prediction value | Residual type and units; time units; which prediction is used, if applicable |
| Residual distribution or quantile plot | One row per eligible residual record | Residual value | Residual type; inclusion and exclusion rules; any reference distribution or transformation |
| Individual fit panel | One row per observation or prediction time within subject; subject and record keys | Subject identifier, time, observed response, individual prediction; population prediction if displayed | Panel grouping; axes and units; prediction series shown |
| Subject summary table | One row per subject | Subject identifier and explicitly defined subject-level summaries | Summary definitions, units, denominators, and handling of unavailable values |
| Run summary or status table | One row per run, or one row per run and status category | Run identifier, status, and required run-level counts or metrics | Run configuration and provenance fields needed to identify and reproduce the execution |
The table is a map, not a universal schema. A particular report may add fields for stratification or omit an artifact that is not relevant. But a report should not silently substitute one prediction type for another, change the grain, or combine quantities with incompatible units. If an artifact requires a residual, its contract should identify the residual field already supplied to it and state the residual definition; selecting among residual types is a separate modeling decision.
Why Keys and Grain Need a Justification
When a plot combines observed data with predictions stored separately, the rows must be paired by keys that identify the same observation. A subject identifier alone is generally insufficient when a subject has multiple records. A robust key may include a subject identifier and record identifier, or other fields that uniquely identify the observation under the data collection and modeling conventions.
The reason can be seen from join counts. Suppose an observed-output table contains \(m\) records with a particular key and a prediction-output table contains \(n\) records with that same key. A join on that key produces \(m \times n\) paired rows. If \(m=2\) and \(n=3\), the result has \(2 \times 3=6\) rows, not the intended three or fewer observation matches. The join is valid for a one-to-one artifact only when each key occurs once in each input and the paired keys identify the same record.
This simple count explains two common checks: test key uniqueness before joining, and compare the resulting row count with the expected number of eligible records. If the source tables are already aligned in one row per observation, the same principle still applies: verify that the row identifier is unique and that all plotted fields refer to that row.
For example, specify one point per eligible observation for an observed-versus-prediction plot.
Name the observed response and the intended prediction, then document units, prediction type, and eligibility rules.
Check uniqueness at the intended grain and confirm that paired records match before plotting or summarizing.
Store labels, axis units, grouping, and the source run identifier with the artifact or its report metadata.
Worked Example: Mapping Observation Outputs to a Prediction Plot
Consider hypothetical concentration data for one subject. The intended artifact is observed concentration versus population prediction, with one point per eligible observation.
| Subject | Record | Time (h) | Observed (mg/L) | Population prediction (mg/L) |
|---|---|---|---|---|
| S01 | R1 | 1 | 8.0 | 7.0 |
| S01 | R2 | 4 | 5.0 | 4.8 |
| S01 | R3 | 8 | 2.5 | 3.0 |
The contract is one point per record, keyed by subject and record. Its required plotted fields are observed concentration and population prediction, both in mg/L. Time is not required for the two axes, but retaining it can help identify points or support another artifact. The first plotted pair is \((7.0, 8.0)\) mg/L, with prediction on the horizontal axis and observation on the vertical axis. The three matched rows yield three points; no arithmetic transformation is needed to create this artifact.
Before plotting, confirm that the record key is unique and that all three rows are eligible observations. If a separate prediction table contained three rows for S01 but no record identifier, matching on subject alone would not establish which prediction belongs to which observation. A join could multiply records, so the plot should not be generated until a valid record-level match is available.
Subject-Level Artifacts Need Explicit Grouping
Individual fit panels and subject summaries require subject identifiers, but they do not have the same grain. A fit panel typically retains multiple time-ordered records for each subject. A subject summary reduces a set of records to one or more defined values per subject. The report specification should therefore say whether the panel’s rows represent observed records or prediction times, and should state the summary rule for every subject-level table column.
Worked Example: Preparing an Individual Fit Panel
Suppose a hypothetical report displays observed and individual-predicted concentrations over time for two subjects. The input has one row per subject and observation record:
| Subject | Record | Time (h) | Observed (mg/L) | Individual prediction (mg/L) |
|---|---|---|---|---|
| S01 | R1 | 1 | 8.0 | 7.5 |
| S01 | R2 | 4 | 5.0 | 4.6 |
| S02 | R1 | 1 | 6.0 | 6.4 |
| S02 | R2 | 4 | 3.5 | 3.2 |
The panel contract uses subject as the panel grouping field, record as the within-subject identifier, time in hours for the horizontal axis, and observed concentration and individual prediction in mg/L for the vertical display. It requires four observation records, split into two subject panels. If population predictions are also drawn, that series must be supplied as an additional, clearly labelled field; it should not replace the individual-prediction series.
A subject summary would have a different input contract. For example, if it reports the number of eligible observations per subject, the hypothetical counts are two for S01 and two for S02. Its grain is one row per subject, and the count’s definition must state which records qualify. Keeping the count rule explicit prevents a missing or excluded record from being interpreted as a subject with no observations.
Run-Level Artifacts and Completeness
A run-level artifact reports execution status or other quantities that describe a whole model run, rather than an observation. Its required fields can include a run identifier, a status, counts of attempted, completed, failed, or invalid records, and identifying metadata such as model version or configuration. If multiple runs appear in one report, each summary must be associated with the run it describes. Do not copy a run-wide count onto every observation row and later treat those repeated values as independent record-level data.
Worked Example: Documenting Run Counts
A hypothetical simulation run attempts 1,000 replicates. Of these, 992 complete; eight fail. Among the 992 completed replicates, two have nonfinite outputs, leaving 990 valid replicates for a downstream summary.
The run summary should state the run identifier and report the counts with their denominators: 1,000 attempted, 992 completed, eight failed, and two nonfinite among completed replicates. The total number excluded from the valid-output summary is \(8+2=10\), so the valid fraction is \(990/1{,}000=0.990\), or 99.0% of attempted replicates. The failed fraction is \(8/1{,}000=0.008\), or 0.800%; the nonfinite fraction among completed replicates is \(2/992\approx0.00202\), or about 0.202%.
Those percentages answer different questions because their denominators differ. A field contract should identify each denominator and whether a count refers to attempted or completed replicates. Reporting only “10 invalid” would conceal the distinction between failed execution and completed execution with a nonfinite result.
Diagnostics, Alternatives, and Limitations
Field mapping has practical checks of its own. Before generating an artifact, check that required fields exist, units agree, keys are unique at the declared grain, and records satisfy the stated inclusion rules. Check for missing values in the fields needed by that artifact, and report how many records are omitted if omission is permitted. A plot that silently drops unmatched or missing records can look complete while representing only a subset of the intended data.
- Prediction mismatch: A plot requests population predictions but receives individual predictions. Correct the mapping and label the prediction source explicitly.
- Repeated or missing records: A join key is not unique or does not match across sources. Inspect key counts and unmatched records before creating the plot.
- Wrong grain: Observation-level rows are used as if they were one row per subject. Define a subject-level summary or retain the appropriate subject-by-record grain.
- Ambiguous residuals: A field is labelled only “residual.” Document which residual quantity it contains, its units, and its eligibility rules before using it in a residual artifact.
- Unclear denominators: Run-level rates are reported without saying whether the denominator is attempted, completed, or valid outputs. Include the denominator alongside the rate.
One implementation approach is to prepare a separate, artifact-specific input table for every plot or summary. This makes each table’s grain and required fields easy to inspect, but may repeat data preparation. Another approach is to draw several artifacts from a shared observation-level table. This can reduce repeated processing, but only if every artifact’s field contract remains explicit and subject- or run-level summaries are not confused with observation-level rows. Neither approach removes the need to validate keys, definitions, and units.
The mapping also has limits. It does not determine which diagnostic is scientifically appropriate, choose a residual type, or establish whether a pattern indicates model inadequacy. It ensures that the intended artifact is built from the intended, well-described outputs. Interpretation of the resulting diagnostic is a separate task.
Check Your Understanding
For each question, identify the mapping decision that protects the artifact from ambiguity.
- A plot needs one point per observation, but the observed and prediction tables each have repeated rows for a subject. What must be checked before joining?
- An individual fit panel displays time, observed response, and individual prediction. Name its grain and the essential grouping and plotting fields.
- A run reports 12 failures among 600 attempted replicates. What failure rate should be reported, and what denominator must be stated?
- A residual-versus-time plot uses a column named “residual.” What additional documentation is needed before the field contract is complete?
- Why is a one-row-per-observation table not automatically a valid subject-summary table?
Solutions. (1) Check that the join keys identify the same observation and are unique in each input; otherwise the join can multiply rows. (2) Its grain is one row per subject and observation record or prediction time; group by subject and supply time, observed response, and individual prediction, with units. (3) The rate is \(12/600=0.0200\), or 2.00% of attempted replicates; state that the denominator is attempted replicates. (4) State the residual definition or type, its units, the source record grain, and any eligibility or exclusion rules. (5) Subjects can contribute multiple observation rows, so a subject summary needs a defined aggregation and one output row per subject.