Why Audit Outputs Before Interpreting Them?
A parameter estimate can be numerically plausible and still be the wrong result to interpret. Its table may come from an earlier fit, while a diagnostic plot comes from the current fit; the table may use one parameterization and the plot another; or both may be based on a dataset different from the intended analysis dataset. These are not minor presentation problems. They break the connection between the scientific question and the results being interpreted.
An output audit checks that each item belongs to the intended analysis run and means what its label says it means. It is distinct from deciding whether the model is adequate or whether a parameter is scientifically important. Those evaluations presume that the output has first been identified correctly.
The audit relies on traceable run information: for example, a run identifier, data and code versions, model configuration, and output-generation records. The precise records depend on the software workflow, but the central question is always the same: can this output be tied to the exact analysis object it claims to describe? An output filename or a familiar appearance alone is weak evidence.
What Must Agree Across Outputs?
In an NLME model, the population parameter vector is commonly denoted by \(\theta\), subject-level random effects by \(\eta_i\), and residual errors by \(\epsilon_{ij}\). A model prediction for subject \(i\) at observation \(j\) can be represented generally as \(f(t_{ij},\theta,\eta_i)\), with the precise function and error structure depending on the model. An audit does not require re-deriving that model. It does require checking that outputs refer to the same definition of its parameters and the same observations.
For a parameter table, check names, definitions, transformations, and units. A reported value might be a population typical value, a variance, a standard deviation, a correlation, or a transformed quantity. Those are not interchangeable. For plots, check the run identifier, the data or prediction layer shown, axes and units, and whether the plotted observations are the intended ones. For diagnostics, check that the underlying records and predictions come from the same run and dataset as the results they are meant to assess.
The intended run and dataset must be specified before checking outputs. Here, the intended analysis dataset means the exact dataset selected for the analysis, including its documented inclusion and exclusion rules. This is not a claim that those rules are scientifically correct; it is a check that the reported outputs use the dataset that was actually intended.
Record the run identity and the data, code, and configuration versions that define it. Do not infer identity from a chart title alone.
Compare dataset identity, subjects, records, and relevant inclusion or exclusion information against the intended dataset.
Match names and values to the intended parameterization, then use unit and numerical checks to catch implausible transformations or mismatches.
Confirm that tables, plots, and diagnostics refer to the same run and compatible data, parameters, and units. Resolve discrepancies before interpretation.
Worked Checks
Worked Example: A Current Plot and a Stale Parameter Table
Suppose an analyst intends to interpret run R24, fitted to dataset D7. The run record gives a typical clearance estimate of 5.2 L/h. A parameter table in the draft folder lists clearance as 4.8 L/h, and its file metadata identifies run R23. A concentration-time plot is labelled R24. Is the table-and-plot combination ready to interpret?
Check the run identities. The plot identifies R24, whereas the table identifies R23. The artifacts do not yet form a consistent set, even if both look plausible.
Check the parameter values. The discrepancy is \(5.2-4.8=0.4\) L/h. Relative to the R24 estimate, the difference is \(0.4/5.2=0.07692\), or about 7.7%. This calculation describes the discrepancy; it does not establish which estimate is appropriate. The run record identifies 5.2 L/h as the R24 value, so the 4.8 L/h table must not be presented as an R24 result.
Conclusion. The table is stale for the intended run. It should be regenerated from R24 or removed from the set being interpreted. The plot itself still needs its own checks, including confirmation that its predictions and data layers use R24 and D7. Matching one output does not validate every other output.
Worked Example: Parameterization and Time-Unit Mismatch
Consider a hypothetical one-compartment IV model with concentration prediction
Suppose the intended table reports \(CL=4\) L/h and \(V=40\) L, and the plot uses time in hours. The implied elimination rate constant is
The half-life implied by this model is \(t_{1/2}=\ln(2)/k\), because half the initial concentration remains when \(\exp(-kt)=1/2\). Thus
Now suppose a curve-generation script mistakenly interprets the numerical clearance value 4 as L/day, while still using \(V=40\) L and plotting time in hours. Then \(CL=4/24=0.1667\) L/h, so
The numerical clearance value in both places may appear as “4,” but the implied half-lives differ by a factor of 24. The calculation assumes that the model is one-compartment IV elimination with the stated units and that the displayed time is in hours. It is a consistency check, not proof that the model is scientifically adequate. A table in L/h paired with a curve computed using L/day is a parameterization-and-units mismatch; it must be corrected before comparing the curve with the table or interpreting either together.
Worked Example: Output Based on the Wrong Analysis Dataset
The intended dataset D7 contains 120 subjects and 840 concentration records. A prediction-versus-observation plot labelled “D7” contains 118 distinct subject identifiers and 824 points. A saved exclusion list documents that two subjects, with four records each, were excluded in another analysis. Do the plot counts agree with the intended dataset?
The expected counts after those exclusions would be \(120-2=118\) subjects and \(840-(4+4)=832\) records. The plot has 118 subjects, which is consistent with that exclusion, but it has 824 points: \(832-824=8\) records are unaccounted for. Relative to the intended dataset, the plot contains \(840-824=16\) fewer records, not the 8 records explained by the two subject exclusions.
Therefore, the plot does not match D7 as described, and the counts do not fully match the documented alternate exclusion either. Possible explanations include additional filtering, a different dataset version, or an error in the plot-generation step. These are hypotheses to investigate, not conclusions established by the counts. The analyst should compare subject and record identifiers against D7, trace any filters, and regenerate or relabel the plot only after resolving the difference. The audit cannot decide which records ought to be included; it detects that the plotted data do not yet have a verified connection to the intended dataset.
A Consistency Justification
Why check units and simple implications rather than trusting labels? A model output is produced by a chain: selected data and model inputs determine fitted quantities, which are then used to construct tables, predictions, or diagnostics. If a downstream artifact uses a different input or interpretation, it is not guaranteed to represent the same analysis. Dimensional checks provide a small but direct test of that chain.
For the one-compartment example, concentration in dose-per-volume units requires \(\mathrm{Dose}/V\) to have concentration units. The exponent must be dimensionless. Since \(CL/V\) has units \((\mathrm{volume}/\mathrm{time})/\mathrm{volume}=1/\mathrm{time}\), multiplying it by time gives a dimensionless quantity. Using clearance in L/day with time in hours without conversion violates this consistency: the numerical exponent no longer represents the intended rate over the displayed time scale. This explains why unit conversion is not cosmetic.
More generally, a table, plot, and diagnostic that claim to describe the same run should agree on run identity, parameter meaning, and data basis. This is a necessary consistency condition, not a sufficient condition for a good model or valid scientific conclusion. Passing an audit does not establish model adequacy, convergence, or suitability for a particular use.
Diagnostics, Alternatives, and Limits
Useful audit signals include conflicting run identifiers, inconsistent subject or observation counts, unexpected units, parameter values that cannot be reproduced from the stated transformation, and plots whose axes or labels conflict with the analysis configuration. Automated checks can compare identifiers, counts, hashes, and metadata across files. Independent review can catch ambiguities in parameter meanings or plotting choices that a file comparison cannot.
The strongest practical approach combines these checks rather than relying on one alternative:
| Approach | What it can establish | Important limitation |
|---|---|---|
| Visual review of labels and plots | Can reveal obvious mislabelling, unexpected scales, or unusual counts. | A plausible appearance does not establish run or dataset identity. |
| Automated metadata and identifier checks | Can systematically compare run records, versions, counts, and identifiers. | Incorrect or incomplete metadata can make an automated match misleading. |
| Independent regeneration from the identified run | Can test whether a table or plot is reproducible from the intended inputs. | Requires a working, correctly configured generation process and clear definitions. |
A mismatch should be treated as unresolved until its cause is established. Do not select whichever output looks more plausible, average values across outputs, or silently relabel a file. If a source record is missing or an artifact cannot be reproduced, record that limitation and withhold interpretation of the unverified output. An audit can confirm identity and consistency only to the extent that the underlying records are reliable and complete.
Common mistake: “The estimate is close to the expected value and the plot looks reasonable, so the table must belong to the current run.” Plausibility is not provenance. A stale output can be plausible precisely because it came from a nearby candidate run. The correction is to verify its run identity and inputs, then cross-check its content, not to infer its origin from its appearance.
Check Your Understanding
Answer each question using the audit principles above. Short solutions follow the questions.
- A table and a diagnostic plot have the same model name but different run identifiers. What should be checked before interpreting them together?
- A clearance estimate is reported as 12 L/day, while a plot-generation input uses the number 12 as L/h. What conversion is needed, and why does the numerical match not suffice?
- The intended dataset has 75 subjects and 500 records. A plot has 75 subjects and 492 points. Does the subject count establish that the plot used the intended records?
- Why is an output audit necessary but insufficient to establish that a model is adequate?
Solutions. (1) Verify the run identities and the data, code, and configuration versions for each artifact. The matching model name does not show that they came from the same run. (2) Convert \(12\) L/day to \(12/24=0.5\) L/h before using an hourly time scale. The same numerical value has different rate units and therefore implies a different rate in the model. (3) No. The plot is short by \(500-492=8\) records, so record identifiers and any documented filters must be checked even though the subject count matches. (4) An audit checks that outputs correspond to the stated run, parameterization, units, and dataset. It does not by itself test whether the model assumptions or structure are scientifically adequate.