Why Check Simulated Outputs Against Model Intent?
A simulation can run to completion and still produce outputs that do not represent the requested design. A concentration column might be in the wrong units, a scheduled observation might be absent, or an exposure summary might use a different time interval from the one specified. These problems are easy to miss if review stops at “the program finished” or at a quick scan of a few values.
Quality-checking begins by translating the model and simulation specification into expectations that can be tested. The relevant expectations include which subjects and times should appear, which variable represents concentration, what units apply, which doses precede the observations, how missing values are represented, and exactly how each exposure measure is defined. A check can identify an inconsistency; by itself, it does not establish the cause.
Write Down the Expected Structure First
Before inspecting values, describe the output grain: what one row represents. For example, the file might contain one row per subject per scheduled concentration observation, or it might also contain separate rows for dosing events. The expected row count depends on this choice. If each of \(N\) subjects has \(K\) scheduled observation records and no additional record types, the expected number of records is \(N K\). If dose events are also stored as rows, count those separately according to the design.
A matching total count is necessary in many designs but is not sufficient. One missing subject-time record and one unintended duplicate can cancel in the total. Review the record keys, such as subject identifier and scheduled time, as well as the count. Check for unexpected subjects, unexpected times, duplicate keys, and records that cannot be assigned to a planned observation or event.
Next, compare actual and expected values for time and units. Time may be represented in hours relative to the first dose, hours relative to a later dose, or clock time; those are not interchangeable. Concentration units must also be explicit. For example, \(1\) mg/L equals \(1000\) ng/mL. A value that appears implausibly large in mg/L might be a unit-label or conversion problem, but that possibility must be verified against metadata or the producing code rather than assumed.
The model and output definition determine what range checks mean. A concentration predicted by a standard nonnegative compartment model is expected to be nonnegative, but a simulated observation that includes additive residual error can be negative unless the model applies a constraint or transformation. Exposure measures also depend on their definition: an interval, dose-normalized quantity, or extrapolation beyond the last observation changes what should be compared. Do not treat an unusual value as an error until the relevant variable and definition are confirmed.
List the planned subjects, doses, observation times, record types, and output variables. State the time origin and units.
Compare the expected subject-time or event keys with the actual keys. Report missing, duplicate, and unexpected records separately.
Confirm variable definitions and units, scan for missing or non-finite values, inspect model-appropriate ranges, and independently recalculate a small set of exposure summaries.
Record the affected subjects, times, variables, and expected-versus-observed values. Trace the source only after the discrepancy is reproducible.
Worked Checks
Worked Example: Rechecking an AUC Calculation
Suppose a hypothetical subject has simulated concentrations at 0, 1, 2, and 4 hours of 1, 7, 6, and 1 mg/L, respectively. The output reports \(\mathrm{AUC}_{0,4}=22\) mg·h/L. The stated definition is area under the concentration-time curve from 0 through 4 hours, calculated by the linear trapezoidal rule. Check the reported value.
For adjacent observations at times \(t_i\) and \(t_{i+1}\), the linear trapezoidal area is the average of the two endpoint concentrations multiplied by the time interval. Thus,
Substituting the hypothetical data gives the three segment areas:
Therefore,
The reported value exceeds this calculation by \(22-17.5=4.5\) mg·h/L. This is a discrepancy under the stated definition, not evidence by itself of why it occurred. Before investigating implementation, confirm that the output uses the same time interval, concentration units, and integration rule. If those match, the AUC output needs further review.
Worked Example: Detecting a Missing Record Hidden by the Total
A hypothetical simulation requests observations at 0, 1, 2, and 4 hours for each of five subjects. It stores only scheduled concentration observations, with one row per subject and time. The expected count is \(5 \times 4=20\) rows. The output also has 20 rows. Does the count confirm completeness?
No. Key checking shows that subject S04 has no record at 2 hours, while subject S02 has two records at 1 hour. The first discrepancy is one missing planned record; the second is one duplicate. Together they leave the total unchanged:
The correct review compares the planned set of 20 subject-time pairs with the observed set, rather than relying on the grand total. For each planned pair, check that exactly one record is present. The findings should identify S04 at 2 hours as missing and S02 at 1 hour as duplicated. The duplicate should not be silently averaged or selected arbitrarily, because doing so would conceal the record-assembly issue.
Worked Example: Checking Concentration Range and Units
A hypothetical output is specified to report model-predicted concentrations in mg/L. For a particular subject, four scheduled predictions are 5.0, 3.0, 1.2, and 0.3 mg/L. A separate export shows the corresponding values as 5000, 3000, 1200, and 300, but labels the column mg/L. How should these results be reviewed?
The first series is internally compatible with a nonnegative concentration prediction in the stated units. The second series has values exactly 1000 times larger. Since \(1000\) ng/mL equals \(1\) mg/L, the exported numbers would correspond to 5.0, 3.0, 1.2, and 0.3 mg/L if they were in ng/mL. That pattern makes a unit-label or conversion error a testable possibility; it does not prove that this is the cause.
Check the export specification and conversion step. If the second column is actually in ng/mL, correct the unit label or conversion before comparing it with mg/L values. If it is confirmed to be mg/L, the discrepancy remains and requires investigation. A range check should flag values for review based on the variable’s definition and the intended dose and design, not on a universal concentration cutoff.
Justifying the Basic Checks
The expected-record calculation follows from the design. If every one of \(N\) subjects has exactly \(K\) scheduled observation records, then each subject contributes \(K\) records, so the total is the sum of \(K\) records over \(N\) subjects: \(N K\). This expectation assumes the output contains only those records and that no observations are intentionally omitted. If event rows, unscheduled observations, or other record types are included, the expected total must include them explicitly. This is why a row count without a record definition is ambiguous.
The units of a trapezoidal exposure calculation follow from its factors. Concentration has units such as mg/L, while an elapsed interval has units of hours. Their product has units mg·h/L. For the example above, each segment area has those units, and their sum does too. A reported AUC in mg/L alone would therefore be inconsistent with the stated time-integrated definition, even if its numerical value looked plausible.
A useful review record has both an expectation and an observed result. “Concentrations look reasonable” is difficult to reproduce. “The prediction variable is specified in mg/L; the export is labelled mg/L; values in the export are 1000 times the corresponding values in the simulation table” is testable and directs review toward the conversion or labeling step. Keep the expected design, the check performed, and the result together.
Diagnostics, Alternatives, and Limitations
Use simple diagnostics that match the failure being considered. For structure, compare key sets and count missing, duplicate, and unexpected keys. For values, count missing and non-finite entries by variable, inspect minima and maxima, and review summaries by time and subject. For a small number of subjects, a concentration-time table or plot can reveal a wrong time origin, a misplaced observation, or an abrupt jump that a global summary would hide. These checks identify patterns for investigation; they do not replace verification of the model specification.
There are several tempting alternatives, but each answers a different question. A total row count is fast but cannot detect compensating missing and duplicate records. A summary such as the minimum and maximum can reveal extremes but can miss incorrect subject-time assignments or a unit error shared by every value. A plot can show time-pattern problems but cannot establish that the numerical units or exposure definition are correct. The strongest practical review combines these checks with direct comparison to the specified design and an independent calculation for selected derived outputs.
Some outputs may legitimately contain missing values. For example, an exposure measure may be undefined when the required observation interval is unavailable, depending on its definition. Distinguish such planned or definition-driven missingness from an unexplained missing concentration. Likewise, an extreme value is not automatically impossible: its interpretation depends on the model, dose, units, and whether the variable is a prediction or an error-contaminated observation. Record what is expected and why before setting thresholds.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Three subjects each have five planned concentration observations and no other record types. How many rows are expected, and what additional check is needed beyond counting them?
- An AUC is defined over time in hours using concentration in mg/L. What are its units, and why?
- An output contains one negative predicted concentration from a model whose predicted concentration is defined as nonnegative. What should the reviewer conclude?
- A file contains 40 rows as expected, but one planned subject-time pair is absent and another pair appears twice. Is the file complete?
Solutions. (1) \(3 \times 5=15\) rows are expected. Compare the expected subject-time keys with the actual keys and check for missing, duplicate, and unexpected records. (2) The units are mg·h/L because concentration in mg/L is multiplied by elapsed time in hours and the segment areas are summed. (3) The value conflicts with the stated definition and should be flagged for investigation; verify the variable, units, and output path before attributing a cause. (4) No. The total count can remain 40 because the duplicate offsets the missing record; the key comparison exposes both discrepancies.