Why Check Simulated Data Integrity?
A simulation can be reproducible and still be wrong. A fixed random seed makes a run repeatable, but it does not establish that the code drew the intended random effects, applied the right dose events, or generated residuals from the specified error model. Integrity checks compare simulated records with the rules the simulation was meant to implement.
The distinction from model-fit assessment is important. A fit diagnostic asks whether a model describes observed data adequately. An integrity check asks whether the simulated data were generated according to the stated model and design. A simulation can faithfully implement a poorly fitting model, or produce an excellent-looking plot despite an implementation error. The checks here target the second question: did the simulation do what its specification says?
Before checking a simulation, write down its intended structure. For subject \(i\), a typical model draws a random-effect vector \(\eta_i\) from a population distribution such as \(\eta_i\sim N(0,\Omega)\), maps population parameters and random effects to individual parameters, and then generates observations from a structural prediction plus residual error. The exact mapping and error model depend on the specification. Checks must be tailored to those rules rather than to generic expectations such as “all values should look normal.”
Begin With Exact and Stochastic Checks
An exact check evaluates a rule that should hold for every applicable record if the implementation is correct. Examples include one subject identifier per subject, a planned dose at each specified time, and no missing or duplicated event rows. A stochastic check compares a random sample summary with its target distribution. A finite simulated sample is not expected to reproduce a population mean or covariance exactly, so discrepancies must be interpreted relative to sample size and Monte Carlo variability.
For each variable, define its permitted support and units. Clearance and volume, for example, must be positive when represented directly in a conventional compartment model; a log-scale random effect may be any real number before transformation. Check for missing values, infinities, unexpected zeros, implausible extremes, and unit mismatches. A value can be mathematically possible but still signal a bug if it falls outside the model’s specified parameterization or the simulation’s intended covariate range.
Worked Example: Checking Parameter Support
Suppose a hypothetical simulation draws individual clearance values in liters per hour from a model in which clearance is generated by \(CL_i=5\exp(\eta_{CL,i})\), with a positive typical value of 5 L/h. Five simulated values are 4.2, 5.1, 3.8, 6.0, and \(-0.4\) L/h.
The exponential transformation requires \(CL_i>0\) for every finite \(\eta_{CL,i}\). The first four values satisfy this support condition. The value \(-0.4\) L/h does not, so it cannot have been produced by the stated transformation. This is an integrity failure to investigate, not a low-probability tail draw from this parameterization.
A useful implementation check is to compare the stored value with the reconstructed value. If the saved random effect for the fifth subject were, for example, \(\eta_{CL,5}=0.20\), then \(5\exp(0.20)=6.107\) L/h, not \(-0.4\) L/h. That mismatch could arise from a column-order error, a sign or transformation error, or a record-joining problem. The check does not establish that the model is scientifically appropriate; it establishes that this value is inconsistent with the stated generation rule.
Check Random-Effect Covariance Without Demanding Exact Equality
When the random effects are generated with covariance matrix \(\Omega\), their dependence is part of the simulation specification. If \(\eta_i\sim N(0,\Omega)\), then the target population covariance is \(\Omega\), but the sample covariance from a finite set of simulated subjects will usually differ from it. A large discrepancy may indicate a coding error; a small discrepancy is not itself evidence that anything went wrong.
For \(m\) simulated subjects, the sample covariance between two random effects is calculated after centering each effect by its sample mean. For two components \(\eta_1\) and \(\eta_2\), this is \(s_{12}=\sum_{i=1}^m(\eta_{1i}-\bar{\eta}_1)(\eta_{2i}-\bar{\eta}_2)/(m-1)\). Check that the simulated columns correspond to the intended effects, that their means and variances are broadly plausible, and that the covariance has the intended sign and approximate scale. Also check the generation operation itself: if \(\Omega=LL^\mathsf{T}\) and \(z\sim N(0,I)\), then \(\eta=Lz\) has covariance \(\Omega\). An incorrectly ordered multiplication can yield a different covariance.
Worked Example: Verifying a Two-Effect Covariance
Consider four hypothetical paired random-effect deviations, already centered at their sample means: \((0.1732,0.3549)\), \((0.1732,-0.0951)\), \((-0.1732,-0.3549)\), and \((-0.1732,0.0951)\). Suppose the target covariance matrix is
There are \(m=4\) pairs, so the sample covariance denominator is \(m-1=3\). The cross-products are positive for the first and third pairs and negative for the second and fourth pairs. Thus,
This sample covariance is close to the target value \(0.03\). The minus sign matters: using a plus sign would give \([2(0.1732\times0.3549)+2(0.1732\times0.0951)]/3\approx0.05196\), which does not represent the cross-products of the listed pairs. The example demonstrates why pairing and signs must be checked rather than inferred from marginal summaries alone.
This tiny sample is illustrative, not a strong diagnostic of covariance generation. With only four subjects, sample covariance is highly variable. In an actual simulation audit, inspect many independently simulated subjects and compare the empirical covariance and correlation with their targets, allowing for finite-sample variation. A wrong sign or a stable gross discrepancy across large runs is more concerning than a modest difference in one small run.
Audit Dose Histories as Events
Dose histories are often exact design inputs rather than random outcomes. Represent each dose as an event with at least a subject identifier, dose amount, dose time, and units; include route or compartment when the model requires them. Compare the simulated event table with the planned regimen. Check that each subject receives the intended number and amount of doses, that event times use the intended time origin and units, and that doses are assigned to the correct subject. Observation records and dose-event records should not be confused: an event need not coincide with an observation.
Worked Example: Finding a Dose-Event Error
A hypothetical fixed regimen calls for 100 mg at times 0, 12, and 24 hours for each subject. The expected event sequence for subject S07 is therefore \((0,100)\), \((12,100)\), and \((24,100)\), where each pair is (time in hours, dose in mg). The simulated event table instead contains \((0,100)\), \((12,100)\), \((12,100)\), and no event at 24 hours.
The total recorded amount is 300 mg, which equals the planned total, but the history is still wrong: there are three event rows, a duplicated dose at 12 hours, and a missing dose at 24 hours. A total-dose check alone would miss the timing and duplication error. The direct audit sorts events by subject and time, then compares each event with the specified schedule. Here, the expected times are \(\{0,12,24\}\) hours and the actual times are \(\{0,12,12\}\) hours, so both an extra event and a missing event are identified.
If the input schedule were specified in days while the simulation interpreted it as hours, event times would also fail the unit-aware comparison. This check concerns whether the schedule was implemented correctly; it does not determine whether the regimen is clinically suitable.
Check Residual-Error Generation
Residual checks must follow the specified error model. For an additive model, observations may be written \(Y_{ij}=f_{ij}+\epsilon_{ij}\), where \(f_{ij}\) is the structural prediction and, for example, \(\epsilon_{ij}\sim N(0,\sigma^2)\). In that case, residuals \(Y_{ij}-f_{ij}\) have target mean zero and variance \(\sigma^2\), conditional on the model inputs. For a proportional or combined error model, the appropriate scale differs; do not test raw residuals against an additive-error target if the simulation did not use additive error.
Check that errors are generated at the intended level (for example, independently by record if that is what was specified), with the right scale and dependence. Inspect residuals by subject, time, and prediction range for signs of accidental reuse, omitted scaling, or a wrong error formula. These are implementation checks, not a demand that every finite sample have a perfect bell shape or zero sample mean. With few records, summaries and plots can fluctuate substantially even under correct generation.
Worked Example: Checking Standardized Additive Errors
Suppose a hypothetical data set uses an additive residual model with \(\sigma=0.20\) mg/L. Six simulated standardized errors, defined as \(z_{ij}=\epsilon_{ij}/\sigma\), are \(-1.1, 0.4, 0.8, -0.3, 1.5,\) and \(-0.7\). Under the specified normal-error generator, each standardized error should have target mean zero and variance one.
Their sum is \(0.6\), so their sample mean is \(0.6/6=0.10\). The sum of squared values is \(1.21+0.16+0.64+0.09+2.25+0.49=4.84\). Their sample variance is therefore \([4.84-6(0.10)^2]/(6-1)=4.78/5=0.956\). These summaries are near the targets, but six observations are too few to confirm normality or rule out dependence. The calculation is a basic check for gross scaling errors, not a formal proof that the generator is correct.
Also inspect how the errors enter the observation equation. If the code adds \(\epsilon_{ij}\) directly but the intended model multiplies the prediction by a proportional error factor, the standardized residual calculation above is not the right integrity check. Reconstructing observations from the saved prediction and error draw is more direct: verify that the stored \(Y_{ij}\) equals the value given by the specified equation, up to numerical precision.
A Practical Audit and Its Limits
Record parameter transformations and support, the target \(\Omega\), covariate rules, dose events, observation schedule, and residual-error equation with units.
Verify identifiers, expected row and event counts, missingness rules, dose times, and any deterministic transformations that must hold record by record.
Summarize simulated parameters, random effects, and standardized errors. Compare with their targets while recognizing that finite simulated samples vary.
Reconstruct an individual parameter, dose history, prediction, or observation from stored inputs. Compare the reconstructed value with the saved record and investigate discrepancies.
These checks complement, rather than replace, code review, unit tests, and independent implementations of small test cases. Exact checks are strong for deterministic rules but cannot establish that every model component is scientifically correct. Stochastic summaries can reveal gross errors but may miss subtle ones or appear unusual by chance. Increasing the number of simulated subjects can make population summaries more informative, but it does not correct a wrong formula.
The main alternative is to focus on model-fit diagnostics or simulated-versus-observed comparisons. Those are useful for questions about model adequacy, but they do not guarantee correct event handling or random-number generation. Conversely, an integrity audit can confirm that code faithfully implemented the specified model without showing that the specification captures real data. Keep the questions separate: first check that the simulation works as specified; then address whether the specification is adequate for the scientific purpose.
Common Mistakes
- Demanding exact equality for random summaries. The empirical mean or covariance of a finite sample need not equal its population target. Look for gross or repeatable discrepancies, not perfect matching.
- Checking marginal distributions but ignoring paired effects. Two random-effect columns can each have plausible distributions while their covariance is wrong. Preserve subject-level pairings when calculating covariance.
- Checking only the total dose. Equal total amount does not guarantee correct timing, count, or subject assignment. Compare the full event history.
- Applying the wrong residual scale. Raw residuals from a proportional or combined model do not generally have the same target scale as additive residuals. Standardize using the specified error equation.
- Treating an integrity pass as evidence of model fit. Correct implementation establishes consistency with the simulation specification, not agreement between that model and real observations.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A parameter is generated as \(V_i=10\exp(\eta_i)\), but one simulated value is \(-2\) L. What does this imply?
- Why can a sample covariance differ from the target \(\Omega_{12}\) even if random effects were generated correctly?
- A subject is scheduled for three doses with total amount 300 mg, but the event table has four rows because one time is duplicated and another is missing. Is the dose history valid?
- In an additive model with \(\sigma=0.5\) mg/L, a residual is \(0.25\) mg/L. What is its standardized value, and what is the target scale for standardized errors under the stated normal model?
- What does a successful integrity audit establish, and what does it not establish?
Solutions. (1) It is inconsistent with the stated transformation because an exponential is positive, so investigate the stored value, transformation, or record mapping. (2) A finite sample fluctuates around the population covariance; the sample covariance is not required to equal its target exactly. (3) No. The amount total is insufficient: the duplicated and missing event mean the timing and count are wrong. (4) The standardized residual is \(0.25/0.5=0.50\); under the stated additive normal-error model, standardized errors have target mean zero and variance one. (5) It establishes that checked records conform to the specified generation rules within the checks performed. It does not establish that the model is scientifically adequate or fits observed data.