Why Simulate Concentration-Time Data?
A population PK model is useful in development only if it can reproduce concentration patterns that matter for decisions. A fitted model may describe the average concentration-time trend while missing the spread between patients, the late-time decline, or the frequency of unusually high concentrations. Simulation-based diagnostics make these features visible by asking a direct question: if the fitted model generated new data under the study’s design, would those data look like the observed data?
A visual predictive check (VPC) summarizes this comparison over time. It compares observed concentration quantiles with the corresponding quantiles from many simulated datasets. It is therefore complementary to the goodness-of-fit diagnostics used to examine residual patterns: a VPC evaluates population-level predictive distributions, while residual diagnostics can reveal systematic patterns at the observation level. Neither by itself establishes that a model is correct.
What Must Be Simulated?
A simple representation of a concentration observation for individual \(i\) at time \(t_{ij}\) is
Here \(C_{ij}\) is the observed concentration, \(f\) is the structural PK model, \(D_i\) represents that individual’s dosing history, \(X_i\) represents relevant covariates, \(\theta\) denotes population parameters, and \(\eta_i\) represents individual deviations described by the model. The residual term \(\epsilon_{ij}\) represents unexplained observation-level variation under the specified residual error model. The exact form of \(f\), the random effects, and the residual error depends on the model being checked.
For a usual population predictive check, each replicate dataset is generated using the fitted parameter estimates, new individual deviations drawn from the model’s population distribution, and residual error generated according to the fitted residual model. The replicated data should reflect the study being assessed: preserve or appropriately reproduce its dose regimens, covariate values, sampling schedule, and relevant analysis rules. If these features are changed, the check answers a different question.
The check also relies on the model being implemented as specified, the simulated observations being comparable to the observed measurements, and the chosen summaries being informative for the decision. The population and design used for simulation must be explicit. For example, keeping the study’s covariate distribution fixed asks whether the model reproduces observations for that study population; drawing covariates from a different target population asks about that alternative population instead.
Building and Reading a VPC
A common approach divides time into bins, calculates an observed median and lower and upper quantiles within each bin, and repeats those calculations for every simulated dataset. For example, a display might compare the observed 10th, 50th, and 90th percentiles with simulation-based bands for those percentiles. The lower and upper quantiles describe the distribution of concentrations, while the bands describe how much the corresponding quantile varies across simulated datasets.
The quantile convention should be stated, especially in small samples. For the examples here, the nearest-rank sample quantile is used: for \(n\) ordered observations, the \(p\) quantile is the observation at rank \(\lceil np\rceil\), where \(0<p\leq1\). The sample median for an even \(n\) is instead the average of the two central observations. Software may use other interpolation conventions; observed and simulated summaries must use the same convention.
A typical simulation procedure is:
State the population, dosing and sampling design, time bins, concentration summaries, and any decision-relevant tail or threshold to assess.
Using the fitted model, simulate many datasets that reproduce the relevant design and include the model’s individual and residual variability.
For observed data and each replicate, calculate the same quantiles or other summaries within the same time bins.
Identify systematic departures, such as observed quantiles repeatedly outside the simulation bands, and consider their pattern and clinical relevance.
For a particular time bin and quantile, let \(Q_r\) be that quantile from simulated replicate \(r\), for \(r=1,\ldots,R\). A central 95% simulation band can be estimated from the 2.5th and 97.5th percentiles of \(Q_1,\ldots,Q_R\). If \(Q_{\mathrm{obs}}\) is the corresponding observed quantile, its position relative to this band indicates whether the observed summary is unusual under the fitted model and design.
This is a pointwise band for one summary at one time bin. It is not a simultaneous band guaranteeing coverage of the entire plotted curve, and it is not a confidence interval for a population parameter. A derivation clarifies its interpretation: if the fitted model and design adequately describe the data-generating process, the observed quantile and each replicate quantile are generated under comparable conditions. The distribution of replicate quantiles therefore provides a reference for the observed quantile. A value outside the central 95% band is a warning of a mismatch, not a formal proof of model failure; checking many bins and quantiles also makes occasional outside values unsurprising.
Worked Example: Checking the Typical Concentration Profile
Suppose a hypothetical study has eight observations in a time bin around 4 hours after dosing, in mg/L. Ordered observed concentrations are 4, 5, 6, 7, 8, 9, 10, and 12. Across 500 simulated datasets, the 95% bands for the 10th, 50th, and 90th percentiles in this bin are, respectively, 3.2–5.1, 6.8–8.4, and 9.7–13.8 mg/L. Assume each replicate used the study’s dose and sampling design and the model’s fitted variability.
The observed median is the average of the two central values: \( (7+8)/2=7.5\) mg/L. The observed 10th percentile is the first ordered value, because \(\lceil 8(0.10)\rceil=1\), so it is 4 mg/L. The observed 90th percentile is the eighth value, because \(\lceil 8(0.90)\rceil=8\), so it is 12 mg/L.
All three observed summaries fall within their corresponding simulation bands: 4 is between 3.2 and 5.1 mg/L, 7.5 is between 6.8 and 8.4 mg/L, and 12 is between 9.7 and 13.8 mg/L. In this bin, there is no clear evidence that the model misses the typical concentration or these two distributional quantiles. This small example does not establish that the model reproduces the profile at other times or in other subgroups.
Assessing Variability and Clinically Important Tails
The median alone is not enough. A model can reproduce the central concentration while underpredicting between-patient variability. Comparing lower and upper quantiles across time helps reveal that problem. If the observed 10th percentile is persistently below its simulated band while the median is well reproduced, for instance, the model may fail to capture the low-concentration part of the distribution. Conversely, an upper-tail mismatch may matter when high exposure is associated with a safety concern.
Quantiles are not always the most decision-relevant summaries. A VPC can also compare a prespecified threshold exceedance proportion, such as the fraction of patients whose concentration exceeds a clinically relevant concentration \(c^*\). The observed proportion should be compared with the distribution of the same proportion across replicate datasets. The threshold and summary should be chosen for a clinical or development question, not selected after inspecting the data simply because it gives a striking result.
Worked Example: Checking the Upper Tail
In a hypothetical time bin, ten observed concentrations in mg/L are 2, 3, 4, 5, 6, 8, 10, 12, 15, and 20. Use the nearest-rank convention for the 10th and 90th percentiles, and consider concentrations greater than 12 mg/L as a prespecified high-concentration flag. Assume 500 design-matched simulations give a 95% band of 1.5–3.8 mg/L for the 10th percentile, 12.5–16.8 mg/L for the 90th percentile, and 0.10–0.30 for the flag proportion.
The median is \( (6+8)/2=7\) mg/L. The 10th percentile is at rank \(\lceil10(0.10)\rceil=1\), giving 2 mg/L. The 90th percentile is at rank \(\lceil10(0.90)\rceil=9\), giving 15 mg/L. Both quantiles fall within their respective simulation bands.
Two of the ten observations exceed 12 mg/L, so the observed flag proportion is \(2/10=0.20\), also within its simulation band of 0.10–0.30. These summaries do not show an obvious upper-tail discrepancy in this bin. With only ten observations, however, the tail summaries are imprecise and sensitive to individual values; compatibility with the bands should not be read as strong evidence that rare high concentrations are well characterized.
Preserve the Design and the Predictive Target
A VPC can be misleading when simulations do not correspond to the observed design. Differences in dose, covariate composition, or sampling times can change the concentration distribution even if the model is unchanged. For example, comparing an observed subgroup with simulations representing the whole study may obscure a subgroup-specific mismatch. Stratified checks or simulations matched to the subgroup may be more informative, provided the available data support the comparison.
Repeated observations from the same patient are also related. A concentration-time profile is not a collection of independent measurements. Simulations should generate whole profiles under the model, and uncertainty in the plotted summaries should reflect variation among replicate datasets generated at the appropriate individual level. Treating every concentration as an independent patient can make the apparent amount of information much larger than it is.
Worked Example: Checking a High-Exposure Proportion Under the Study Design
A hypothetical study enrolls 30 patients on a fixed regimen. The development question is whether the model reproduces the proportion of patients whose simulated steady-state trough concentration exceeds 8 mg/L. Six observed patients exceed the threshold. Five hundred replicate datasets are simulated using the same 30 patients’ covariates, regimen, and trough sampling times, with individual variability and residual error included. The simulated distribution has a median of 5 exceedances and a central 95% range of 2–9.
The observed proportion is \(6/30=0.20\), or 20%. The simulated median proportion is \(5/30=0.167\), or 16.7%; the simulated range in proportion units is \(2/30=0.0667\) to \(9/30=0.300\). The observed six exceedances fall within the simulated range of two to nine. Thus, this check does not indicate a clear discrepancy in this threshold-based summary under the stated study design.
The conclusion is limited to this summary and target. It does not establish accurate prediction for a different regimen, another covariate distribution, or a rarer tail event. Because the same covariate composition was retained in each replicate, this check asks whether the model reproduces the study’s observed population rather than a broader target population.
Diagnostics, Alternatives, and Limitations
A useful predictive check looks for coherent patterns, not isolated plotted points. Repeated departures across adjacent time bins can suggest a time-dependent structural mismatch. A median that is reproduced while the upper quantile is repeatedly too low can suggest inadequate predictive spread or upper-tail behavior. A mismatch confined to a subgroup may be hidden in an overall VPC, so the overall display should be supplemented with a focused stratification when scientifically justified.
Several practical issues can weaken interpretation. Sparse sampling or few patients per bin makes observed quantiles unstable, especially in the tails. Bins that are too narrow may contain too few observations; bins that are too wide may average over meaningful changes in concentration. The binning strategy should be selected with the sampling design and intended time resolution in mind. Repeating the check with a reasonable alternative binning can help determine whether an apparent pattern is an artifact of bin boundaries.
Simulation bands are also affected by the number of replicates. With too few replicates, their estimated endpoints are noisy; increasing the number of simulations improves Monte Carlo precision but does not repair a misspecified model or unrepresentative design. In addition, a band for a fitted model ordinarily uses its parameter estimates as fixed. It reflects variation generated by the model under those estimates, not uncertainty in the parameter estimates. Parameter uncertainty is a separate issue.
Compared with a VPC, residual-based goodness-of-fit plots can more directly reveal patterns between individual predictions and observations, but may depend on how individual predictions and residuals are constructed. Simulation-based checks avoid relying on a single residual summary and can target clinically meaningful quantiles or thresholds, but they do not pinpoint the cause of a mismatch. External evaluation asks whether predictions transport beyond the data used for fitting; a VPC using the fitting dataset does not answer that question.
Check Your Understanding
Answer each question, then compare with the solutions below.
- What does a 95% simulation band for the median in one time bin represent, and what does it not represent?
- In a bin with ten observations, the ordered concentrations are 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10 mg/L. Using the conventions in this tutorial, calculate the median and 90th percentile.
- Observed upper quantiles are repeatedly above their simulated bands, while medians agree. Give one interpretation and one reason not to treat this pattern as a definitive diagnosis.
- Why should simulated replicate datasets preserve the observed study’s dosing and sampling design when the goal is to check fit to that study?
Solutions. (1) It summarizes the central 95% of the simulated values of the median for that bin under the fitted model and design. It is pointwise, not a simultaneous band for an entire curve, and it is not a confidence interval for a model parameter. (2) The median is \( (5+6)/2=5.5\) mg/L. The 90th percentile is at rank \(\lceil10(0.90)\rceil=9\), so it is 9 mg/L. (3) The model may reproduce typical concentrations but underrepresent the high-concentration tail. However, sparse bins, sampling variation, and multiple comparisons across bins can produce departures, so the pattern requires investigation rather than a single-cause conclusion. (4) Dosing and sampling determine which concentrations are generated and observed; changing them changes the predictive target, making observed and simulated summaries no longer directly comparable for that study.