Why Check Both Pools?
A precursor-pool model can reproduce a product concentration at one time while representing the pathway poorly over the full response. For example, a model might predict the right average product concentration but predict its decline too early, or predict the product well while missing the measured precursor. A visual predictive check (VPC) evaluates whether data simulated from a fitted model reproduce important features of the observed data. For linked pools, it is useful to check both precursor and product concentrations, separated by sampling time and treatment.
A VPC is a model diagnostic, not a test that proves a pathway is correct. It asks whether observed summaries look compatible with summaries generated under the model and its assumptions. A mismatch can indicate inadequate model structure, variability, or measurement-error assumptions; it does not by itself identify the cause.
Model and Simulation Assumptions
For orientation, consider a simple precursor-product system. Let \(P(t)\) and \(Q(t)\) denote the true concentrations of precursor and product at time \(t\). One possible model is
Here \(k_{\mathrm{in}}\) is precursor input, \(k_{\mathrm{conv}}\) is conversion from precursor to product, and \(k_P\) and \(k_Q\) are loss-rate constants for the respective pools. This is an illustrative model, not a requirement for constructing a VPC. A drug effect, baseline variation, additional pools, or other established model features can be included in the model being checked.
A meaningful comparison requires the simulated data to follow the observation process that produced the real data. Simulations should preserve the study design, including treatment assignments, dose histories, sampling times, covariates, and the number of observations in each group. They should also reproduce the model’s relevant sources of variability: for example, between-subject variability, residual error, and any modeled dependence among repeated measurements. If concentrations are measured with lognormal error, the simulation and observed-data summaries must use the same concentration scale and transformations.
The usual plug-in VPC holds the fitted parameter estimates fixed and simulates variability conditional on them. This checks predictive performance at those estimates, but does not fully propagate uncertainty in the parameter estimates. If parameter uncertainty is important, parameter draws from an appropriate uncertainty procedure can be incorporated into the simulation. Either way, the choice must be stated, because it affects the width of the simulated envelopes.
Constructing the VPC
Choose time groups that are clinically and mechanistically useful and contain enough observations for stable summaries. Groups can be exact sampling times in a regular design, or prespecified time bins when times vary. Within each treatment-by-time group, calculate observed concentration summaries, commonly the median and selected percentiles such as the 10th and 90th. Repeat the same calculation for each simulated data set.
For each summary and group, the repeated simulations produce a distribution of simulated summaries. An envelope, such as the 2.5th to 97.5th percentiles of those summaries, describes the range in which that summary falls across 95% of simulated data sets. This is not an interval for an unknown model parameter, nor is it the range expected to contain 95% of individual concentrations. It is an interval for a data summary under the simulation procedure.
The logic is straightforward. If the model and simulation assumptions describe the data-generating process well, an observed summary should generally be compatible with the corresponding distribution of simulated summaries. A summary outside its envelope is a warning that the model may not reproduce that feature. Since many times, treatments, and percentiles may be inspected, an isolated miss can occur by chance; patterns across adjacent times or across both pools are usually more informative than a single point.
Specify time points or bins, treatment strata, concentration scale, percentiles, and any rules for missing or below-quantification observations.
Use the fitted model and preserve the observed design. Include the modeled variability and observation process for both precursor and product measurements.
For observed and simulated data, apply the same grouping, percentile calculation, and any derived-summary rule.
Plot observed summaries against simulated envelopes for each pool and treatment. Look for systematic departures in timing, center, or spread, not just individual points.
Worked Examples
Worked Example: Checking Product Magnitude at One Time
Suppose a hypothetical study measures product concentration at 6 hours in one treatment group. The five observed concentrations are 8.0, 9.0, 10.0, 11.0, and 12.0 mg/L. The observed median is the middle ordered value:
The fitted model is used to generate 1,000 replicated studies with the same five observations at this time and treatment. The median is calculated for each replicate. Suppose the 2.5th and 97.5th percentiles of the 1,000 simulated medians are 8.20 and 9.40 mg/L. The observed median is above the envelope because
Thus the fitted simulation procedure rarely produces a median as high as the observed median in this group. This flags a possible underprediction of product magnitude at 6 hours. It does not establish that the conversion rate is wrong: other model features, variability assumptions, or sampling variation may also contribute. This one comparison also says nothing by itself about the model at other times or for the precursor.
Worked Example: Checking Response Timing on the Observation Grid
In a hypothetical treatment group, consider the observed median product concentrations at scheduled times 0, 2, 4, 6, 8, and 12 hours: 100, 92, 81, 73, 68, and 76 mg/L. Define the summary of response timing as the scheduled time at which this median curve reaches its minimum; if tied, use the earliest time. The observed minimum is 68 mg/L at 8 hours, so the observed minimum time is 8 hours.
Now apply exactly the same rule at exactly the same observation times to each of 1,000 simulated data sets: calculate the simulated median at each scheduled time, then select the time with the lowest simulated median. Suppose the resulting minimum times are 4 hours in 30 replicates, 6 hours in 520 replicates, and 8 hours in 450 replicates. The 5th percentile is 6 hours and the 95th percentile is 8 hours: in sorted order, observations at ranks 50 and 950 both fall within those respective time categories. Thus the central 90% simulation interval for this discrete timing summary is 6–8 hours.
The observed time, 8 hours, falls inside that interval. Under this timing summary, the observed minimum is compatible with the simulated timing distribution. The comparison is valid because both observed and simulated times use the same scheduled grid and the same minimum-of-the-median-curve rule. It would not be valid to compare this observed scheduled-time minimum with simulated minima selected from a much finer grid without defining and applying a common method.
Worked Example: Separating Precursor and Product Checks by Treatment
A hypothetical VPC summarizes concentrations at 4 hours in two treatment groups. The observed medians and 95% simulation envelopes are:
| Group and pool | Observed median | 95% simulated-summary envelope |
|---|---|---|
| Placebo, precursor | 10.2 mg/L | 9.60–10.5 mg/L |
| Placebo, product | 20.1 mg/L | 19.2–20.8 mg/L |
| Treatment, precursor | 6.10 mg/L | 5.50–6.50 mg/L |
| Treatment, product | 13.8 mg/L | 12.4–13.2 mg/L |
For the placebo precursor, \(9.60\leq 10.2\leq 10.5\) mg/L, and for the placebo product, \(19.2\leq20.1\leq20.8\) mg/L. Both observed medians are inside their envelopes. The treatment precursor is also inside its envelope because \(5.50\leq6.10\leq6.50\) mg/L. The treatment product is outside because \(13.8>13.2\) mg/L.
This pattern suggests that the simulated model reproduces the treatment-group precursor median and both placebo medians at this time, but underpredicts the treatment-group product median. Because the precursor and product are linked, the contrast can guide follow-up checks of the pathway and observation model. It does not prove a particular mechanism or parameter is responsible.
Diagnostics, Alternatives, and Limitations
A useful VPC displays observed and simulated medians as well as lower and upper percentiles. A median-only display can hide a model that gets the center right but misses the spread. Inspect precursor and product panels separately, and compare treatment groups at corresponding times. A consistent upward or downward departure across adjacent times is more concerning than one isolated point. Changing envelope width can also matter: for example, observed lower and upper percentiles both falling outside simulated envelopes suggests a problem with the distribution’s spread, not merely its center.
Several practical issues can make a VPC misleading. Very small time-by-treatment groups yield unstable observed percentiles; overly broad bins can conceal a rapid change in response; and bins with unequal observation counts can make visual comparisons difficult. Sparse sampling may make a timing summary coarse or ambiguous. Censoring and below-quantification values require matching rules in observed and simulated data. Ignoring repeated-measure dependence or simulating the wrong residual-error form can make envelopes too narrow or too wide. If the simulation omits important variability, observed departures may look more surprising than they should.
A VPC is often more informative than comparing only fitted curves with observed means, because it displays the variability expected under the model and can reveal mismatches in distributional spread. It differs from an individual prediction interval, which concerns a future individual observation, and from a confidence interval for a parameter, which concerns uncertainty in an estimated model quantity. Residual plots provide a different diagnostic view by displaying discrepancies observation by observation; they do not replace a distributional comparison of simulated and observed data. No single display is definitive, and VPC interpretation should be considered alongside the design and other model diagnostics.
The most important limitation is that a VPC checks the model as a whole, conditional on its implementation and the available design. It may have little power to distinguish competing pool structures if sampling times do not separate their predicted responses. Similarly, parameter estimates obtained from the same data can make a plug-in check optimistic. Simulation-based envelopes should therefore be interpreted as evidence about predictive compatibility, not as a mechanism-selection rule.
Check Your Understanding
Answer each question using the stated VPC definitions and rules.
- What must be held consistent when calculating an observed percentile and its simulated counterpart?
- At one time and treatment, the observed product median is 7.2 mg/L and the 95% simulated-summary envelope is 6.1–7.5 mg/L. What does this comparison say, and what does it not say?
- Why is a product minimum time computed on scheduled visits not directly comparable with a simulated minimum time computed on a fine time grid?
- A VPC reproduces the product median but repeatedly misses the lower and upper product percentiles. What feature may be inadequately represented?
Solutions. (1) Use the same time and treatment groups, data scale, percentile definition, and handling rules; for derived timing summaries, use the same observation grid and selection rule. (2) Since \(6.1\leq7.2\leq7.5\) mg/L, the observed median is compatible with that simulated-summary envelope. This does not validate the model or establish that individual future observations will lie in the envelope. (3) The finer grid can select a different minimum time simply because it offers more candidate times; the summaries must use a justified common grid and rule. (4) The model may misrepresent concentration variability or distributional spread, or its residual-error or between-subject variability assumptions may be inadequate. The VPC flags the feature but does not uniquely identify its cause.