What a Visual Predictive Check Assesses
A visual predictive check (VPC) asks whether a fitted model can reproduce important features of the observed data under the study’s design. It does this by comparing observed summaries with summaries from datasets simulated using the model. The comparison is useful because an overall fit can look adequate while the model misses a pattern at a particular time or in a particular subgroup.
A VPC is a model-checking display, not a single numerical verdict. Its interpretation depends on what was simulated, how the data were grouped, which summaries were plotted, and whether the simulation reflects the observed design. The earlier material on goodness-of-fit displays concerns complementary views of model fit; a VPC specifically uses repeated model-based simulations to establish a reference range for data summaries.
For a continuous pharmacometric outcome, a common VPC plots the observed median and lower and upper quantiles within bins, such as the 5th, 50th, and 95th percentiles. For each simulated dataset, the same summaries are calculated in the same bins. An interval across the simulated values of each summary is then shown, often as a shaded band. Other VPCs compare observed and simulated proportions, counts, or categorical outcomes.
This distinction matters. A 90% prediction interval around a simulated median is not a claim that 90% of individual concentrations should fall in that band. It is an interval for the median that would be calculated from a replicated dataset under the model and the specified design. A separate lower or upper quantile display checks the corresponding tail of the outcome distribution.
How to Read the Simulation Intervals
Let \(B_k\) denote the observations in bin \(k\), and let \(q_{r,k}^{obs}\) be the observed \(r\)th quantile in that bin. Simulate \(M\) replicated datasets. For replicate \(m\), calculate the same quantile, \(q_{r,k}^{(m)}\), using the same bin definition. A central 90% simulation interval can be formed from the 5th and 95th percentiles of the \(M\) replicated quantiles.
Here \(Q_{0.05}\) and \(Q_{0.95}\) are the 5th and 95th percentiles across simulated datasets. The observed quantile is compared with this interval. If it lies inside, that summary is compatible with the range of values commonly generated by the simulations at that bin. If it lies outside, the observed summary is unusually high or low relative to that reference, which is a signal to investigate rather than proof that the model is wrong.
The logic is simulation-based: the fitted model generates replicated datasets, and each dataset produces the statistic being checked. The interval estimates the central range of that statistic under the simulation setup. With a correctly specified model and known parameters, observed and replicated statistics would follow the same distribution conditional on the design. In an applied VPC, parameters are estimated from the observed data, so this is generally a model diagnostic rather than an exact, calibrated hypothesis test.
The comparison is informative only if the simulations reflect the relevant features of the data collection. At minimum, the simulation should use the fitted model and the observed study design, including relevant dosing and sampling times, covariates, and subgroup structure. Data handling that affects the plotted outcome—such as the treatment of censored observations—must also be represented consistently. If simulated datasets use a different design or different bins from the observed data, apparent agreement or disagreement may be misleading.
A standard VPC often conditions on the fitted parameter estimates. Unless parameter uncertainty is explicitly incorporated, the plotted intervals do not include the full uncertainty in those estimates. Reports should identify the prediction interval level, the summaries shown, the binning approach, the stratification, and whether parameter uncertainty was included.
Worked Example: Reading an Overall VPC
Hypothetical data: A model is assessed using 1,000 simulated datasets. In each of three time bins, the observed 5th, 50th, and 95th concentration percentiles are compared with their respective central 90% simulation intervals. Concentrations are in mg/L.
| Time bin (h) | Observed 5th percentile | 90% interval for 5th percentile | Observed median | 90% interval for median | Observed 95th percentile | 90% interval for 95th percentile |
|---|---|---|---|---|---|---|
| 0–2 | 1.2 | 0.8–1.6 | 8.4 | 7.8–9.0 | 18.2 | 16.5–20.1 |
| 2–6 | 0.6 | 0.3–0.9 | 5.2 | 4.7–5.8 | 12.4 | 11.0–13.8 |
| 6–12 | 0.2 | 0.1–0.4 | 2.1 | 1.8–2.5 | 6.0 | 5.1–7.0 |
For the first bin, the observed median of 8.4 mg/L lies between 7.8 and 9.0 mg/L, the simulated 90% interval for the median. The observed lower quantile, 1.2 mg/L, lies between 0.8 and 1.6 mg/L; the upper quantile, 18.2 mg/L, lies between 16.5 and 20.1 mg/L. The same checks hold in the other two bins. For example, the final-bin median of 2.1 mg/L is within 1.8–2.5 mg/L, and its observed 95th percentile of 6.0 mg/L is within 5.1–7.0 mg/L.
This display shows no conspicuous departure in the three plotted summaries across these bins. It supports the limited conclusion that the observed quantiles are compatible with the simulated ranges at the displayed times. It does not establish that every model feature is adequate, or that individual observations should fall inside the shaded bands.
Why Time and Subgroup Patterns Matter
Overall coverage can conceal local misfit. If observed summaries are alternately high and low across time, the pooled or average picture may look satisfactory even though the model misses a systematic time trend. Likewise, opposing discrepancies in different subgroups can cancel in a combined display. A stratified VPC repeats the comparison within clinically or scientifically relevant groups, such as dose level, age category, or a prespecified covariate group.
Stratification is most useful when the groups address a question about model adequacy, rather than simply dividing the data until an unusual pattern appears. Each subgroup needs enough observations across the relevant bins for its observed summaries to be informative. When subgroups are small, the observed quantiles can be unstable and the simulation intervals may be wide. That is a limitation to report, not a reason to interpret a noisy point as decisive evidence.
Worked Example: A Time-Specific Departure
Hypothetical data: At 8–12 hours after dosing, the observed median concentration is 72 mg/L. Across 1,000 simulated datasets, the central 90% interval for that binned median is 58–68 mg/L.
The observed median exceeds the upper endpoint by \(72-68=4\) mg/L. It is therefore above the central 90% simulation interval for the late-time median. Suppose the observed medians in earlier bins are within their respective intervals. The pattern points to a possible late-time discrepancy, not a uniform failure across the full profile.
The report should describe the direction, size, and location of the discrepancy: “At 8–12 hours, the observed median was 72 mg/L, 4 mg/L above the upper endpoint of its 58–68 mg/L simulation interval; earlier binned medians were within their intervals.” It should then consider whether the pattern is supported by other displayed summaries and whether the late-time bin contains enough observations. The VPC alone does not identify the cause. Possible explanations may include model structure, data or simulation setup, or sparse late-time observations; further investigation is needed before attributing the pattern to any one of them.
Worked Example: Stratification Reveals Opposing Misfit
Hypothetical data: In one time bin, two equally sized subgroups have observed concentrations (mg/L) of 11, 12, and 13 in Group A, and 7, 8, and 9 in Group B. Their observed subgroup medians are 12 mg/L and 8 mg/L. In the stratified simulations, the 90% interval for each subgroup median is 9–11 mg/L.
For Group A, the observed median is \(12\) mg/L, which is \(12-11=1\) mg/L above its interval’s upper endpoint. For Group B, the observed median is \(8\) mg/L, which is \(9-8=1\) mg/L below its interval’s lower endpoint. Thus the two subgroup summaries depart in opposite directions.
Combining the six observed values gives the ordered values 7, 8, 9, 11, 12, and 13 mg/L. Under the usual even-sample median calculation, the pooled median is \((9+11)/2=10\) mg/L. If the pooled median’s simulation interval is, for example, 9–11 mg/L, the pooled display appears compatible with simulation while the stratified display shows opposing departures. The pooled median has concealed the subgroup pattern. The example supports checking relevant strata, but it does not imply that every small subgroup departure is reliable: subgroup sample size and the broader pattern remain important.
Diagnostics, Alternatives, and Reporting Limits
Before interpreting a VPC, check whether the display itself is dependable. Bins that are too narrow may contain too few observations for stable quantiles; bins that are too wide can average over a changing profile and hide time-specific misfit. Uneven sampling may leave some bins sparse or make their contents unrepresentative of the intended time range. Review bin boundaries and the number of observations per bin, and use consistent definitions for observed and simulated data.
Also check whether the simulated datasets reproduce the study design and data-processing choices, whether the observed and simulated summaries are the same statistics, and whether the plot’s axes and units are correctly labelled. A band that is extremely wide may reflect substantial variability or weak information in that bin; it should not be read as evidence of especially good fit. If many bins or subgroups are inspected, some pointwise observations outside their intervals are unsurprising. Adjacent bins and summaries are also related, so the departures should not be counted as independent formal tests.
A VPC and observed-versus-predicted or residual displays answer different diagnostic questions. The latter can show individual-level patterns or relationships between residuals and predictions; a VPC asks whether simulated datasets reproduce selected distributions or summaries under the design. Neither display replaces the other. A VPC also does not establish that parameters are precise, that the model is uniquely correct, or that a simulated result outside the displayed data range is supported by observations.
State which outcome summaries, bins, and subgroups are displayed, and what simulation interval surrounds each summary.
Confirm that observed and simulated data use consistent study-design features, bin definitions, and relevant data handling; inspect the amount of data supporting each bin or stratum.
Note whether observed summaries lie inside or outside intervals, in which direction, and whether departures cluster by time or subgroup.
Report what the display supports and its limitations. Do not convert pointwise interval crossings into a formal pass/fail rule or a causal explanation.
Reporting a VPC Conclusion
A useful report names the summaries and strata examined, gives the direction and location of notable departures, and makes clear whether the comparison used pointwise simulation intervals. Avoid saying “the model passed the VPC.” Prefer a bounded statement such as: “Observed medians and the plotted lower and upper quantiles were within their pointwise 90% simulation intervals across the displayed time bins. This supports compatibility for these summaries and bins; it does not establish adequacy for unplotted features.” If a departure appears, report its magnitude and location, and state what is uncertain about its interpretation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A plotted 90% interval surrounds the simulated binned median. What quantity does this interval describe, and what does it not describe?
- In a hypothetical bin, the observed 95th percentile is 15 mg/L and its simulation interval is 12–14 mg/L. State the comparison accurately and give one reason not to treat it as a formal model rejection.
- Why can a pooled VPC appear adequate when a stratified VPC shows departures in opposite directions?
- Name two features of the simulation or display that should be checked before interpreting an apparent time-specific discrepancy.
Solutions. (1) It describes a central range for the binned median calculated across replicated datasets under the model and simulation setup. It is not an interval intended to contain 90% of individual observations. (2) The observed 95th percentile is 1 mg/L above the interval’s upper endpoint, so it is outside the central 90% simulation interval for that summary. A pointwise interval crossing is not a formal rejection, and the bin may have limited data or be one of many inspected bins. (3) Opposing subgroup differences can cancel when data are combined, leaving a pooled summary near the simulated range even though subgroup summaries depart in different directions. (4) Examples include checking that simulations reflect the observed dosing and sampling design, that the observed and simulated summaries use identical bin definitions, and that the bin has enough observations for a stable quantile.