Why Review a Visual Predictive Check?
A visual predictive check (VPC) compares summaries of observed data with the corresponding summaries from data simulated under a fitted model. It is useful only when the comparison is constructed on a consistent basis. If the simulated dosing history, time variable, observation process, bins, or strata do not match the observed data, a plot can look concerning—or reassuring—for reasons unrelated to the model’s ability to describe the data.
The review task is therefore twofold: assess whether the observed summaries fall within the simulated prediction bands, and verify that the code producing those summaries compares like with like. The latter includes checking the inputs to simulation, the records retained for plotting, and the rules used to group records. A VPC is a diagnostic, not a substitute for checking that the model implementation agrees with its intended specification.
For example, a VPC might display the observed 5th, 50th, and 95th concentration quantiles in each time bin, together with a separate simulation-based band for each quantile. The band is not the range of concentrations in simulated individuals. It describes variation across simulated replicates in the chosen summary—for example, the variation in the simulated median from replicate to replicate.
What Must Match for the Comparison to Be Meaningful?
The observed data and simulated data should be processed using the same time origin, time units, bin boundaries, quantile definitions, and strata. Simulations should represent the relevant design and observation process: this can include dose histories, covariates, scheduled sampling times, residual variability, and the handling of censored or missing records. Which features must be reproduced depends on the model and the purpose of the VPC, but any feature that affects inclusion or the plotted summaries needs to be made explicit.
A basic model-based VPC is conditional on the parameter values used for simulation and usually on the observed design. If parameter uncertainty is not propagated, the bands describe variation from simulated data at those parameter values; they do not include all uncertainty about the parameter estimates. Bands are typically pointwise: their stated coverage applies separately at each time bin and quantile, not to the probability that every plotted point lies inside every band at once.
Before interpreting a discrepancy, check that each plotted quantity has the same meaning on both sides. If observations are concentrations but simulated predictions are on a log scale, for example, summaries cannot be compared directly unless both are transformed consistently. Likewise, a median based on all records in a bin is not the same summary as a median based only on records with a particular dose history.
Identify the dependent variable, time origin, displayed quantiles, bin boundaries, and any strata in the intended VPC.
Check that simulated records reflect the relevant dosing, covariate, sampling, and observation rules used for the comparison.
Check counts, bin assignments, transformations, quantile definitions, and stratum labels before comparing plotted lines.
Assess where observed summaries fall relative to simulation-based bands, remembering that an isolated pointwise departure is not by itself proof of an implementation error.
Worked Examples
Worked Example: Checking a Median Against Its Simulation Band
Suppose 20 hypothetical observed concentrations (in mg/L) from one time bin, sorted from smallest to largest, are 4.1, 4.5, 4.8, 5.1, 5.4, 5.8, 6.0, 6.4, 7.0, 7.5, 7.8, 8.2, 8.7, 9.1, 9.8, 10.4, 11.2, 11.8, 12.6, and 14.0. For this example, define the sample median for an even sample size as the average of the two central observations.
The observed median is therefore
Assume 1,000 replicate datasets were simulated using the same bin and observation design. The 2.5th and 97.5th percentiles of the 1,000 simulated medians are 6.9 and 8.4 mg/L, respectively. The observed median, 7.65 mg/L, lies inside this pointwise 95% prediction band because \(6.9 \leq 7.65 \leq 8.4\).
This says the observed median is compatible with the range of medians produced by these simulations at this bin. It does not prove the model is correct: other bins, other quantiles, or implementation checks may reveal problems. Also, this median band is not an interval for the true population median and is not a confidence interval for a model parameter.
Worked Example: Finding a Time-Origin Mismatch
In a hypothetical study with doses every 12 hours, the intended VPC uses time since the most recent dose and bins of 0–4, 4–8, and 8–12 hours. The observed records include samples taken after the second dose. A review of the plotting code finds that observed samples are assigned time since the most recent dose, but simulated samples are assigned time since the first dose.
A sample taken 2 hours after the second dose has time since most recent dose \(2\) hours, but time since first dose \(12+2=14\) hours. Under the intended bins, the observed record belongs in the 0–4 hour bin, whereas the simulated record, if binned by the incorrect time variable, falls beyond the final bin.
Suppose the 100 observed samples are distributed across the intended bins as 36, 31, and 33. If the same records in the simulation output are instead counted using time since first dose, records collected after 12 hours are not represented in those bins. A difference in the plotted quantiles could then reflect which records were included, not a failure of the model to reproduce concentration behavior.
The review should trace the time variable from the source records through simulation output and bin assignment. Once both datasets use time since the most recent dose, the analyst can regenerate the summaries and make the model comparison. The observed counts alone do not determine the corrected simulated counts; those must be recalculated from the simulated records.
Worked Example: Detecting Reversed Stratum Labels
A hypothetical VPC separates records by renal-function category. The specified rule is “impaired” for estimated glomerular filtration rate (eGFR) below 60 mL/min/1.73 m2, and “not impaired” for eGFR of 60 or higher. A check of 40 observed records finds 12 below 60 and 28 at or above 60.
The plot’s count summary reports 28 records in “impaired” and 12 in “not impaired.” Those counts are the reverse of the counts implied by the stated rule. The arithmetic check is direct: \(12+28=40\), so no records are missing from this two-category count, but the displayed labels appear to be swapped.
Before interpreting either stratum’s quantile lines, check the category-generation code and the mapping of category values to legend labels. If the categories are reversed, relabeling the legend alone may not be enough: confirm which records were used to calculate each observed summary and each simulated band. The VPC should be regenerated or relabeled only after the underlying membership is verified.
Why the Bands Have Their Interpretation
The simulation band is based on the distribution of a summary across replicated datasets. Its calibration can be understood by imagining that the model and simulation design are correct and that the observed dataset and simulated replicates are generated under the same conditions. For one fixed bin and one fixed quantile, the observed summary and the \(R\) simulated summaries are then exchangeable: before seeing their values, no one of the \(R+1\) summaries is systematically more likely to be high or low than another.
If the summaries have distinct values, the rank of the observed summary among these \(R+1\) values is equally likely to be any rank from 1 to \(R+1\). A central 95% band formed from the simulated summaries will therefore contain the observed summary approximately 95% of the time, subject to the chosen empirical percentile convention and finite \(R\). Ties and finite-simulation percentile rules can affect the exact coverage.
This justification depends on the shared generating conditions. If the simulations use a different time origin or observation process, the observed summary and simulated summaries are not exchangeable under that setup. A band crossing or a point outside it then cannot be interpreted as a clean model check until the mismatch is resolved. Even when the setup is correct, pointwise bands do not give 95% simultaneous coverage across all bins and all displayed quantiles.
Diagnostics, Alternatives, and Limitations
Useful implementation checks include comparing observed and simulated record counts by bin and stratum, confirming the time units and time origin, checking that bin edges include boundary observations consistently, and verifying that the same transformation and quantile method are applied throughout. A sudden absence of simulated records in a bin, an unexpectedly reversed stratum count, or a shift that begins exactly at a dose or bin boundary can point to a data-processing or configuration discrepancy. These are clues to investigate, not automatic diagnoses.
A quantile outside its prediction band may be a genuine indication that the model does not reproduce that aspect of the data. It may also occur by chance under a correctly specified model, particularly when many bins and quantiles are inspected. Conversely, observed summaries inside the bands do not establish that every model component is correct. Check the construction first, then interpret the pattern in context. The earlier material on residual diagnostics and individual and population prediction diagnostics addresses complementary questions; a VPC should not be treated as a replacement for those checks.
| Approach | What it compares | Review implication |
|---|---|---|
| Visual predictive check | Observed binned summaries against replicate-based prediction bands | Useful for distributional behavior over time, provided records and summaries are aligned |
| Residual diagnostics | Residual-based summaries against their expected behavior | Can highlight patterns not shown by a small set of VPC quantiles |
| Individual or population prediction diagnostics | Predictions and observations at individual or population level | Offers a different view of predictive behavior; it does not verify VPC binning or strata |
A VPC compresses data into a few summaries and can conceal behavior between displayed quantiles or within bins. Sparse bins make tail quantiles particularly unstable, while wide bins can blend distinct time-dependent behavior. A basic VPC with fixed parameter estimates also omits parameter uncertainty unless that uncertainty is explicitly included in the simulation procedure. These limitations make construction checks and careful, appropriately qualified interpretation essential.
Common Mistake
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- An observed median is 5.2 mg/L and the corresponding simulation band is 4.6–5.8 mg/L. What does this comparison show, and what does it not show?
- Observed records are binned by hours since the most recent dose, but simulated records are binned by hours since the first dose. Name the first review action.
- A two-stratum VPC has 15 records below a defined cutoff and 25 at or above it. The plot labels the strata with counts 25 and 15, respectively. What should be checked before interpreting the stratum lines?
- Why does a pointwise 95% prediction band not imply a 95% chance that all plotted quantiles in all bins lie inside their bands?
Solutions. (1) The observed median is inside the band, so it is compatible with the simulated median range at this bin; this does not prove the model or its implementation is correct. (2) Reconcile the time origin used for both datasets, then recompute bin membership and summaries using the intended time variable. (3) Check the category rule, record membership, and mapping of category values to labels for both observed and simulated data; the reversed counts suggest a label or coding mismatch. (4) Pointwise coverage applies to each summary at each bin separately. The summaries across bins and quantiles are dependent, and pointwise bands do not provide simultaneous coverage for the whole plot.