Why Use a Visual Predictive Check?
A PD model can produce plausible individual curves and still fail to reproduce the pattern of observations across the study. A visual predictive check (VPC) makes this question explicit: if the fitted model generated new data under the study’s design, would the observed time-course summaries look typical? It compares summaries of the observed continuous PD response with the corresponding summaries from simulated replicate data sets.
Unlike a plot of fitted values alone, a VPC includes expected variability. A curve from one simulated data set is not a prediction interval: it is only one possible realization. Repeating the simulation gives a distribution of summaries, from which a prediction band can be calculated.
Constructing the Predictive Bands
Let \(Y_{ij}\) denote the continuous PD response for subject \(i\) at observation time \(t_{ij}\). Let \(\hat{\theta}\) denote the fitted model parameters. In a conventional plug-in VPC, each replicate data set is simulated using \(\hat{\theta}\), while also simulating the individual-level variability and residual error specified by the model. The simulation should reproduce the design relevant to the check: the subjects, covariates, doses, observation times, and planned or observed sampling pattern. This makes the simulated summaries comparable to the observed summaries.
Choose time bins \(k=1,\ldots,K\), or use distinct observation times when that is appropriate. In each bin, calculate an observed summary such as the median or a selected response quantile. For replicate \(b=1,\ldots,B\), calculate the same summary using the simulated observations in that bin. Write these as \(S_{\mathrm{obs},k}\) and \(S_{bk}\), respectively. A central 90% pointwise prediction interval for the summary in bin \(k\) is formed from the 5th and 95th percentiles of the replicate summaries:
The observed summary \(S_{\mathrm{obs},k}\) is plotted against this interval, usually along the time axis. The same procedure can be applied to several summaries. For example, compare the median to assess location, and compare lower and upper response quantiles to assess spread. If the objective is to check the distribution of responses within each bin, first calculate each selected response quantile separately in the observed data and in every simulated replicate. Then calculate a prediction interval for each such quantile across replicates.
A prediction interval for a summary is not the same as a confidence interval for a model parameter. It describes variation in a summary that could arise in repeated data sets under the simulated model and design. In a plug-in VPC, that variation is conditional on \(\hat{\theta}\); it does not fully include uncertainty in the estimated parameters. Parameter uncertainty can be incorporated by drawing parameters from an appropriate uncertainty distribution or by using a bootstrap procedure, but the resulting check must state how that uncertainty was represented.
Why the Simulation-Based Interval Is Appropriate
The quantity of interest is a statistic calculated from a finite study, such as a bin-specific median. Even if the model were the data-generating process, that statistic would vary between replicated studies because of between-subject variability, residual error, and the finite number and timing of observations. The replicate summaries approximate this sampling distribution under the fitted model.
For example, sort the \(B\) simulated median summaries in a bin. The interval endpoints are selected from that ordered set (or estimated by an interpolation rule) at the chosen probabilities. With enough simulations, the central 90% interval will contain roughly 90% of simulated summaries, subject to discreteness and Monte Carlo error. Thus, a value outside the band signals that the observed summary is unusual relative to this particular predictive distribution—not that a specific alternative model is proved.
Worked Checks
Worked Example: A Bin-Specific Median Suggests Location Bias
Hypothetical data. In one time bin, 20 observed continuous PD responses, in response units, are \(8.9, 9.4, 9.7, 10.0, 10.2, 10.4, 10.6, 10.8, 11.0, 11.1, 11.3, 11.5, 11.7, 11.9, 12.1, 12.3, 12.5, 12.8, 13.0,\) and \(13.4\). The observed median is the mean of the 10th and 11th ordered values:
Suppose 100 replicate data sets are simulated under the fitted model, preserving the bin’s sample size and study design. After calculating the median for each replicate, the 5th and 95th ordered simulated medians are 9.6 and 10.8 response units, respectively. Using the nearest-rank empirical quantile convention, the 90% prediction interval is therefore \([9.6,10.8]\) response units. Since \(11.2>10.8\), the observed median lies above the interval.
This is evidence that the fitted model underpredicts the typical response in this bin, relative to its simulated variability. It does not identify the cause: baseline structure, time course, exposure-response behavior, or another model component could be relevant. The conclusion is local to this bin and should be assessed alongside the rest of the VPC and other diagnostics.
Worked Example: Detecting a Dispersion Mismatch
Hypothetical data. In a second bin, 20 observed responses, ordered from smallest to largest, are \(3.0, 4.2, 5.0, 6.1, 7.2, 8.0, 8.8, 9.3, 9.7, 9.9, 10.1, 10.3, 10.7, 11.2, 12.0, 13.1, 14.4, 15.8, 17.0,\) and \(18.5\) response units. Use the nearest-rank convention: the empirical \(p\) quantile is the ordered observation at rank \(\lceil pn\rceil\). Here \(n=20\), so the observed 10th percentile is at rank \(\lceil0.10(20)\rceil=2\), and the 90th percentile is at rank \(\lceil0.90(20)\rceil=18\):
Suppose the prediction intervals across 100 replicate 10th percentiles and 90th percentiles are \([7.0,8.2]\) and \([11.8,13.1]\) response units, respectively. The observed lower quantile is below its interval, and the observed upper quantile is above its interval. If the observed median is \(10.0\) response units and lies inside its own simulated interval of \([9.4,10.6]\), the central location is reasonably reproduced while the observed distribution is more dispersed than the model predicts.
The pattern is consistent with underprediction of response variability in this bin. It is not enough to conclude that the residual error alone is too small: the source might instead involve subject variability or another feature represented in the simulations. The VPC flags the discrepancy; model knowledge and further diagnostics are needed to investigate it.
Worked Example: Reading a Time-Course Pattern
Hypothetical data. A VPC compares observed bin medians with 90% prediction intervals from 100 simulated data sets. The results are:
| Time bin (hours) | Observed median (response units) | Simulated 90% interval (response units) |
|---|---|---|
| 0–2 | 1.0 | 0.8 to 1.2 |
| 2–4 | 1.8 | 1.1 to 1.6 |
| 4–6 | 1.9 | 1.2 to 1.7 |
In the first bin, \(0.8\leq1.0\leq1.2\), so the observed median is within its prediction interval. In the second and third bins, \(1.8>1.6\) and \(1.9>1.7\), so both observed medians are above their intervals. The two consecutive departures suggest that the model may increasingly underpredict the typical response over this part of the time course. This is more informative than treating each bin as an unrelated discrepancy, but it remains a diagnostic pattern rather than a formal test or proof of a particular mechanism.
Diagnostics, Choices, and Alternatives
A VPC is only as informative as the simulations and summaries behind it. Before interpreting a departure, check that the simulated replicates use the same analysis rules as the observed data. For example, if observations are missing in the study, simulations that assume complete sampling may produce the wrong precision for bin summaries. The simulation should also preserve the subject-level structure: repeated observations from one subject are correlated through the subject’s random effects and other modeled features. Treating all observations as independent when generating replicates can make prediction intervals too narrow or otherwise misrepresent their variability.
Binning is a practical compromise. Narrow bins preserve temporal detail but may contain few observations, making empirical quantiles unstable. Wide bins produce more stable summaries but can conceal short-lived departures or blur a changing response. State the bin boundaries and inspect the number of observed subjects and observations contributing to each bin. Consider whether bins align with the study’s sampling schedule and the rate at which the PD response changes.
The number of replicates also matters. With a small \(B\), the estimated endpoints of the prediction interval can be noisy. For \(B=100\), for example, a nearest-rank 5th percentile is determined by only the fifth ordered simulated summary. Increasing \(B\) can reduce this Monte Carlo variability. It does not repair a misspecified model or a simulation that fails to reproduce the design.
Finally, separate a single pointwise crossing from a sustained pattern. If many bins are examined, some summaries may fall outside nominal 90% pointwise intervals even when the model is adequate. The intervals are not a familywise-error-controlled test, and crossings should not be counted as though they were independent. Look for coherent departures across adjacent times or across related summaries, and assess them alongside residual plots and observed-predicted calibration checks.
| Approach | What it emphasizes | Main limitation |
|---|---|---|
| Visual predictive check | Whether simulated data reproduce selected time-binned summaries and their variability | Depends on simulation assumptions, summaries, and binning choices |
| Residual diagnostics | Patterns in observation-level discrepancies relative to model predictions | Residual behavior can be difficult to interpret when predictions and variability are complex |
| Observed-predicted plot | Calibration between observed responses and model predictions | Agreement in an overall relationship can obscure time-local distributional mismatch |
These approaches are complementary. A VPC gives a direct, design-aware comparison of observed and simulated data summaries; it does not replace residual analysis or explain the mechanism behind a discrepancy. A pooled VPC can also hide differences among subsets of subjects. The chosen summaries and bins should therefore be guided by the diagnostic question, while recognizing that a pooled check cannot establish that the model performs equally well for every covariate or subgroup.
Common Mistakes and Limitations
- Taking quantiles of one simulation. One simulated summary is a single value. Prediction-interval endpoints must be calculated across the \(B\) simulated summaries for that bin: \(Q_{0.05}(S_{1k},\ldots,S_{Bk})\) and \(Q_{0.95}(S_{1k},\ldots,S_{Bk})\).
- Calling the band a confidence interval for the model. A VPC band describes simulated data summaries under a model and design; it is not an interval estimate for a parameter.
- Reading every crossing as decisive evidence. Pointwise intervals do not provide simultaneous coverage across time. Chance crossings are possible, especially when many bins and summaries are inspected.
- Ignoring the simulation design. Incorrect sample sizes, sampling times, covariates, or repeated-measure structure can produce misleading bands, even if the fitted model is otherwise appropriate.
- Overinterpreting the cause. A median departure suggests a location mismatch, and tail departures suggest a spread mismatch, but neither uniquely identifies which model component needs revision.
A VPC is most useful when its purpose is stated in advance: which response summaries, which time bins, and which model-generated features are being checked. It provides an interpretable view of predictive adequacy for those choices, not a complete validation of every aspect of the model.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In a bin, an observed median is 5.7 response units. The 5th and 95th percentiles of the \(B\) simulated bin medians are 4.8 and 5.5. What does the comparison suggest, and what does it not establish?
- Why must a VPC simulate the study’s repeated observations and sampling pattern rather than generate independent response values at each time?
- Observed 10th and 90th response quantiles fall outside their respective simulated prediction intervals, while the median is inside its interval. What kind of mismatch is suggested?
- Why is a 90% pointwise interval at each of many time bins not a 90% simultaneous band for the entire time course?
Solutions. (1) Since \(5.7>5.5\), the observed median is above the simulated interval, suggesting that the model underpredicts the typical response in this bin relative to simulated variability. This alone does not prove a specific mechanism or establish that the model is inadequate in every time region. (2) Repeated observations from a subject are not generally independent; subject-level variability and the observation schedule affect the sampling distribution of bin summaries. Replicates must reflect those features for the bands to be comparable. (3) The tails being more extreme than predicted, with the median reproduced, suggests that the observed response distribution is more dispersed than the simulated distribution. It does not identify the source of the dispersion mismatch. (4) Each pointwise interval targets coverage for one bin-specific summary. The probability that all intervals cover simultaneously is not generally 90%; it depends on the joint behavior across bins and is usually lower than the individual pointwise coverage.