Constructing VPC Prediction Intervals

What a VPC Prediction Interval Describes

A visual predictive check (VPC) compares summaries of the observed data with the same summaries calculated from replicate data simulated under a fitted model. Within each time bin, these summaries might include the median or selected quantiles. A prediction interval shows how much a particular summary can vary across replicated data sets if the simulation model and design are appropriate.

The interval is for a summary, such as the median concentration in one bin, not for an individual concentration and not for a model parameter. Its width reflects variation in that summary across simulated data sets. It depends on the model, simulation procedure, number and timing of observations, bin definitions, and the number of individuals represented.

For example, a 95% interval for the median in one bin describes the central 95% of replicated medians under the specified simulation procedure. It is pointwise: its coverage refers to one chosen bin and one chosen summary. It does not, by itself, give 95% coverage simultaneously for every bin and every displayed quantile.

Assumptions and Calculation

The calculation is straightforward, but its interpretation depends on the simulation being aligned with the target predictive check. The simulated data should represent replicate data sets for the intended study design. In particular, use the same relevant observation times, bin boundaries, and summary definition for observed and simulated data. The simulation should also include the sources of variability intended by the check, such as between-subject variability and residual error when the target is a population-level predictive check.

A common plug-in procedure uses the fitted parameter estimates and generates new random effects and residual errors from the fitted model. The resulting interval is conditional on those parameter estimates. Parameter uncertainty is not automatically included merely because many data sets were simulated; it must be included by the simulation procedure if it is part of the intended predictive target. The distinction matters when estimates are imprecise.

For reproducibility, choose and state a quantile convention. In the examples here, empirical quantiles use the nearest-rank rule: for ordered values \(x_{(1)}\leq \cdots \leq x_{(m)}\), the empirical \(q\) quantile is \(x_{(\lceil mq\rceil)}\), where \(\lceil \cdot \rceil\) means round up to the next integer. Other software conventions may interpolate between adjacent ordered values, so results can differ slightly, especially with few replicates.

The ordered simulated summaries approximate the predictive distribution of the statistic. The interval endpoints are estimates of its lower and upper predictive quantiles. This construction is a direct Monte Carlo approximation: as the number of independent replicate data sets increases, the empirical distribution of their summaries generally becomes a more stable approximation to the simulation distribution.

Worked Calculations

Why the Procedure Works, and What to Check

Let \(T^*_{b,p}\) denote the summary that would be calculated from a new replicate data set under the specified model, design, and simulation scheme. If \(L\) and \(U\) are its true lower and upper \(\alpha/2\) predictive quantiles, then, for a continuous predictive distribution,

The approximation allows for endpoint conventions and possible discreteness. In practice, \(L\) and \(U\) are unknown. The simulated summaries act as draws from the predictive distribution, and their empirical quantiles estimate \(L\) and \(U\). Thus the interval has its stated predictive interpretation only relative to the assumed model and simulation procedure; it is not a model-free guarantee.

Several checks help identify a calculation that is going wrong:

• Check bin membership and design. Observed and simulated summaries must refer to the same time boundaries and intended observation schedule. A mismatch changes the distribution being compared.
• Check counts and empty bins. If a bin has no observations, its observed summary is undefined. If it has very few observations, its median or tail quantile can change sharply when one value changes. Do not silently treat an empty-bin summary as zero.
• Check the replicate summaries, not just the plotted endpoints. Recalculate selected medians or quantiles from a replicate to verify that the right observations and quantile convention were used.
• Check Monte Carlo stability. With too few simulations, tail endpoints depend heavily on a few ordered values. Repeating the simulation with more replicates can reveal whether the endpoints are stable.
• Check the simulation target. A narrow interval can result when relevant variability was omitted from simulation. Conversely, if the intended check is conditional on particular subject effects or design features, simulating a different target may produce an interval that answers the wrong question.

The empirical interval is often preferable to imposing a normal distribution on the simulated summaries: medians and tail quantiles can have asymmetric or otherwise non-normal sampling distributions, particularly in small or sparse bins. A normal-approximation interval may be a useful comparison when its assumptions are justified, but it does not replace checking the simulated distribution. A different alternative is to bootstrap the observed data to quantify uncertainty in the observed summary. That addresses sampling uncertainty around the observed summary; it is not the same as predicting the summary in new data sets under the fitted model.

Interpretation and Limitations

An observed summary inside its prediction interval is compatible with the simulated variability for that particular bin and summary. An observed summary outside the interval flags a discrepancy worth investigating; it is not, by itself, proof of model failure. The discrepancy may reflect model misspecification, an unrepresentative observed sample, sparse-bin instability, simulation error, or a mismatch in design or data processing.

Intervals are commonly examined for several bins and several summaries. Because each interval is pointwise, the chance that at least one observed summary falls outside its interval across a collection of checks can exceed the nominal tail probability for any single interval. These intervals should therefore be read as diagnostic comparisons, not as simultaneous coverage bands or independent hypothesis tests. Their interpretation also depends on the fitted model: when parameter uncertainty is not propagated, the intervals can understate predictive uncertainty if that uncertainty is substantial.

The introductory material on visual predictive checks establishes the role of comparing observed summaries with simulated summaries. The practical extension here is to preserve that comparison at the level of each bin-specific statistic, including its uncertainty across replicate data sets. Binning and sample size remain important limitations: a large number of simulated data sets cannot make an observed within-bin quantile precise when few observed values contribute to it.