Two Summaries, Different Predictive Questions
A visual predictive check (VPC) and normalized prediction distribution errors (NPDEs) both compare observed data with predictions from a population PK model, but they summarize different features. A VPC asks whether selected observed concentration quantiles over time are consistent with the corresponding quantiles predicted under the model. NPDE summaries ask whether prediction discrepancies, standardized against the predictive distribution and decorrelated within profiles, behave as expected overall or in selected groups.
The choice should follow the question and the design. When the key concern is calibration of concentration levels across time, VPC quantiles are often more direct. When the concern is the overall distribution of predictive discrepancies, or systematic patterns across observations or covariates, NPDE summaries can be more useful. Neither summary is a universal substitute for the other; their conclusions depend on what is retained by the summarization and on whether the predictive simulations reflect the study design and relevant uncertainty.
The model simulations must represent the intended predictive target. In particular, simulated datasets should use a design comparable to the observed study: sampling times, dosing, relevant covariates, and data-handling rules should be aligned. Parameter uncertainty should be included when the predictive question calls for it; otherwise, both summaries can appear more precise than warranted. These checks evaluate predictive compatibility, not whether a model is uniquely true or whether any discrepancy has a particular cause.
What Each Summary Retains
A VPC preserves an important feature of the time-course question: the observed concentration distribution is displayed against time. Binning time makes this possible when observations do not share identical times. In each bin, a typical VPC compares observed quantiles with the distribution of the corresponding quantiles across simulated datasets. The median can reveal location bias, while lower and upper quantiles can reveal differences in spread or tail calibration. The choice of quantiles should match the scientific concern and the amount of information available.
A simulation interval around a VPC quantile is an interval for that summary statistic under repeated datasets generated by the model. It is not an interval expected to contain a specified proportion of individual concentrations. For example, a 90% simulation interval for the median is about the variation in the median across simulated datasets, not a 90% prediction interval for individual concentrations.
NPDEs instead put discrepancies on a common scale relative to the predictive distribution. For subject \(i\), let \(\mathbf{y}_i\) be the vector of observed concentrations and let \(\mathbf{Y}_i\) denote a simulated predictive vector at the subject’s design points. The predictive distribution of \(\mathbf{Y}_i\) accounts for the modeled variability. A decorrelation step uses the simulated within-profile dependence, and a probability transformation maps the resulting predictive discrepancies to a standard normal scale. In an ideal calibrated setting, the resulting NPDEs have mean near zero and variance near one, and their distribution is approximately standard normal.
This is a schematic description, not a recipe for computing NPDEs. Their calculation requires suitable simulations and a stable estimate of the predictive dependence structure; the earlier material on normalized prediction distribution errors and decorrelation develops those steps. Summaries might include a histogram or normal quantile plot, mean and variance, or NPDE plotted against time or covariates. A single pooled histogram can look satisfactory even when errors at different times have opposing biases, so stratified or pattern-based summaries are often essential.
A useful selection rule is to ask what discrepancy would matter most in context. If it would be important to know that concentrations are systematically too high during the elimination phase, a time-resolved VPC or time-pattern NPDE display addresses that concern. If the concern is whether the predictive distribution is too narrow overall, NPDE spread and distributional displays are relevant, while VPC tail quantiles offer a complementary view in time. The answer is not always “choose one”: complementary summaries are appropriate when they answer distinct, prespecified questions.
Worked Comparisons
Worked Example: A Time-Specific Median Discrepancy
Suppose a hypothetical study has 40 observed concentrations in a prespecified late-time bin. Their median is 8.2 mg/L. Across 500 simulated datasets using the study’s sampling design, the median simulated median is 7.5 mg/L, and the central 90% simulation interval for the median is 6.9 to 8.1 mg/L. Is the observed median compatible with this VPC summary?
The observed-minus-simulated-median difference is \(8.2-7.5=0.7\) mg/L. The observed median, 8.2 mg/L, is 0.1 mg/L above the upper endpoint of the simulation interval, 8.1 mg/L. Thus this bin shows a discrepancy in the observed median relative to the simulated distribution of medians. The result is a localized warning worth investigating, not proof that the model is wrong: Monte Carlo error, binning choices, sampling variability, and other predictive checks still matter.
The interval is for the median statistic. It does not mean that 90% of individual concentrations should lie between 6.9 and 8.1 mg/L. An individual prediction interval would be a different quantity.
Worked Example: NPDE Location and Spread
For a small hypothetical diagnostic illustration, consider eight NPDE values: \(-1.4,-0.8,-0.3,0.0,0.2,0.5,0.7,1.1\). Calculate their mean and sample variance, then describe what they can and cannot establish.
The values sum to zero, so their mean is \(\bar{x}=0/8=0\). Their squared deviations from zero sum to \(1.96+0.64+0.09+0+0.04+0.25+0.49+1.21=4.68\). The sample variance is \(s^2=4.68/(8-1)=0.669\), and the sample standard deviation is \(s=\sqrt{0.669}=0.818\), to three significant figures.
The mean is centered at the target of zero, while this small sample’s spread is below the standard normal target variance of one. With only eight values, that difference is highly uncertain and is not enough to declare under-dispersion. A larger dataset should be examined with distributional displays and relevant stratification. This pooled calculation also cannot rule out time-specific bias or residual dependence.
Worked Example: Choosing a Summary for Sparse Sampling
In a hypothetical study, 30 subjects each contribute one concentration, with samples taken at different times from 1 to 24 hours. The scientific question is whether the model predicts the overall distribution of concentrations and whether discrepancies vary with sampling time. There are only three observations in each of several narrow time bins. Which summary is more informative?
A conventional time-binned VPC would have very few observations per bin. Its observed quantiles would be unstable, especially tail quantiles, and small shifts in bin boundaries could materially change them. Treating the resulting quantile curves as precise evidence would be unjustified.
An NPDE display against time can use each observation while standardizing discrepancies to the predictive distribution. It is therefore a reasonable primary summary for the stated distributional question, with time-pattern diagnostics to look for systematic changes. The design has one observation per subject, so there is no within-subject profile to decorrelate; that avoids one source of covariance-estimation difficulty, but does not remove uncertainty or guarantee adequate power. If the central question were instead detailed calibration at particular phases of the concentration-time curve, the sparse design might simply not support a reliable answer. The best summary cannot replace missing design information.
Assumptions, Diagnostics, and Failure Modes
Both approaches rely on predictive simulations that represent the data-generating and observation processes relevant to the question. A check can be misleading if simulations use the wrong sampling schedule, omit relevant variability, or handle concentrations differently from the observed data. Whether parameter uncertainty is included also changes the predictive distribution. State the target clearly: for example, prediction for new subjects under a specified design, or replication of the observed study under fitted population parameters.
For a VPC, inspect how many observations contribute to each time bin, whether bins combine meaningfully comparable times, and whether there are enough simulated datasets to estimate quantile intervals stably. Tail quantiles are especially sensitive to small counts. Check whether patterns persist under defensible alternative binning, and inspect the observed and simulated data handling for limits of quantification. A VPC with smooth-looking bands can still conceal poor calibration if the bins are too broad or if sparse tail information is obscured.
For NPDEs, inspect more than the pooled mean. Consider spread and distributional shape, plots against time and covariates, and whether conclusions are driven by a small subset of subjects. When subjects have multiple observations, the decorrelation calculation depends on the predictive dependence structure. Sparse or highly correlated profiles can make that structure difficult to estimate; numerical instability or extreme transformed values should prompt checks of the calculation and simulations rather than automatic interpretation as model failure. Formal distribution tests may supplement displays, but a non-significant result does not establish adequate predictive performance.
A concise comparison helps keep the purpose of each summary clear.
| Question or feature | VPC quantiles | NPDE summaries |
|---|---|---|
| Primary view | Observed versus simulated concentration quantiles over time | Standardized predictive discrepancies overall or by predictors |
| Most direct use | Time-course calibration and selected distributional quantiles | Location, spread, shape, and patterns in normalized errors |
| Design pressure point | Few observations per time bin make quantiles unstable | Repeated profiles require reliable decorrelation and predictive simulations |
| Common blind spot | Pooling in bins can hide within-bin time patterns | A pooled distribution can hide opposing or localized patterns |
| Useful complement | NPDE plots can reveal patterns not apparent in quantile bands | VPCs show concentration-scale calibration at particular times |
A practical choice can be made in three steps.
Specify whether the concern is time-specific concentration calibration, distributional shape, overall error spread, or patterns by time or covariate.
For VPCs, assess observations per bin and quantile stability. For NPDEs, assess profile structure and whether the predictive dependence calculation is reliable.
Read VPC intervals as uncertainty in quantile summaries and NPDE displays as evidence about standardized discrepancies; follow up with diagnostics targeted to any apparent pattern.
Interpretation and Common Pitfall
Neither a VPC nor an NPDE summary provides a single pass-or-fail verdict. A visible discrepancy may reflect model misspecification, limited data, simulation uncertainty, or a design feature that makes the chosen summary unstable. Conversely, a close match in one summary does not establish that the model predicts every relevant feature adequately. The strongest assessment connects the display to a prespecified predictive question, checks the assumptions behind its simulations and calculations, and considers complementary evidence.
Common mistake: “The observed median is inside the 90% VPC band, so 90% of individual concentrations are covered and the model is validated.” This confuses a simulation interval for a quantile statistic with an individual prediction interval, and it overstates what one comparison can establish. The correction is to identify exactly what the interval summarizes, assess the other time bins and relevant distributional features, and describe the result as predictive compatibility for the examined summary—not validation in general.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A VPC shows the observed 10th percentile just outside its simulation interval in one early-time bin, while other bins agree. What is the direct conclusion, and what should be checked next?
- Why is a 90% simulation interval for the VPC median not a 90% prediction interval for individual concentrations?
- A pooled NPDE histogram is centered near zero, but NPDEs are mostly positive early and mostly negative late. What does the pooled result miss?
- A study has two observations per subject and highly correlated predicted concentrations. Which NPDE calculation feature deserves particular scrutiny, and why?
Solutions. (1) The observed 10th percentile is unusual relative to the model’s simulated sampling distribution for that bin; it is a localized predictive discrepancy, not proof of misspecification. Check counts, binning, simulation uncertainty, and whether the pattern recurs or has a plausible time-course explanation. (2) The interval describes variation in the estimated median across simulated datasets. An individual prediction interval describes the predictive range for an individual concentration; these are different random quantities. (3) Opposing time-specific biases can cancel in the pooled mean and be obscured in the pooled distribution. Plot or summarize NPDEs by time and consider a time-resolved VPC if the design supports it. (4) Scrutinize the decorrelation and estimation of within-profile predictive dependence. Strong correlation can make the transformation sensitive to errors in the estimated covariance structure, so inspect numerical stability and whether simulations adequately represent that dependence.