Why combine predictive checks?
A population pharmacokinetic model can reproduce an overall concentration distribution yet miss an important part of the profile: the absorption peak, late concentrations, or differences between dose groups. Predictive checks make those discrepancies visible by comparing observed concentrations with datasets simulated under the fitted model and the study’s sampling design. A stratified visual predictive check (VPC), normalized prediction distribution errors (NPDEs), and predictive-interval coverage summarize different aspects of that comparison. Their agreement is more informative than any one summary alone.
The analysis below uses entirely hypothetical data. Its purpose is to show how model predictions are assessed, not to establish a universal acceptance threshold. A check is only meaningful when the model, simulation procedure, observations, and strata correspond to the same estimand and study design.
Model and check assumptions
Consider an oral dose with first-order absorption into a gut compartment, followed by linear disposition in a central and a peripheral compartment. Let \(A_g\), \(A_c\), and \(A_p\) denote amounts in those compartments, and let \(C_c=A_c/V_c\) and \(C_p=A_p/V_p\) be their concentrations. The model is
Here \(k_a\) is the absorption rate constant, \(F\) is bioavailability, \(CL\) is clearance, \(Q\) is intercompartmental clearance, and \(V_c\) and \(V_p\) are the central and peripheral volumes. For a dose \(D\), the initial gut amount is \(A_g(0)=D\), with the other amounts set by the dosing and initial-condition specification. The prediction for a measured plasma concentration is based on \(C_c\). A population model may allow parameters to vary between individuals; for example, a positive parameter \(\theta_i\) can be represented by \(\theta_i=\theta_{\mathrm{pop}}\exp(\eta_i)\), where \(\eta_i\) is a subject-level random effect. Simulated observations must also include the fitted residual-error model.
The checks assume that the fitted model and simulation code represent the intended dosing, sampling times, covariates, between-subject variability, residual error, and any handling of concentrations below quantification limits. Simulated datasets should preserve the observed design—for example, each subject’s dose and planned sampling times. If uncertainty in estimated population parameters is part of the predictive target, it should be represented in the simulation procedure; otherwise the check is conditional on the fitted estimates.
For a VPC, the simulation envelope around a quantile describes how that quantile varies across simulated datasets. It is not an interval intended to contain a particular individual concentration. By contrast, an individual 90% predictive interval is constructed to contain 90% of future observations under the specified predictive model and conditions. The earlier material on Choosing Between VPC and NPDE Summaries distinguishes their roles; here they are applied together.
Worked analysis: stratified VPC
Worked Example: Checking the Median and Tails by Dose
Suppose 120 hypothetical participants receive either 100 mg or 200 mg orally, with concentrations measured across the post-dose profile. Investigators create 500 simulated datasets under the fitted two-compartment model and use the same doses and sampling times as observed. Within each dose group and time bin, they calculate the observed median and 5th and 95th percentiles. They also calculate those summaries in each simulated dataset. The table reports selected results; the “median envelope” is the central 90% of the 500 simulated medians, not an individual prediction interval.
| Dose | Time bin (h) | Observed median (mg/L) | 90% envelope for simulated median (mg/L) | Observed 5th–95th percentiles (mg/L) | Simulated median 5th and 95th percentiles (mg/L) |
|---|---|---|---|---|---|
| 100 mg | 0.5–1 | 2.40 | 2.10–2.90 | 0.80–4.90 | 0.70, 5.10 |
| 100 mg | 8–12 | 0.72 | 0.62–0.82 | 0.22–1.55 | 0.20, 1.60 |
| 200 mg | 0.5–1 | 4.90 | 4.40–5.60 | 1.60–9.80 | 1.40, 10.20 |
| 200 mg | 8–12 | 1.55 | 1.25–1.48 | 0.52–2.80 | 0.42, 2.60 |
At 100 mg, both observed medians fall within their simulation envelopes: \(2.10<2.40<2.90\) mg/L early and \(0.62<0.72<0.82\) mg/L late. The observed tail quantiles are also close to the simulated tail summaries in these selected bins. At 200 mg, the early median \(4.90\) mg/L is within its envelope, but the late median \(1.55\) mg/L exceeds the upper envelope, \(1.48\) mg/L. The late observed 95th percentile, \(2.80\) mg/L, is also higher than the simulated median 95th percentile, \(2.60\) mg/L. This points to a possible late-time underprediction in the higher-dose group, though one bin alone does not establish a cause.
The comparison is interpretable only if the bins contain enough observations and the simulated summaries use the same grouping and binning. A sparse final bin can yield unstable observed quantiles. Binning also blurs within-bin time patterns, so the time boundaries should be chosen in advance or checked for sensitivity. A coherent departure across adjacent bins is generally more persuasive than an isolated point.
Worked analysis: NPDE time pattern
Worked Example: Detecting a Late Prediction Discrepancy
For the same hypothetical analysis, suppose decorrelated NPDEs are plotted against time and summarized in four post-dose bins. Each bin has 20 NPDEs. Their means are \(-0.10\), \(0.05\), \(0.40\), and \(0.75\), respectively. Under a correctly specified predictive model and appropriate NPDE calculation, the reference distribution is approximately standard normal, with mean 0 and standard deviation 1. The approximation and finite-sample variability matter; the reference is not a claim that every small sample must look exactly normal.
With equal bin sizes, the overall mean is
The overall mean is modestly positive, but the time pattern is more informative: the mean rises from near zero early to \(0.75\) late. A positive NPDE indicates that an observation is high relative to its predictive distribution. In this example, the late pattern is consistent with underprediction at later times, in line with the high-dose late VPC departure. It does not prove that the elimination phase is the cause: dose-related differences, covariates, sampling, or other model features could also contribute.
Inspect the NPDE-versus-time plot as well as a distribution summary. Look for systematic trends, clusters, changing spread, and extreme points, while accounting for repeated measurements and the NPDE construction. A formal distribution test or a single overall mean can miss a localized time pattern. Conversely, an apparent trend based on a few observations may be sampling noise. The earlier material on NPDE distributions and time-pattern checks provides the relevant reference interpretation.
Worked analysis: predictive-interval coverage
Worked Example: Calculating Coverage by Time Bin
Suppose each observed concentration is compared with its model-based 90% predictive interval at the corresponding subject, dose, and time. In three time bins, the numbers inside the intervals are 19 of 20 early observations, 16 of 20 middle observations, and 13 of 20 late observations. The observed coverage proportions are
Overall, \(19+16+13=48\) of \(20+20+20=60\) observations are covered, so
The target is 0.90 under the specified predictive-interval construction, but the observed overall coverage is 0.800, with especially low coverage late. This pattern is consistent with the late NPDE and VPC findings. It does not mean the model has a known 10-percentage-point error: coverage is estimated from a finite sample, and observations from the same participant may be dependent. In particular, treating all 60 observations as independent would overstate the effective information if within-subject dependence is substantial.
Coverage results depend on interval definition. A central 90% interval has 5% in each tail when the predictive distribution is continuous and calibrated; a one-sided bound answers a different question. State whether intervals are individual predictive intervals, whether parameter uncertainty is included, and how censored observations are handled. Also inspect which observations fall outside the interval and whether misses are concentrated in time, dose, or one tail; the total proportion alone loses that information.
Putting the evidence together
The three summaries answer complementary questions. A stratified VPC shows whether observed concentration quantiles track simulated quantiles across time and groups. NPDE plots show standardized discrepancies and can reveal structured patterns across time. Coverage quantifies how often observations fall within a stated predictive interval. In the worked data, the late high-dose VPC median is above its simulation envelope, NPDE means increase at later times, and late predictive-interval coverage is only 65%. Their agreement makes a late-time predictive discrepancy a reasonable follow-up target.
The evidence remains diagnostic, not causal. Before changing the structural model, check that data processing, dose histories, sampling times, units, strata, simulation settings, and residual-error assumptions are correct. Then investigate whether the discrepancy is present across adjacent bins and subjects, whether it is driven by a few extreme concentrations, and whether the same pattern appears under reasonable choices of bins and summaries.
A VPC is often the clearest choice for communicating concentration quantiles and group-specific calibration. NPDEs can give a standardized view of discrepancies, especially across time, but depend on a valid predictive distribution and appropriate treatment of repeated observations. Coverage offers a compact numerical summary, but can conceal where and in which tail failures occur. None replaces the others, and none can diagnose a model uniquely.
Common pitfall and limitations
Predictive checks can have limited power when sample sizes are small, sampling is sparse, or relevant strata contain few participants. Quantile estimates in the tails are especially unstable. An apparently good fit to the median does not guarantee adequate tail prediction, while occasional points outside a nominal interval are expected. Simulations also assess the model and the simulation assumptions together: a coding error or incorrect design replication can produce a misleading check. Report the simulation count, predictive target, stratification, and relevant uncertainty choices so that the comparison can be interpreted.
Specify which observations, doses, time points, strata, and sources of uncertainty the simulated datasets should represent.
Compare observed and simulated VPC quantiles within the same dose and time strata; then inspect NPDE patterns and individual-interval coverage.
Look for agreement across neighboring times, groups, tails, and diagnostics, while considering sample size and within-subject dependence.
Verify data and simulation implementation, then evaluate plausible model features without treating a predictive-check pattern as a unique explanation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A VPC’s observed median lies within the central 90% envelope of simulated medians. What does this establish, and what does it not establish?
- In a late-time stratum, 27 of 30 observations fall inside their 90% predictive intervals. Calculate coverage and describe it relative to the nominal target.
- Mean NPDEs are near zero early and increasingly positive late. What prediction pattern is suggested, and why is that not a unique diagnosis?
- Why should a predictive check preserve each participant’s dose and sampling schedule in its simulated datasets?
Solutions. (1) The observed median is compatible with the simulated distribution of medians for that stratum. It does not show that 90% of individual concentrations are covered; that requires individual predictive intervals and a separate coverage calculation. (2) Coverage is \(27/30=0.900\), or 90.0%, equal to the nominal target in this sample. This is descriptive agreement, not proof of calibration in other samples or strata. (3) Positive NPDEs mean observations tend to be high relative to predictions, suggesting late-time underprediction. Other model features, design or data issues could produce a similar pattern, so the diagnostic is not a unique causal explanation. (4) The observed design determines which predictions are compared and how summaries are distributed. Simulating the same doses and times makes the observed and simulated summaries comparable under the intended predictive target.