Why Predictive-Check Reporting Needs Design Details
A predictive check asks whether data generated under a fitted population pharmacokinetic model resemble the observed data in features relevant to the analysis. The plot alone cannot answer that question for a reviewer. A useful report explains which participants and observations were represented, how the original sampling design was reproduced, what was simulated, and which observed summaries were compared with the simulations.
For a visual predictive check (VPC), the comparison is often made across time bins. Within each bin, the observed concentration distribution is summarized by selected quantiles, such as the 5th, 50th, and 95th percentiles. Replicated datasets simulated from the model provide corresponding quantiles and a reference range for each summary. The report should state both the comparison and its scope: a check at scheduled sampling times does not, by itself, demonstrate reliable predictions for every time, population, or dosing regimen.
State What Was Simulated and Compared
The simulation design should be sufficiently explicit for a reviewer to understand what “under the model” means. A typical VPC simulates concentrations for the observed participants, retaining their relevant covariates, dose histories, and observation times. It then generates individual variability and residual variability according to the final model. If the check instead uses a different population or sampling schedule, say so: that is a different predictive question.
Also report the model version and data set used, the number of simulated replicates, the time-bin definitions, the quantiles compared, and how any censoring or below-quantification observations were treated. State whether parameter estimates were held fixed at their fitted values or parameter uncertainty was incorporated. In the usual fixed-estimate check, the simulation reference is conditional on those estimates; it does not automatically represent uncertainty in the estimates themselves.
The comparison is meaningful only if the observed and simulated summaries use compatible rules. For example, bin boundaries, inclusion criteria, units, and handling of below-quantification data should be consistent. If binning or data handling differs, explain the difference and why the comparison remains interpretable. Sparse bins can make observed quantiles unstable, while very broad bins can conceal time-localized discrepancies. Give the number of observations or participants in bins when it helps readers assess that limitation.
Name the population, concentration observations, time range, bins, and quantiles that the check evaluates.
State what design features were retained, what model variability was simulated, the number of replicates, and whether parameter estimates were fixed or varied.
Describe agreement or discrepancy by location and quantile, then relate the finding to the model’s intended use without claiming more than the check supports.
How to Summarize the Comparison
Let \(q_{p,b}^{(r)}\) denote the \(p\)th concentration quantile in time bin \(b\) from simulated replicate \(r\). Across \(R\) replicates, the values \(q_{p,b}^{(1)},\ldots,q_{p,b}^{(R)}\) form a simulation reference for the \(p\)th quantile in that bin. A central 90% simulation interval can be formed from the 5th and 95th percentiles of those \(R\) values. The observed \(p\)th quantile in bin \(b\), written \(q_{p,b}^{\mathrm{obs}}\), is then compared with that range.
Here \(Q_a\) denotes the \(a\)th quantile across the simulated replicate summaries. If the observed summary falls outside the range, that is a potential discrepancy to describe and investigate, not a stand-alone proof that the model is wrong. A simulation interval for a median, for example, concerns the median across replicated datasets. It is not the range expected to contain 90% of individual concentrations; that would be a different quantity.
A short justification clarifies why this comparison is useful. If the fitted model and the specified design generate data that reproduce the observed feature, the observed feature should not be unusually far from the features generated in replicated datasets. The empirical interval provides a visual reference for that question. Its interpretation is conditional on the simulation procedure and summaries chosen. The check is not automatically a formal hypothesis test: several bins and quantiles may be inspected, and the intervals do not adjust for that visual multiplicity or for every source of model uncertainty.
Worked Example: Reporting a VPC With Agreement Across Quantiles
Suppose a hypothetical analysis includes 80 participants with concentration measurements in a 2-to-4-hour bin. A VPC uses 500 simulated replicates, retains the participants’ covariates and observation times, and simulates individual and residual variability under the fitted model. The observed 5th, 50th, and 95th percentiles are 1.2, 4.8, and 13.0 mg/L. Across replicates, the corresponding 90% simulation intervals are 0.9–1.6, 4.2–5.5, and 11.5–15.2 mg/L.
For this bin, the observed 5th percentile is within its reference range because \(0.9 \leq 1.2 \leq 1.6\) mg/L. The observed median is also within range: \(4.2 \leq 4.8 \leq 5.5\) mg/L. The observed 95th percentile is within range as well: \(11.5 \leq 13.0 \leq 15.2\) mg/L. Thus, all three observed quantiles fall within their respective simulation intervals in this bin.
A suitable report would say that the observed 5th, 50th, and 95th concentration percentiles in the 2-to-4-hour bin were within their 90% simulation intervals. It would identify the simulated design and variability as above, and avoid saying that the model “proved accurate”: this is agreement for the stated summaries and bin, not a guarantee of performance in other settings.
Worked Example: Describing a Lower-Tail Discrepancy
In a second hypothetical VPC, 60 observations fall in a 6-to-8-hour bin. The observed median is 3.6 mg/L, within its 90% simulation interval of 3.2–4.0 mg/L. The observed 5th percentile is 0.4 mg/L, below its simulation interval of 0.7–1.0 mg/L. The observed 95th percentile is 9.0 mg/L, within its interval of 8.2–9.8 mg/L.
The lower-tail discrepancy is \(0.7 - 0.4 = 0.3\) mg/L below the lower end of the simulation interval. The median and upper-tail summaries are inside their respective intervals. Therefore, the report should identify a lower-tail mismatch in this time bin, rather than characterize the whole concentration distribution as uniformly too high or too low.
This pattern may warrant checking relevant model and data features, such as the residual-error specification, observations near the quantification limit, or the consistency of data handling. The VPC alone does not identify the cause. The report should give the bin’s sample size and avoid attributing the discrepancy to a specific mechanism unless additional evidence supports that explanation.
Relate Discrepancies to Intended Use
The same mismatch can matter differently for different purposes. If a model is intended to support prediction of trough concentrations, a systematic discrepancy in bins around the trough is more directly relevant than one isolated at an unrelated time. If the intended use depends on the upper tail, an upper-quantile mismatch deserves specific attention even when the median agrees. State the relevant region, direction, and magnitude of the discrepancy, then explain what it may limit.
Worked Example: Interpreting a Trough-Time Pattern
Consider a hypothetical VPC for a model intended to predict concentrations immediately before the next dose in a particular regimen. In the 23-to-25-hour bin, the observed median is 2.1 mg/L, while the central 90% simulation interval for the median is 2.4–3.0 mg/L. The difference from the nearest boundary is \(2.4 - 2.1 = 0.3\) mg/L. In an earlier 7-to-9-hour bin, the observed median is 5.0 mg/L and its interval is 4.6–5.4 mg/L.
The report can state that the observed median was below its simulation interval in the trough-time bin, by 0.3 mg/L relative to the lower boundary, while the earlier-bin median was within its interval. Because the discrepancy occurs near the time relevant to the stated use, it raises a concern about predictive performance for that trough-time summary under this regimen. It does not establish the size or direction of error for every participant, nor does it alone determine whether a clinical decision would change.
Diagnostics, Alternatives, and Limitations
Interpret a predictive check cautiously when simulation and observation are not comparable. Warning signs include bins with very few observations, inconsistent handling of below-quantification data, incorrect dose or sampling-time records, or simulations that omit a source of variability present in the fitted model. A check can also look reassuring while hiding a problem: broad bins may average over a localized mismatch, and pooled summaries can conceal opposing patterns across covariate groups. When a particular covariate or subgroup matters to intended use, report a relevant stratified comparison if the data support it.
A prediction-corrected VPC is an alternative when differences in typical predictions across participants—for example, due to dose or body size—make pooled concentration quantiles difficult to interpret. It adjusts concentrations using model predictions before comparing quantiles. Such a report must identify the correction and its rationale; a corrected plot is not interchangeable with an uncorrected concentration-scale VPC. Individual observed-versus-predicted plots answer a different question about individual-level fit, while a VPC focuses on distributions of observations within design-defined groups. No single display establishes that a model is suitable for every purpose.
A common mistake is to write, “The 90% prediction interval contains 90% of individual concentrations, so the model fits.” In a VPC, the 90% interval may instead describe simulated values of a selected summary, such as the median. Even when an interval is constructed for individual predictions, its meaning depends on the exact procedure and conditioning assumptions. Name the quantity the interval describes. Also avoid treating one summary inside its interval as conclusive evidence of adequacy, or one summary outside as a definitive rejection. Report the pattern, its uncertainty and context, and the consequences for the intended use.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In a VPC, the observed median is inside its 90% simulation interval. What quantity does that interval describe, and what does it not establish?
- A bin’s observed 5th percentile is 0.6 mg/L and the simulation interval for that percentile is 0.8–1.1 mg/L. State the discrepancy precisely.
- Why should a report state whether parameter estimates were fixed or varied in the simulations?
- A pooled VPC agrees overall, but the model is intended for a covariate-defined subgroup. What limitation should be considered?
Solutions. (1) It describes the range of simulated values for the median summary under the stated simulation procedure. It does not establish that 90% of individual concentrations lie in that interval or prove adequate predictions for all settings. (2) The observed 5th percentile is 0.2 mg/L below the lower boundary of its simulation interval; this flags a lower-tail discrepancy for that bin, not its cause. (3) With fixed estimates, the reference is conditional on those estimates and does not automatically include their uncertainty; varying estimates changes the scope of the predictive comparison and should be disclosed. (4) Pooling can conceal subgroup-specific mismatch. A relevant stratified check may be needed, if data support it, before drawing conclusions about that subgroup’s use.