Reading Predictive Checks Without Overclaiming
A predictive check compares observed data with data simulated from a fitted population pharmacokinetic model. Its purpose is to reveal discrepancies that matter for the intended use of the model—not to turn every visible difference into a rejection. The same display can look reassuring at a glance while concealing poor individual trajectories, or look alarming because one noisy summary falls just outside an envelope.
Interpretation depends on what was simulated, what statistic was summarized, and how much information the observations contain. A visual prediction check (VPC) typically compares observed concentration summaries within time bins with corresponding summaries from simulated data. NPDE-based checks examine prediction discrepancies on a standardized scale. Neither kind of display should be interpreted without attending to its construction and limitations.
A central 90% envelope for a binned median and a 90% prediction interval for an individual concentration answer different questions. The former concerns how much the median summary might vary across replicated data sets. The latter concerns the range of individual observations predicted under the model. Confusing these quantities can make an ordinary-looking envelope crossing seem more meaningful—or less meaningful—than it is.
Isolated Envelope Crossings
An observed summary outside an envelope is a signal to investigate, not a stand-alone verdict. If an envelope is central 90%, then, under a well-calibrated simulation procedure, a simulated summary has about a 10% chance of falling outside it at a particular bin. With many bins, occasional crossings are therefore unsurprising. This reasoning applies to the expected number of crossings even when bins are dependent; the exact probability of a particular pattern of crossings does depend on their dependence.
The derivation is direct. Let \(I_j=1\) when the summary at bin \(j\) lies outside its envelope, and \(I_j=0\) otherwise. If each bin has marginal crossing probability 0.10, then \(E(I_j)=0.10\). For \(m\) bins, the total number of crossings is \(C=\sum_{j=1}^{m} I_j\), so linearity of expectation gives:
This does not say that the crossings are independent or that \(C\) follows a binomial distribution. It says that one should not treat each crossing as a separate, definitive failure. A run of related deviations, a coherent trend, or a large and clinically important discrepancy deserves more attention than one small excursion. The width and interpretation of the envelope also depend on the simulation design, the statistic displayed, and whether relevant parameter uncertainty was included.
Worked Example: One Crossing Among Many Bins
Suppose a hypothetical VPC displays the observed median concentration in 12 time bins against a central 90% simulation envelope for the median. Eleven observed medians fall inside their envelopes; the median in one bin is just above the upper limit.
If each bin has a marginal 0.10 crossing probability under the simulation procedure, the expected number of crossings is \(12(0.10)=1.2\). Observing one crossing is therefore not, by itself, surprising evidence against the model. This calculation does not give the probability of exactly one crossing: the summaries across time may be dependent, and their joint distribution is not specified.
The next step is to inspect the size and pattern of the discrepancy. Is it a slight excursion with no adjacent deviation, or does it sit within a sustained upward trend? Does it occur in a region with adequate data, and does it appear in another relevant summary, such as a different concentration percentile? The crossing motivates these questions; it does not answer them.
Bin Occupancy and Unstable Summaries
Time bins are summaries of the observations assigned to each interval, not equally informative units. A bin with few observations can have a noisy empirical median or percentile. If the bins have different occupancy, a jagged VPC may partly reflect changing precision rather than a time-dependent model failure. Sparse sampling, censoring, and uneven visit schedules can make this especially important.
Before interpreting a binned summary, inspect the number of observations contributing to each bin and, where relevant, how those observations are distributed across subjects. A bin with many samples from a few individuals may not represent the population as broadly as its raw count suggests. Binning choices also matter: changing a boundary can move observations between bins and alter the summaries.
Worked Example: A Median Driven by Sparse Occupancy
Consider a hypothetical time bin containing four observed concentrations, in mg/L: 8, 9, 10, and 30. Its sample median is the mean of the two middle ordered values:
Now suppose one observation changes from 10 to 25 mg/L, with the other three unchanged. The ordered values are 8, 9, 25, and 30 mg/L, so the median becomes \((9+25)/2=17\) mg/L. A single changed value has moved the median by 7.5 mg/L. With only four observations, the central summary can be sensitive to which observations happen to fall in the bin.
This calculation does not prove that the bin is uninformative: 9.5 mg/L and 17 mg/L are both valid sample medians for their respective data. It shows why a visible change in a sparsely occupied bin needs to be interpreted alongside occupancy and simulation variability. If the observed median crosses an envelope there, do not treat that crossing as equivalent to a coherent discrepancy in well-populated bins.
Nonsignificance Is Not Evidence of Calibration
A formal test can summarize evidence against a stated null hypothesis, but a nonsignificant result does not establish that predictions are calibrated. It means that the data did not provide sufficient evidence to reject the null at the chosen significance level under the test’s assumptions. Weak information, high variability, or a small sample can make an important discrepancy difficult to detect.
For example, a test of whether the mean NPDE is zero addresses a particular feature of the discrepancy distribution. A nonsignificant result does not establish that its variance is one, that its tails are appropriate, or that discrepancies lack time or covariate structure. Those are different features and may require different summaries or checks. The assumptions of the test also matter: for the simple calculation below, NPDE values are treated as independent standard-normal observations under the null.
Worked Example: A Nonsignificant Mean-NPDE Test
In a hypothetical analysis, 16 independent NPDE values have sample mean 0.25. Test \(H_0:\mu_{\mathrm{NPDE}}=0\) against a two-sided alternative using a standard-normal reference, with the NPDE standard deviation treated as 1 under \(H_0\).
The standard error under the null is \(SE=1/\sqrt{16}=0.25\). The test statistic is:
The two-sided p-value is \(2P(Z\geq 1.00)\approx 0.317\), where \(Z\sim N(0,1)\). At significance level \(\alpha=0.05\), the result is not statistically significant. It does not show that the mean discrepancy is exactly zero or that the model is calibrated. The observed mean is 0.25 NPDE units, and this sample of 16 values does not provide strong evidence against the null using this test.
The conclusion is also conditional on the stated assumptions. If the NPDE values are dependent, or if the reference distribution is not appropriate, this simple standard error and p-value may not apply. The result should be read with other relevant checks, not used as a certificate of predictive adequacy.
Marginal Agreement Does Not Guarantee Individual Trajectories
A population-level summary can agree while subject-specific predictions are poor. Marginal predictive checks summarize the distribution of observations across the population at a time or within a bin. They can detect some systematic population-level discrepancies, but they do not necessarily show whether the model reproduces each person’s trajectory over time.
This distinction is especially important when within-subject patterns cancel in a pooled summary. A model may predict the population median well while underpredicting one group of individual profiles and overpredicting another. A marginal VPC is not a substitute for examining diagnostics that preserve information about subjects or trajectories. The appropriate complementary display depends on the question and sampling design.
Worked Example: Matching Marginal Means but Missing Trajectories
Suppose two hypothetical subjects have concentrations observed at three times, in mg/L. At every time, the model predicts 3 mg/L for each subject. Subject A’s observed profile is 2, 4, 6 mg/L; subject B’s is 4, 2, 0 mg/L.
At the first time, the observed mean is \((2+4)/2=3\) mg/L. At the second, it is \((4+2)/2=3\) mg/L. At the third, it is \((6+0)/2=3\) mg/L. The corresponding model-predicted mean is also \((3+3)/2=3\) mg/L at each time.
Thus, the marginal means match exactly at all three times. Yet neither observed trajectory is flat at 3 mg/L: subject A rises from 2 to 6 mg/L, while subject B falls from 4 to 0 mg/L. Agreement in this marginal summary does not demonstrate that individual trajectories are correctly reproduced. A reviewer should consider the intended predictive question and use subject-level or trajectory-sensitive diagnostics when individual prediction matters.
Choosing the Right Interpretation and Follow-Up
Different checks answer different questions. A VPC shows whether selected observed summaries resemble their simulated counterparts over time. NPDE summaries and formal distribution tests examine transformed prediction discrepancies and particular distributional features. Predictive-interval coverage checks whether observed values fall within specified intervals at an appropriate frequency. A subject-level trajectory display, by contrast, retains the within-person sequence that a pooled summary can obscure.
These approaches are complementary, not interchangeable. A formal test may be more reproducible than visual judgment, but it tests a defined feature and can be insensitive to other departures. A graphical check can reveal structure that a single test does not capture, but it is vulnerable to overreading isolated departures. A coverage result summarizes inclusion frequency but may hide where in time or which subjects misses occur. The question is not which check is universally best; it is what discrepancy each check can reveal.
Interpretation is most defensible when the simulation reproduces the study design and observation process relevant to the check, including sampling times and the handling of residual variability. If parameter uncertainty is omitted, an envelope may describe predictions conditional on the fitted parameter estimates and may be narrower than one incorporating that uncertainty. An envelope should not be treated as a universal threshold independent of how it was generated.
Determine whether the plot shows individual observations, a binned statistic, a prediction interval, or another summary. Do not interpret one as another.
Review bin occupancy, subject representation, sampling design, and simulation assumptions before judging a departure.
Consider magnitude, persistence, clinical relevance, and consistency across related summaries. A single crossing is a prompt for investigation, not a conclusion.
Use a complementary distributional, coverage, time-pattern, or subject-level check as appropriate. State what the results support and what they cannot establish.
Common Interpretation Errors
- “One envelope crossing proves model failure.” A central envelope allows some simulated summaries to fall outside it, and multiple bins create opportunities for isolated excursions. Examine the pattern, uncertainty, and context before drawing a conclusion.
- “The binned medians look smooth, so the model fits every subject.” A smooth marginal summary can conceal opposing individual deviations. Use trajectory-sensitive checks if the question concerns individual predictions.
- “The formal test is nonsignificant, so calibration is established.” A nonsignificant test is not proof of the null, and the test addresses only its specified feature. Consider information, assumptions, and other predictive dimensions.
- “Every bin is equally reliable.” Sparse occupancy can make an empirical summary unstable. Inspect the observations contributing to each bin before interpreting apparent changes.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A VPC has 20 bins and a central 90% envelope for a binned summary. What is the expected number of out-of-envelope summaries under the stated marginal calibration, and why does that not give the probability of exactly two crossings?
- Why should a crossing in a bin containing three observations be interpreted cautiously?
- A mean-NPDE test has \(p=0.24\) at \(\alpha=0.05\). What conclusion is justified, and what conclusion is not?
- Observed and predicted population means agree at every visit, but individual profiles show opposing trends. What does the marginal agreement establish, and what does it leave unresolved?
Solutions. (1) The expected number is \(20(0.10)=2\), by linearity of expectation. The probability of exactly two depends on the joint distribution of crossings, including dependence across bins; the marginal crossing rate alone does not determine it. (2) With only three observations, the empirical summary can be sensitive to individual values and may represent few subjects. Check occupancy and simulation variability before treating the crossing as a stable signal. (3) At the 0.05 level, the test does not reject its null. It does not prove the mean NPDE is zero, establish calibration, or rule out discrepancies in other features. (4) The means agree for the displayed population summary. This does not establish that individual trajectories are reproduced; the opposing subject-level trends remain a concern if individual prediction matters.