When Evaluation Results Point in Different Directions
A population pharmacokinetic (PK) model can appear adequate in familiar fit plots and still perform poorly on a predictive check aimed at a particular feature of the data. This is not necessarily a contradiction. The evaluations ask different questions: a plot may show that predictions track the overall concentration pattern, while a predictive check may show that the model does not reproduce the distribution, variability, or subgroup pattern that matters for a particular use.
The reporting task is not to choose the most reassuring result or to declare the model either “good” or “bad.” It is to state what each evaluation examined, describe the discrepancy precisely, consider its scope, and limit the conclusions and proposed uses accordingly. The discussion below uses hypothetical data throughout.
Neither label identifies a single universal diagnostic. The interpretation depends on what was plotted or simulated, how predictions were constructed, and which observations were included. For example, individual predictions use information about the individual’s observations to estimate that individual’s random effects. They can track the data closely without demonstrating that the model predicts a new individual well. Population predictions do not condition on an individual’s observed concentrations in the same way. A comparison of the two can therefore reveal different aspects of model adequacy.
Simulation-based checks also require a defined simulation design. The simulated datasets should reflect the features relevant to the question, such as dose, sampling times, covariates, and the residual-error model. A check generated under a design that omits an important feature may not evaluate the intended prediction. Even with an appropriate design, a predictive check is evidence about selected features, not proof that the model will predict every future dataset correctly.
How to Read a Discrepancy
Suppose observed-versus-individual-predicted concentrations are reasonably close to the identity line, and conditional weighted residuals show no obvious trend with time. These findings are reassuring about the displayed aspects of the fitted data. They do not establish that simulated new subjects will have the right distribution of concentrations at every time or in every subgroup. Conversely, an unusual predictive-check result does not by itself identify the cause. It may reflect a model inadequacy, a data feature not represented in the check, limited data, or simulation variability.
A useful interpretation starts with the specific mismatch. State whether the observed data are systematically above or below the predictive distribution, which time or population stratum is affected, and whether the discrepancy concerns a central tendency, spread, tail, or another prespecified feature. Avoid turning an isolated point outside a predictive interval into a general claim that the model fails everywhere.
Worked Example: A Late-Time Predictive-Check Discrepancy
In a hypothetical analysis, an observed-versus-individual-predicted plot for 120 participants shows no marked overall departure from the identity line. A residual-versus-time plot shows no clear trend. A visual predictive check, however, compares the observed 90th percentile in each of 30 late-time bins with a 90% simulation interval for that percentile. The observed percentile lies above the interval in 8 bins.
Under a well-calibrated 90% simulation interval for a percentile, an observed percentile can fall outside the interval by chance. The rough nominal expectation is \(0.10 \times 30 = 3\) bins outside, counting both tails. Here, 8 of 30 are outside, and all are above the upper limit. The arithmetic is \(8/30 = 0.267\), or 26.7% of the bins. The direction and clustering make this more concerning than a few scattered exceedances on both sides.
The defensible conclusion is that the fitted-data plots do not show an obvious overall pattern, but the predictive check suggests that the model may underpredict the upper part of the late-time concentration distribution under this design. The result does not establish why the upper tail is underpredicted, nor does it establish that predictions are inadequate at early times or in every subgroup. The count is descriptive; it is not a formal significance test, and the bins need not be independent.
When a check uses intervals, interpret the interval as a reference distribution under the fitted model and simulation procedure. It is not a confidence interval for a fixed model parameter. Nor does a nominal 90% interval guarantee that exactly 90% of observed summaries will fall inside it in one finite dataset. The degree and pattern of departure matter, alongside the amount of data and the intended application.
Worked Interpretation of a Single-Time Check
Worked Example: An Observed Concentration Above the Predictive Range
At a specified late sampling time for a hypothetical dose group, 1,000 simulated datasets produce a median concentration of 0.71 mg/L and a central 90% predictive interval from 0.42 to 1.12 mg/L. The observed group’s corresponding concentration summary is 1.30 mg/L. In the simulations, 978 of 1,000 summaries are below 1.30 mg/L.
The observed value exceeds the upper predictive limit by \(1.30 - 1.12 = 0.18\) mg/L. Its empirical percentile among the simulated summaries is approximately \(978/1000 = 0.978\), or the 97.8th percentile. Thus, the observed summary is above the central 90% predictive interval and near the upper tail of the simulated distribution.
This gives a concrete description of the mismatch: the model’s simulations tend to produce lower values than the observed late-time summary for this dose group. It does not mean there is a 97.8% probability that the model is wrong. The percentile is a position relative to these simulations, conditional on the fitted model, design, and simulation assumptions. It also does not identify whether the issue is the typical concentration, between-subject variability, residual error, or another feature.
Checking Whether the Result Is Robust and Relevant
Before writing a broad conclusion from a concerning check, verify that the comparison is meaningful. Confirm that observed and simulated summaries use the same dose and sampling-time definitions, the same handling of records, and comparable covariate strata. Check that simulations include the residual variability and between-subject variability relevant to the prediction target. If the check uses prediction-corrected concentrations or another transformation, make sure the observed and simulated values were processed consistently.
Then ask whether the mismatch is concentrated or widespread. A pattern at late times in one dose group should be reported as such, rather than generalized to all concentrations. If the model is intended to support simulation at those times or doses, that pattern has direct relevance. If it is intended for a different, well-evaluated range, the discrepancy may be less directly limiting—but it should still be documented, and extrapolation beyond the evaluated range should not be implied.
The check’s resolution also matters. Sparse sampling, few individuals in a subgroup, or a small number of simulations can make predictive summaries imprecise. Increasing the number of simulations can reduce Monte Carlo noise in estimated simulation quantiles; it cannot compensate for limited observed information or a misspecified model. A visual pattern should be considered with its data support and the model’s intended use, rather than treated as a standalone pass/fail decision.
Say which fit plots appear acceptable and identify the predictive check that raises concern. Specify its summary, time range, and relevant group.
Give its direction and scale where possible, such as the number of bins outside an interval or the observed value relative to the simulated range.
Distinguish a localized pattern from a general one, and note relevant limits such as sparse data or a descriptive, nonformal comparison.
Explain which predictions appear supported, which remain uncertain, and which proposed uses require further evaluation or should not be claimed.
From Evaluation to a Balanced Conclusion
Worked Example: Limiting a Proposed Use
A hypothetical model has acceptable observed-versus-individual-predicted and residual plots over the study’s sampled concentration range. A predictive check shows that late-time upper-tail concentrations in the highest dose group frequently exceed the model’s 90% simulation envelope. The analysis team is considering using the model to simulate typical concentrations at the studied doses and to predict late-time upper-tail exposure at a higher dose.
A balanced conclusion would separate these uses. The observed plots provide some support for describing the fitted data over the sampled range. The predictive-check pattern raises concern about upper-tail prediction at late times even at the highest studied dose. It therefore does not support an unqualified claim that upper-tail predictions are reliable there. Prediction at a higher dose is still less directly evaluated and should not be presented as established by these diagnostics.
For example: “Fit plots showed no prominent overall trend over the sampled concentration range. In contrast, the predictive check showed repeated undercoverage of observed late-time upper-tail concentrations in the highest dose group. These findings support cautious description of the fitted data but leave late-time upper-tail predictions uncertain, particularly for doses beyond those evaluated. The discrepancy does not by itself identify its cause.”
This wording reports both results, identifies the location and direction of concern, and distinguishes observed support from unsupported extrapolation. It does not claim that the model is universally invalid or that acceptable plots prove the proposed simulations are dependable.
Common Pitfalls, Alternatives, and Limitations
A common mistake is to reason, “The observed-versus-predicted plot looks good, so the model predicts well.” A correction is to ask what predictions were plotted and whether they condition on observed individual data. A good individual-prediction plot can coexist with weak population-level prediction. Another mistake is to treat every point outside a predictive interval as proof of failure. The correction is to assess the pattern, direction, number of comparisons, data support, and relevance to the intended use.
A predictive check is not interchangeable with a residual plot or a single numerical goodness-of-fit summary. Fit plots can make local patterns visible in the fitted observations, while simulation-based checks can assess whether distributions or selected summaries are reproduced under a specified design. These approaches complement each other. A formal test, where appropriate, can address a prespecified hypothesis, but it does not replace inspecting what feature is discrepant or deciding whether that feature matters for the application. No one evaluation should be treated as a universal verdict.
Predictive checks have limitations: their conclusions depend on the model used to simulate, the design and summaries chosen, and the amount of observed information. A check can be reassuring for the features it examines while missing an unexamined feature. A failure can signal a real limitation without showing which model component should change. Any proposed explanation—for example, an unmodeled covariate pattern or an inadequate variability structure—is a hypothesis to investigate, not a finding established by the discrepancy alone.
This is why a report should distinguish results from interpretation, make limitations explicit, and connect conclusions to the proposed clinical or simulation use. If the concern affects a decision-relevant prediction, the appropriate conclusion may be that the current evaluation is insufficient to support that use without further investigation. It is not necessary to resolve the cause before reporting the observed discrepancy accurately.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- A residual plot has no obvious time trend, but observed late-time concentrations are repeatedly above the predictive envelope. What do the two findings say, and what should not be inferred?
- In 20 predictive-check bins, 5 observed summaries are above the upper limit and none are below the lower limit. What is the observed fraction, and why is it not automatically a formal test result?
- Why might a good observed-versus-individual-predicted plot fail to establish reliable prediction for a new individual?
- A concern appears only in a sparse subgroup. Name two details to check before generalizing the result to the full population.
- Write one sentence that reports acceptable fit plots and a concerning upper-tail predictive check without claiming a cause.
Solutions. (1) The residual plot shows no obvious time pattern in the displayed residuals, while the predictive check suggests underprediction at late times. Neither result proves that the model is adequate or inadequate for every use, and the discrepancy alone does not identify its cause. (2) The fraction is \(5/20 = 0.25\), or 25%. It is a descriptive count; bins may be dependent, the number of comparisons matters, and no formal test or reference distribution for that count has been specified. (3) Individual predictions can use observed concentrations to estimate individual random effects, so their apparent agreement does not establish how well the model predicts without those observations. (4) Check the subgroup’s sample size and sampling-time/dose coverage, and verify that observed and simulated records and subgroup definitions match. (5) One suitable sentence is: “Fit plots showed no prominent overall trend, whereas the predictive check showed repeated undercoverage of the observed upper tail at late times; the diagnostics do not establish the cause of this discrepancy.”