Why Check Predictions Against Clinical PK?
A PBPK model is useful for a development decision only if its predictions are sufficiently consistent with relevant clinical pharmacokinetic evidence. Agreement is not a single number: a model may predict average exposure well while missing a dose trend, a particular population, or an important study condition. The comparison therefore needs to preserve the conditions and PK quantity that matter to the decision.
Before calculating an error, specify the predicted quantity (for example, AUC or \(C_{\max}\)), its units and summary type, the dose and population, and the clinical estimate to which it will be compared. The prediction and clinical estimate must refer to comparable dosing, sampling, formulation, and covariate conditions. A comparison of steady-state AUC with single-dose AUC, for instance, is not an agreement check unless the difference is explicitly accounted for.
These definitions presume that the clinical estimate is a meaningful comparator and that the prediction and observation are aligned on the target quantity and condition. If the clinical estimate itself is imprecise, the discrepancy should be interpreted in light of that precision, rather than treated as an exact truth. The comparison is not a causal analysis and does not, by itself, establish that a particular mechanism explains agreement or disagreement.
Summarizing Prediction Error
The ratio gives an intuitive fold comparison: \(R_i=1\) is exact agreement, \(R_i=1.20\) means the prediction is 20% above the clinical estimate, and \(R_i=0.80\) means it is 20% below. Ratios are often more useful than absolute differences when PK values span a wide range, but an absolute difference can also matter when a decision depends on a particular exposure increment.
For a set of \(m\) matched conditions, the geometric mean ratio (GMR) summarizes average multiplicative bias. It should be accompanied by condition-level results, because a near-one GMR can conceal offsetting over- and underpredictions.
A commonly used symmetric fold-error criterion is a ratio between 0.80 and 1.25. The symmetry is on the log scale: \(\log(0.80)=-\log(1.25)\). Such a bound is a possible prespecified criterion, not a universal rule or automatic regulatory standard. Its relevance depends on the decision, the PK metric, and the consequences of error in either direction.
Worked Example: Ratios Across Five Matched Conditions
Hypothetical data. For a defined PK metric, a PBPK model makes predictions for five matched clinical conditions. The predictions and clinical estimates are in the same units.
| Condition | Prediction \(P_i\) | Clinical estimate \(O_i\) | Ratio \(P_i/O_i\) | Relative error |
|---|---|---|---|---|
| 1 | 90 | 100 | 0.90 | -10% |
| 2 | 110 | 100 | 1.10 | +10% |
| 3 | 75 | 100 | 0.75 | -25% |
| 4 | 120 | 100 | 1.20 | +20% |
| 5 | 105 | 100 | 1.05 | +5% |
For condition 3, \(R_3=75/100=0.75\), so \(E_3=0.75-1=-0.25\), or 25% underprediction. Applying the prespecified 0.80–1.25 ratio range, conditions 1, 2, 4, and 5 are within the range; condition 3 is not. Thus \(4/5=0.8\), or 80%, of these five ratios meet that criterion.
The GMR is calculated from the ratios: \(0.90\times1.10\times0.75\times1.20\times1.05=0.93555\), so \(\mathrm{GMR}=0.93555^{1/5}\approx0.987\). The geometric average is close to 1, indicating little aggregate multiplicative bias across these five conditions. It does not erase the 25% underprediction in condition 3. Both the summary and the individual errors should be reported.
Prediction-Interval Coverage
A point prediction does not show how much uncertainty the model assigns to a future clinical result. When the model supplies a prediction interval for each matched condition, an observed clinical estimate can be checked against that interval. Coverage is the proportion of the selected observations that lie inside their corresponding intervals.
A prediction interval describes a range for a future or held-out observation under the model and its stated predictive assumptions. It is not a confidence interval for a model parameter. If the intervals are intended to have 90% predictive coverage, that is a long-run property under repeated comparable conditions and a calibrated predictive procedure; it does not guarantee that exactly 90% of a small set of observations will be covered. Empirical coverage from few conditions is correspondingly imprecise.
Worked Example: Checking Coverage Against a Prespecified Rule
Hypothetical data. Before examining five clinical estimates, a model team generated a 90% prediction interval for each matched condition. The prespecified operational criterion is that at least 80% of the five clinical estimates fall within their intervals.
| Condition | 90% prediction interval | Clinical estimate | Covered? |
|---|---|---|---|
| 1 | 8–12 | 10 | Yes |
| 2 | 16–24 | 20 | Yes |
| 3 | 25–35 | 30 | Yes |
| 4 | 36–44 | 40 | Yes |
| 5 | 42–48 | 50 | No |
Four observations are within their intervals, so \(C=4/5=0.8\), or 80%. The criterion is \(C\geq0.80\), so this five-condition check meets the stated operational rule exactly. It does not demonstrate that the model has reliable 90% coverage in general: five observations provide limited evidence, and the rule was a decision criterion, not a statistical proof of calibration.
Prespecify Acceptance Criteria by Decision
An acceptance criterion translates a modeling question into a decision rule. It should be set before the relevant clinical results are used to judge performance. Criteria may address the magnitude of errors, interval coverage, or both. They should also state how results will be examined across doses, populations, and study conditions. A pooled score alone can conceal a failure in a subgroup or at an influential dose.
For example, a plan might require each critical condition’s prediction-to-observation ratio to fall within a stated range, require a specified fraction of held-out estimates to be covered by model prediction intervals, and separately inspect dose and population patterns. The exact thresholds are application-specific. A boundary appropriate for exploratory context may be unsuitable when the result determines a dose adjustment or supports a high-consequence decision.
The comparison is interpretable only if its evidence set and calculations reflect the intended use. In particular, the clinical conditions should represent the target use or have a justified relationship to it; the model should not be tuned on the same observations and then described as independently validated against them; and the PK estimand and units should be consistent. If an acceptance rule is missed, report what failed and where. Do not revise the threshold after seeing the result and then present the revised rule as prespecified.
Worked Example: A Dose-by-Population Gate
Hypothetical data. A team has prespecified that the prediction-to-observation ratio must be between 0.80 and 1.25 for every critical dose-by-population condition. The metric and clinical estimates are comparable and use the same units.
| Population | Dose | Prediction | Clinical estimate | Ratio | Condition passes? |
|---|---|---|---|---|---|
| A | 10 mg | 95 | 100 | 0.95 | Yes |
| A | 100 mg | 108 | 100 | 1.08 | Yes |
| B | 10 mg | 102 | 100 | 1.02 | Yes |
| B | 100 mg | 132 | 100 | 1.32 | No |
For population B at 100 mg, \(R=132/100=1.32\), a 32% overprediction. Since \(1.32>1.25\), that condition fails. The other three ratios are within 0.80–1.25, but the prespecified rule requires every critical condition to pass; therefore the overall gate fails.
Pooling population B across its two doses would give a GMR of \(\sqrt{1.02\times1.32}=\sqrt{1.3464}\approx1.160\), which is within 0.80–1.25. That pooled summary would conceal the high-dose failure. The condition-level table therefore changes the conclusion under this gate and should accompany any pooled summary.
Diagnostics, Alternatives, and Limitations
A useful diagnostic display puts predictions and clinical estimates on the same scale, often with a line of identity, and shows ratio or log-ratio errors by dose, population, and study condition. Look for consistent overprediction or underprediction, errors that grow with dose, and misses concentrated in one population or design. Such patterns can signal that agreement is condition-dependent even when an overall average is near 1. Also check that the comparison has not mixed units, estimands, or study conditions.
No single summary answers every question. A GMR describes average multiplicative bias; the proportion within a ratio range describes how many point comparisons satisfy a chosen tolerance; prediction-interval coverage assesses whether observed estimates fall inside stated predictive ranges. A plot can reveal patterns that these summaries hide. A regression of predictions on observations can describe association or calibration patterns, but high correlation alone is not agreement: predictions can track observed differences while being systematically too high. Formal hypothesis tests are also not substitutes for prespecified practical criteria; a nonsignificant test does not establish adequate agreement.
Coverage estimates depend on how intervals were constructed and on whether the observations are relevant to the predictive target. Repeated measurements or multiple estimates from the same study may not constitute independent evidence. A handful of covered observations cannot establish general calibration, and broad intervals can achieve high coverage while providing little decision value. Report interval width and the conditions represented, not coverage alone.
Common mistake: “The GMR is close to 1, so the model agrees with clinical PK across the intended use.” That conclusion does not follow: opposing errors can cancel in an average, and a subgroup or dose-specific miss can matter greatly. Instead, report the aggregate summary alongside condition-level ratios, interval coverage, and the prespecified acceptance decision. Agreement supports the intended use only to the extent that the checked quantities and conditions represent it.
A Practical Evaluation Sequence
Name the PK metric, summary, units, dose, population, and study conditions; confirm that prediction and clinical estimate are matched.
Specify ratio or error bounds, any prediction-interval coverage rule, the critical conditions, and how failure of a condition affects the decision.
Compute each ratio and relative error, summarize bias if useful, and inspect condition-level results and prediction-interval coverage.
State which criteria passed, which conditions did not, and what the evidence does and does not support about the model’s intended use.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A model predicts an AUC of 240 mg·h/L and the matched clinical estimate is 200 mg·h/L. Calculate the ratio and relative error and interpret the direction.
- Four of five prediction-to-observation ratios satisfy a prespecified range. What proportion meets the criterion, and what can this result establish?
- Why can an overall GMR within 0.80–1.25 fail to demonstrate acceptable agreement at every dose and in every population?
- What is the difference between a 90% prediction interval and a confidence interval for a model parameter?
Solutions. (1) \(R=240/200=1.20\), and \(E=1.20-1=0.20\), or +20%. The prediction is 20% above the matched clinical estimate. (2) \(4/5=0.8\), or 80%. It establishes that four of these five comparisons meet the stated rule; it does not establish general performance across unobserved conditions. (3) Over- and underpredictions can offset in a pooled geometric mean, and a condition-specific miss may be important despite a near-one aggregate. Inspect and report the prespecified condition-level results. (4) A prediction interval concerns a future or held-out clinical result under the predictive model and assumptions. A parameter confidence interval concerns uncertainty in a model parameter; the two intervals address different quantities.