Prediction Evidence Must Be Judged Against a Use
A PBPK model can predict an exposure endpoint closely in one setting and poorly in another. Whether that performance is adequate depends on the proposed use: which population, drug, dosing conditions, scenarios, endpoints, and decisions are in scope. A prediction error has no context-free label of “good” or “bad.” It becomes decision-relevant when compared with criteria appropriate to the intended use and considered alongside the consequences of being wrong.
The central task is therefore not to find one accuracy statistic and declare the model accurate. It is to ask whether the observations test the endpoints and scenarios that matter, whether prediction errors are acceptably small, and whether their direction could systematically distort the decision. Acceptance criteria should be specified in relation to the intended use before examining the performance results; a criterion selected after seeing the errors risks making the evidence appear more favorable than it is.
Put Predictions and Observations on Comparable Terms
For a positive endpoint such as AUC or Cmax, define the prediction ratio for scenario \(i\) as the predicted value divided by the corresponding observed value. The ratio is dimensionless; a value above 1 indicates overprediction, and a value below 1 indicates underprediction. A percentage error expresses the same comparison as a percentage of the observed value.
The comparison is meaningful only if the prediction and observation refer to the same endpoint definition, units, dose or dosing conditions, population, and scenario. For example, a prediction of steady-state AUC should not be compared with a single-dose observed AUC as though they were interchangeable. If the observed endpoint is itself an estimate, its sampling uncertainty also matters: a difference between a prediction and a noisy estimate is not automatically evidence of a model defect. The uncertainty in the observation and the predictive distribution should be considered when the evidence available includes them.
Ratios are useful for positive endpoints because relative error is often more interpretable than an absolute difference across endpoints with different scales. A ratio of 1.20 means 20% overprediction, whether the AUC is expressed in one set of units or another. For endpoints that can be zero or negative, a ratio may be undefined or misleading; a difference on a scientifically meaningful scale may be more suitable. The measure must match the endpoint and the use.
Why the Log Ratio Is Useful
For positive endpoints, the log ratio gives a signed measure of relative error. It has a useful symmetry: predicting twice the observed value gives a log ratio of \(\ln(2)\), while predicting half the observed value gives \(-\ln(2)\). The two errors are equal in magnitude and opposite in direction on this scale. This can help summarize multiplicative errors, but it does not replace examination of individual scenarios.
The symmetry follows directly from reversing the ratio. If the prediction is twice the observation, \(R_i=2\). If it is half the observation, \(R_i=1/2\), and \(\ln(1/2)=-\ln(2)\). Thus the sign records direction and the magnitude records multiplicative discrepancy. Exponentiating a mean log ratio gives the geometric mean prediction ratio, a measure of average multiplicative bias. A mean near 1 does not imply that every scenario is close to 1: overpredictions and underpredictions can cancel.
Examine Endpoints and Scenarios Separately
Evidence should be organized around the actual claims being made. If a use depends on both AUC and Cmax, adequate AUC performance does not establish adequate Cmax performance. Likewise, evidence from a typical population may not directly support a prediction for a population with altered physiology or enzyme activity. The scenario coverage should be compared with the intended population and conditions, not merely counted.
Decision-relevant acceptance criteria provide a reference for judging error magnitude. For example, a use might require each primary endpoint to remain within a prespecified ratio range in designated scenarios. Another use might allow a larger error for an endpoint that does not change the decision. These are use-specific choices, not universal PBPK thresholds. The earlier material on decision-relevant acceptance criteria establishes the need to define such limits in advance; here the emphasis is on interpreting the observed evidence against them.
Worked Example: Different Endpoints, Different Conclusions
Suppose a hypothetical PBPK evaluation is intended to support an exposure assessment. Before reviewing results, the team specifies that both AUC and Cmax must have prediction ratios from 0.80 through 1.25 in the evaluated scenario. The observed AUC is 120 mg·h/L and the predicted AUC is 144 mg·h/L. The observed Cmax is 10 mg/L and the predicted Cmax is 7.5 mg/L. Assess the results.
For AUC, the prediction ratio is \(144/120=1.20\). The percentage error is \(100(1.20-1)=20\%\), so AUC is overpredicted by 20%. Since \(0.80\leq1.20\leq1.25\), this endpoint meets the stated criterion.
For Cmax, the prediction ratio is \(7.5/10=0.75\). The percentage error is \(100(0.75-1)=-25\%\), so Cmax is underpredicted by 25%. Since \(0.75<0.80\), this endpoint does not meet the criterion.
The conclusion is not that the model “passes overall” because its AUC prediction meets the limit, nor that both endpoints fail because Cmax misses it. Under the stated rule, the AUC result meets its criterion and the Cmax result does not. If the intended use requires both endpoints to meet their limits, the evidence does not meet that acceptance condition.
Worked Example: A Good Aggregate Can Hide a Scenario Failure
A hypothetical evaluation compares predicted and observed AUC in four scenarios. The prediction ratios are 1.10, 0.90, 1.40, and 0.80. The prespecified acceptable range is 0.80 to 1.25 in every scenario. Calculate the geometric mean ratio and interpret it alongside the scenario results.
The geometric mean ratio is the exponential of the mean log ratio:
The geometric mean is about 1.026, indicating a small average multiplicative overprediction across these four results. But scenario 3 has a ratio of 1.40, corresponding to 40% overprediction, and exceeds the upper limit of 1.25. Scenario 4 is exactly at the lower boundary, 0.80, and meets the stated inclusive criterion. Scenarios 1 and 2 also meet it.
The aggregate summary does not erase the scenario 3 failure. If the use depends on performance in every one of these scenarios, the evidence does not meet the prespecified condition. If some scenarios are outside the intended use, their relevance should be addressed explicitly rather than hidden by pooling.
Direction of Error Can Matter to a Decision
Two prediction sets with similarly sized errors can have different implications if one is mainly high and the other mainly low. A systematic direction may shift a decision even when the average absolute discrepancy appears modest. Interpret the direction in the decision’s causal sequence: which model output feeds the decision, how that output changes the chosen action, and which consequence follows from an error in that direction.
Worked Example: Underprediction and a Dose Recommendation
Consider a hypothetical use in which a model estimates AUC per dose to choose a dose for an exposure target. Assume, for this calculation, dose proportionality over the relevant range: predicted AUC is proportional to dose. In four scenarios the prediction-to-observation ratios for AUC at a fixed dose are 0.95, 1.05, 0.70, and 0.72. Assess the error pattern and its possible decision consequence.
The log ratios are approximately \(\ln(0.95)=-0.0513\), \(\ln(1.05)=0.0488\), \(\ln(0.70)=-0.3567\), and \(\ln(0.72)=-0.3285\). Their mean is \((-0.0513+0.0488-0.3567-0.3285)/4=-0.1719\). Exponentiating gives a geometric mean ratio of \(\exp(-0.1719)\approx0.842\). Thus, on average on the multiplicative scale, predicted AUC is about 15.8% below observed AUC.
The two predictions near 1 do not cancel the relevance of the two larger underpredictions. In a scenario with ratio 0.70, the model predicts only 70 units of AUC for every 100 units observed at the same dose. Under the stated dose-proportionality assumption, it would estimate that a dose \(1/0.70\approx1.43\) times the appropriate dose is needed to reach a target based on observed exposure. The model could therefore recommend an unnecessarily high dose in that scenario. This is a conditional consequence, not a conclusion about all uses: it depends on the dose-proportionality assumption, the target, and the scenario being in scope.
Compare Evidence Summaries, Not Just Numbers
Different summaries answer different questions. A scenario-by-scenario comparison shows where errors occur and whether criteria are met. A geometric mean ratio summarizes average multiplicative direction, while an average absolute percentage error summarizes typical magnitude without retaining direction. Neither alone identifies whether the worst error occurs in a decision-critical subgroup. Prediction intervals and observed outcomes can also be compared when uncertainty is part of the claim, but interval coverage does not by itself demonstrate small errors or adequate performance for every important endpoint.
| Approach | What it helps assess | What it can miss |
|---|---|---|
| Scenario-specific ratios against limits | Whether each relevant comparison meets a defined criterion | Performance in untested scenarios; uncertainty in observed estimates |
| Geometric mean ratio | Average multiplicative bias and its direction | Large individual errors or opposing errors that cancel |
| Average absolute error | Typical error magnitude without sign cancellation | Whether errors are mainly over- or underpredictions |
| Prediction-interval comparison | Whether observed outcomes are compatible with a stated predictive range | Whether the interval is too wide for the decision or errors are systematically directional |
| Decision-level assessment | Whether plausible prediction errors change an action or its consequences | Whether the underlying endpoint predictions are accurate across the full intended domain |
Diagnostics, Limitations, and Common Pitfalls
Before interpreting a numerical summary, check that each observed value is paired with the prediction for the same endpoint and conditions, that units and definitions match, and that the scenario label is correct. Then inspect the individual signed errors and compare them with the prespecified criteria. A repeated direction of error, a few unusually large discrepancies, or a pattern concentrated in a relevant scenario can be important even when an overall summary is close to 1. These observations flag questions for further assessment; by themselves they do not identify the cause of the pattern.
Evidence is limited by what was observed. Good agreement in tested scenarios does not establish performance in an untested population or exposure range. Conversely, a result outside a criterion is evidence about that comparison, not automatic proof that the model is unusable for every other purpose. The conclusion must stay within the endpoints and scenarios actually evaluated. Measurement uncertainty, sparse evidence, and uncertainty in the model prediction can also make a sharp pass/fail interpretation less informative than a transparent account of the estimates and their uncertainty.
Common mistake: “The geometric mean prediction ratio is close to 1, so the model is adequate.” This reasoning treats an average as if it guaranteed acceptable performance in every relevant scenario. For example, a strong overprediction and a strong underprediction can offset on the log scale. The correction is to report the aggregate measure together with scenario-level errors, endpoint-specific criteria, and the decision relevance of any exceptions.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A hypothetical model predicts a Cmax of 18 mg/L when the observed value is 15 mg/L. What is the prediction ratio and percentage error, and which direction is the error?
- Two endpoints are required for a proposed use. One meets its prespecified criterion and the other does not. Can the first endpoint’s result establish that both meet the intended-use requirement? Explain.
- Four scenario ratios are 0.95, 1.00, 1.02, and 1.50. If the acceptance range is 0.80 to 1.25 in every scenario, does a near-unity aggregate summary remove the failure?
- Why may a ratio be unsuitable for comparing an endpoint that can be zero or negative?
- Under dose proportionality, a model underpredicts AUC at a fixed dose by 20%. In which direction might its dose recommendation shift when it is used to reach an AUC target?
Solutions. (1) The ratio is \(18/15=1.20\). The percentage error is \(100(1.20-1)=20\%\), so the model overpredicts Cmax by 20%. (2) No. If the stated requirement is that both endpoints meet their criteria, one endpoint’s performance cannot substitute for the other’s; the evidence fails that combined requirement. (3) No. The ratio 1.50 exceeds the upper limit of 1.25, so the criterion is not met in every scenario. An aggregate near 1 cannot erase that result. (4) Division by zero is undefined, and ratios near zero can behave unstably or fail to convey meaningful relative differences. A scientifically interpretable difference may be more appropriate. (5) If predicted AUC per dose is 80% of the observed value, the model may estimate that more dose is needed to reach the target. Under the stated proportionality assumption, it would multiply the appropriate dose by \(1/0.80=1.25\), a 25% increase, potentially producing excess exposure.