Clinical Evaluation Is a Comparison With a Defined Target
A PBPK prediction is informative only when the clinical evidence being used to evaluate it represents a clearly described target. A simulated concentration at a particular time cannot be compared meaningfully with an observation from a different dose, population, analyte, or sampling schedule without explaining why the comparison is appropriate. The reporting task is therefore not simply to show that a curve looks close to a data point. It is to document what clinical data were evaluated, how predictions were aligned with those data, what discrepancies remain, and whether they matter for the stated regulatory use.
For each clinical dataset, report the study design and population, dose and regimen, formulation when relevant, analyte and biological matrix, sampling schedule, sample size, and the summary statistics being compared. State whether the observations are individual values, arithmetic means, geometric means, or another study-level summary. Also describe relevant data handling, such as exclusions or below-quantification observations, and identify which simulation output corresponds to the comparison. The introductory material on organizing a PBPK submission report establishes the broader report structure; here the emphasis is the evidence and performance comparison itself.
Observed variability and prediction discrepancy are not the same quantity. A clinical mean with a wide spread may be consistent with a population prediction even when individual observations differ substantially. Conversely, a prediction can match an overall mean while missing a systematic time trend or an important subgroup. Report the observed data and the prediction on the same scale, but retain enough information about variability and study design to avoid treating a mean as if it described every participant.
Prediction Errors on Concentration and Exposure Scales
For a positive observed value \(O_i\) and its corresponding PBPK prediction \(P_i\), define the prediction-to-observation ratio \(R_i\). A ratio above 1 indicates overprediction; a ratio below 1 indicates underprediction. The index \(i\) may represent a matched time point, study-level summary, or other defined comparison unit. State what one unit represents before summarizing ratios.
Because concentration and exposure values are often positive and discrepancies are naturally described as fold differences, summaries on the logarithmic scale can be useful. The geometric mean ratio summarizes direction, while the average absolute fold error summarizes typical discrepancy without allowing over- and underprediction to cancel. Both depend on the selected comparison units, so a report should list or describe those units and not present a single summary without its underlying evaluations.
Here \(m\) is the number of positive, defined prediction-observation comparisons, and AAFE is the average absolute fold error. A geometric mean ratio of 1 indicates no net directional bias on the log scale; a value above 1 indicates net overprediction and a value below 1 net underprediction. AAFE is at least 1, with 1 indicating exact agreement for every comparison. It does not show direction, and it can hide whether a few comparisons account for most of the discrepancy. A percentage within a prespecified fold range can add an easily interpreted description, but the range must be justified for the intended use rather than treated as a universal acceptance rule.
These log summaries require positive values. A zero, negative, or below-quantification observation cannot be inserted into a log ratio as if it were an ordinary positive measurement. Report how such observations were handled and why that handling is appropriate; do not silently replace them with a convenient value. If observations are summarized at the study level, make clear that the metric describes those study-level comparisons, not individual-level prediction performance.
Worked Example: Summarizing Matched Concentration Predictions
Suppose a hypothetical clinical evaluation reports concentrations at four matched times. The observed concentrations are 2.0, 4.0, 6.0, and 3.0 mg/L; the corresponding PBPK predictions are 2.4, 3.2, 9.0, and 6.0 mg/L. Assume all values refer to the same analyte, matrix, dose, population summary, and time points.
The prediction-to-observation ratios are \(2.4/2.0=1.20\), \(3.2/4.0=0.800\), \(9.0/6.0=1.50\), and \(6.0/3.0=2.00\). The first, third, and fourth predictions are above observation; the second is below it.
The geometric mean ratio is calculated as:
Across these four time-point comparisons, the geometric mean prediction is about 1.30 times the observation on the log scale. For AAFE, use absolute log ratios:
Thus, the average absolute fold error is about 1.46. All four ratios are between 0.5 and 2, so 4 of 4 comparisons are within two-fold. That count does not establish acceptability by itself: its relevance depends on the intended use and the acceptance criteria. The early underprediction followed by larger late overpredictions also signals a possible profile-shape discrepancy that the aggregate ratios do not explain.
Profiles, Exposure Metrics, and Diagnostic Displays
A concentration-time comparison should show the observed summaries and the PBPK predictions against time, using compatible units and clear labels. Include the observation count or other indication of data support at each time when available. If the clinical observations are variable, show an appropriate display of that variability rather than implying that a mean is a precise individual prediction. A profile plot can reveal delayed peaks, mismatched decline, or a time-dependent bias that one exposure metric conceals.
Exposure comparisons answer a different question from pointwise concentration comparisons. AUC summarizes exposure over a stated interval and requires the interval and calculation method to be reported. Cmax is the maximum concentration and should be based on a clearly specified observed or simulated profile and sampling or prediction convention. These metrics have different units: for example, AUC may be in mg·h/L and Cmax in mg/L. Report prediction-to-observation ratios separately for each metric, not as if they were interchangeable measures.
Worked Example: Interpreting AUC and Cmax Together
In a hypothetical study, observed AUC over the specified interval is 24 mg·h/L and observed Cmax is 4.0 mg/L. The corresponding PBPK predictions are 30 mg·h/L and 3.2 mg/L.
For AUC, the ratio is \(30/24=1.25\): predicted AUC is 25% higher than observed. For Cmax, the ratio is \(3.2/4.0=0.800\): predicted Cmax is 20% lower than observed. The calculations use the same direction convention, prediction divided by observation, but the results point in opposite directions.
The appropriate report is not that the model is simply “high” or “low.” It is that this comparison overpredicts AUC while underpredicting Cmax. This combination could be consistent with a profile that is too broad or has a different peak shape, but the two values alone do not identify the cause. Examine the concentration-time data and relevant assumptions before drawing that interpretation. Whether either discrepancy affects the regulatory conclusion depends on the intended use and the applicable decision criteria.
Useful diagnostic displays include observed-versus-predicted plots with a line of identity, prediction-to-observation ratios by time, and concentration-time plots with observations and predictions overlaid. A line of identity helps identify directional departures but does not by itself quantify variability or show whether the evaluation data are representative of the intended use. Ratio plots can reveal patterns: ratios consistently above 1 suggest overprediction, while ratios that increase or decrease over time suggest time-dependent disagreement. Where the data support it, stratify displays by clinically relevant study or population features. Avoid subdivisions that leave too few observations to interpret reliably.
A discrepancy should be investigated against the comparison definition and the data before it is attributed to model behavior. Check that time points, units, dose, regimen, analyte, matrix, and summary level align. Then examine whether the difference is isolated or repeated, whether it occurs at a decision-relevant time or exposure measure, and whether it changes the conclusion under the stated acceptance criteria. Preserve discordant evidence in the report: excluding an inconvenient study or metric without a defensible reason can make the overall assessment misleading.
A Short Justification for Log-Scale Fold Summaries
The log scale provides a symmetric way to describe reciprocal fold errors. A prediction twice the observation has ratio 2; a prediction half the observation has ratio 0.5. Their log errors have equal magnitude and opposite sign:
Averaging signed log ratios therefore allows opposite directional errors to cancel, which is why the geometric mean ratio describes net direction rather than typical absolute discrepancy. Taking absolute values before averaging prevents that cancellation. Exponentiating returns the average from log units to a multiplicative fold scale. For example, if every prediction is exactly twice its observation, each \(|\ln R_i|=\ln 2\), so AAFE is \(\exp(\ln 2)=2\). This derivation explains the interpretation of the metrics; it does not make any particular fold threshold a regulatory standard.
The calculations also assume that the chosen comparisons are meaningful and that each receives the intended weight. If many time points from one study are included alongside only a few from another, the first study may dominate a pooled summary. Report study-level results or an explicit weighting strategy when pooling would otherwise obscure that imbalance. A pooled metric should not substitute for showing important study-specific discrepancies.
Failure Modes and Regulatory Interpretation
Common signs that the evaluation is being summarized poorly include unmatched comparison targets, a pooled number with no description of its units, a profile plot without observed-data context, and a favorable overall metric alongside an unexplained important outlier. Ratios can also appear precise while being based on sparse or variable observations. If the clinical evidence is limited, state that limitation rather than letting a numerical summary imply stronger support than the data provide.
No single diagnostic plot or error metric proves that a PBPK model is valid for every use. The report should connect the observed performance to the intended use, decision-relevant acceptance criteria, and any remaining limitations. Where a discrepancy could alter the regulatory conclusion, show the evidence and explain the consequence. Where it does not alter that conclusion, explain why, rather than merely labelling it acceptable. This makes the reasoning reviewable without claiming that agreement establishes causation or guarantees performance outside the evaluated conditions.
Worked Example: Reporting a Time-Dependent Discrepancy
Consider the four matched concentrations in the first example. The ratios rise from 1.20 at the first time to 2.00 at the last, although one intermediate ratio is 0.800. A hypothetical evaluation criterion states that concentration predictions should be within two-fold at the prespecified comparison times, and all four ratios meet that numerical bound.
A defensible report would state that all four comparisons are within two-fold, with a geometric mean ratio of 1.303 and AAFE of 1.456. It would also state that the pattern is not directionally uniform: one time point is underpredicted, while later points are overpredicted, including a two-fold prediction at the final time. The profile and ratio-by-time displays should be reviewed for a systematic shape difference.
The criterion is met for the stated numerical comparisons, but that fact alone does not establish that the model is suitable for every decision. If the late-time concentration is important to the context of use, its discrepancy requires explicit consideration. If it is not decision-relevant under the stated use, the report should give the rationale and retain the discrepancy in the evidence summary. This conclusion distinguishes the observed calculation from the regulatory interpretation.
Check Your Understanding
Answer each question using the definitions and comparison conventions in this tutorial.
- A prediction is 8 mg/L and its matched observation is 10 mg/L. What is the prediction-to-observation ratio, and what direction of error does it indicate?
- Why can a geometric mean ratio near 1 coexist with substantial prediction errors?
- A report gives AUC in mg·h/L and Cmax in mg/L. Why should their prediction ratios be interpreted separately?
- What pattern in a ratio-by-time plot could suggest that the PBPK prediction has a time-dependent discrepancy?
- Why is a favorable pooled metric not enough if one decision-relevant study or time point shows a marked discrepancy?
Solutions. (1) \(8/10=0.800\), indicating 20% underprediction. (2) Signed log errors above and below 1 can cancel when averaged, even if individual ratios are far from 1; inspect AAFE and the individual comparisons. (3) They describe different exposure features and have different units, so agreement in one does not imply agreement in the other. (4) Ratios that systematically rise or fall with time, rather than fluctuating without a pattern, may indicate time-dependent disagreement. (5) Pooling can conceal a localized but important failure; report the discrepancy and assess its relevance to the intended regulatory use and criteria.