Why Check Predictions After Reduction?
Reducing a PBPK model can make it faster or easier to estimate, but a compact model is useful only if it preserves the predictions needed for its intended purpose. A parameter may have appeared dispensable for one endpoint, dose, or physiological setting and still matter elsewhere. Verification therefore compares the original and reduced models on a prespecified prediction domain: the combinations of doses, times, outputs, and physiological scenarios that the model is expected to represent.
The comparison is paired: each reduced-model prediction is matched with the original-model prediction for the same inputs and conditions. This isolates the change associated with reduction. It does not establish that either model is biologically correct; both models can agree and still be wrong. Nor is it a statistical hypothesis test. In a deterministic simulation, the computed difference describes model disagreement for the specified inputs, not sampling uncertainty or a probability that the models are equivalent.
Specify the Comparison Before Running It
First state what predictions matter. A domain might include several administered doses, a set of evaluation times, plasma concentration and tissue concentration, and physiological scenarios such as different organ-function assumptions. Include conditions relevant to intended use, not only conditions that are convenient or already known to show agreement. The selected dose range and scenarios should be scientifically justified, and time points should cover important parts of the profile, such as the rise, peak, and decline.
For each condition, the original and reduced models should use the same dose, physiological inputs, initial conditions, output definition, and units. Use compatible numerical solvers and sufficiently tight tolerances; otherwise, numerical approximation differences can be mistaken for effects of reduction. If the reduced model uses a different state representation, compare the same defined observable from each model rather than comparing internal quantities that do not have the same meaning.
Before examining the results, choose what counts as a material discrepancy for each important output. A tolerance should reflect the prediction’s intended use and scale. For example, a small relative change may matter near a decision threshold, while a large percentage change in a negligible concentration may not. There is no universal acceptable percentage for every PBPK application. State whether the criterion applies to each evaluation point, a summary metric, or both.
Measures of Prediction Change
The absolute difference, \(\Delta\), retains the units of the prediction and is easy to interpret on the original scale. The relative difference, \(r\), expresses the change as a fraction of the original prediction; multiplying by 100 gives a percentage. Its denominator makes it unsuitable when the original prediction is zero or close to zero. In those cases, report the absolute difference, or scale it by a prespecified reference value that is meaningful for that output.
For a time course evaluated at \(m\) times \(t_1,\ldots,t_m\), a useful summary is the root mean squared difference (RMSD). It gives more weight to larger discrepancies because differences are squared. A scale-normalized version divides RMSD by a meaningful, prespecified scale \(S\), such as a target concentration or the original model’s peak concentration. This allows comparison across outputs with different units, but the chosen scale must be stated. A summary does not replace inspection of the pointwise differences: a brief but important peak error can be diluted by many well-matched time points.
For a time-course summary such as area under the concentration-time curve (AUC), calculate that quantity from each model using the same time range and integration rule, then compare the two summaries. AUC agreement does not guarantee that peak concentration, peak timing, or the shape of the curve agrees. When those features matter, check them separately.
Worked Example: Comparing Peak and AUC Predictions
Suppose two hypothetical models are compared at one dose and one physiological scenario. The original model predicts a peak concentration of 12 mg/L and an AUC of 84 mg·h/L. The reduced model predicts 11.1 mg/L and 78 mg·h/L. For this example, suppose the prespecified materiality criterion is a relative difference within 10% for each of these endpoints.
Peak concentration. The paired difference is \(11.1-12=-0.9\) mg/L. The relative difference is
The reduced model’s peak is 7.5% lower, so it meets the stated 10% criterion for this endpoint.
AUC. The difference is \(78-84=-6\) mg·h/L, and the relative difference is
The AUC also meets the example criterion. This supports agreement for these two summaries under this one condition; it does not establish agreement across other doses, times, or physiological scenarios, and it does not establish that either model is correct.
Checking the Shape of a Time Course
Pointwise comparison can reveal where trajectories diverge, while a summary such as RMSD gives one measure of overall discrepancy on the chosen time grid. Consider the hypothetical concentration values below. The time points and values are identical inputs to both simulations.
| Time (h) | Original (mg/L) | Reduced (mg/L) | Difference (mg/L) |
|---|---|---|---|
| 0 | 0 | 0 | 0 |
| 1 | 8.0 | 7.2 | -0.8 |
| 2 | 5.0 | 4.5 | -0.5 |
| 4 | 2.0 | 2.4 | 0.4 |
Worked Example: Summarizing a Time-Course Discrepancy
Use the four listed times, and take the original peak concentration, \(S=8.0\) mg/L, as the prespecified normalization scale. The squared differences are \(0^2=0\), \((-0.8)^2=0.64\), \((-0.5)^2=0.25\), and \(0.4^2=0.16\), in \((\mathrm{mg/L})^2\). Thus,
and
The pointwise relative differences at 1, 2, and 4 hours are \(-10\%\), \(-10\%\), and \(+20\%\), respectively; at time zero the relative difference is undefined because the original prediction is zero. The NRMSD indicates a modest average discrepancy on the selected scale, but it does not erase the 20% relative difference at 4 hours. If the terminal concentration is important, that local deviation must be assessed against its own prespecified criterion. The RMSD also depends on the chosen time grid, so sparse sampling may miss a divergence between listed times.
AUC can provide a complementary summary. Using the trapezoidal rule over 0 to 4 hours, the original model gives \(4+6.5+7=17\) mg·h/L. The reduced model gives \(3.6+5.85+6.9=16.35\) mg·h/L. The relative AUC difference is \((16.35-17)/17\times100=-3.82\%\). The modest AUC difference coexists with the 20% relative difference at 4 hours: integrated exposure and a particular time-point prediction answer different questions.
Look Across Doses and Physiological Scenarios
The same comparison should be repeated over the prespecified domain. A useful report shows endpoint-specific differences by dose and physiological scenario, rather than only an overall average. This matters because agreement in one condition cannot be presumed to extend to another. A scenario-specific discrepancy can identify where reduction affects the target prediction, even if most of the domain shows small differences.
Worked Example: Finding a Scenario-Specific Change
Consider hypothetical AUC predictions at two doses and two physiological scenarios. All values are in mg·h/L. The materiality criterion for this example is an absolute relative difference no greater than 10% at every dose-scenario combination.
| Scenario | Dose (mg) | Original AUC | Reduced AUC |
|---|---|---|---|
| Lower clearance | 10 | 20 | 17 |
| Lower clearance | 100 | 200 | 184 |
| Higher clearance | 10 | 4.0 | 3.8 |
| Higher clearance | 100 | 40 | 34 |
For each row, calculate \(100(Y_{\mathrm{red}}-Y_{\mathrm{orig}})/Y_{\mathrm{orig}}\). The results are \(-15.0\%\), \(-8.0\%\), \(-5.0\%\), and \(-15.0\%\), respectively. The reduced model exceeds the 10% criterion at the 10 mg dose under lower clearance and at the 100 mg dose under higher clearance. It meets the criterion in the other two cells.
The correct conclusion is not that the models agree overall because two of four comparisons pass. Under the stated criterion, there are two material discrepancies in the tested domain, and each should be reported with its dose and scenario. Further investigation may determine whether the changed predictions matter for the intended use, but averaging these results would conceal where the reduction changed predictions.
Justifying the Paired Difference
The paired difference is useful because both predictions refer to the same evaluation condition. If the goal is to assess prediction preservation, changing the dose or physiological inputs between models would confound the comparison with those input changes. At a fixed condition \(x\), subtracting the original prediction from the reduced prediction leaves the model-to-model discrepancy for that condition:
This is a direct consequence of holding \(x\) fixed in both terms; it is not an estimate of a population effect or a test statistic with a sampling distribution. If an output is positive and the aim is to express the change relative to the original prediction, division by \(Y_{\mathrm{orig}}(x)\) yields \(r(x)\). The denominator also explains why relative differences become unstable near zero: a fixed absolute difference divided by a very small reference value can produce an arbitrarily large percentage. In that case, retain the absolute difference or use a scientifically meaningful nonzero scale.
Diagnostics, Alternatives, and Limitations
Inspect plots of both time courses and their pointwise differences, alongside tabulated endpoint summaries across conditions. Check for systematic direction, isolated peaks, shifts in peak timing, and discrepancies concentrated in particular doses or physiological scenarios. Confirm that the time grid is dense enough for the features of interest and that integration uses the same interval and method. If numerical solver settings differ, repeat with compatible settings and check that solver error is small relative to the discrepancy being evaluated.
Different comparison approaches answer different questions. A single endpoint such as AUC is compact but can hide time-local changes. Pointwise differences preserve timing but can be cumbersome across a large domain. RMSD summarizes overall discrepancy on a specified grid but can dilute a localized failure and depends on the grid and scale. Maximum absolute or relative difference highlights the worst evaluated point but may be sensitive to a point with little practical importance. A sound report combines measures selected for the intended predictions, and explains why their scales and thresholds are appropriate.
These checks verify preservation of selected predictions, not every possible output or condition. They also do not quantify uncertainty in parameters, inputs, or future predictions. The comparison is only as informative as its prespecified domain, numerical accuracy, and chosen materiality criteria. If model agreement is required for a critical decision, evaluate the relevant decision quantity directly rather than assuming agreement in a related summary guarantees it.
Common Pitfalls
- Checking only one dose or endpoint. Agreement at one condition does not demonstrate agreement across the domain. Include the doses, times, and physiological scenarios relevant to intended use.
- Using a relative difference near zero. Division by a tiny original prediction can make a percentage unstable or undefined. Report the absolute difference or use a justified nonzero scale.
- Trusting an average summary alone. A small RMSD or AUC difference can coexist with a material deviation at a particular time or scenario. Inspect local results as well.
- Treating a tolerance as universal. A percentage threshold is a decision criterion, not a law of PBPK modeling. Justify it for the output and use case.
- Calling model agreement validation. Similar predictions show that reduction preserved selected outputs; they do not show that those outputs match biological reality.
Check Your Understanding
Work through these questions, then compare your reasoning with the solutions.
- The original model predicts 5.0 mg/L and the reduced model predicts 4.6 mg/L. What are the absolute and relative differences?
- Why is the relative difference undefined when both models predict zero?
- Two models have similar AUC values but different concentrations at the time of peak. What does the AUC comparison establish, and what does it not establish?
- A reduced model passes a 10% criterion at three of four prespecified scenario-dose combinations. Is it accurate to report that it passes the criterion throughout the domain? Explain.
Solutions. 1. The absolute difference is \(4.6-5.0=-0.4\) mg/L. The relative difference is \((-0.4/5.0)\times100=-8.0\%\), so the reduced prediction is 8.0% lower. 2. The relative difference divides by the original prediction, and \(0/0\) is undefined; report the zero absolute difference instead. 3. Similar AUCs support agreement in integrated exposure over the specified time interval and integration method, but do not establish agreement in peak concentration, timing, or curve shape. 4. No. One of the four comparisons fails the stated criterion, so report the specific dose-scenario combination and do not claim the criterion is met throughout the domain.