Why Use a Risk-Informed Matrix?
A PBPK model can be correct in its implementation yet unsuitable for a particular decision. Conversely, a model may be adequate for a low-consequence exploratory comparison even when some uncertainties remain. A credibility matrix helps connect these two facts: it organizes the evidence available for a model and makes the depth of evidence proportionate to the risk of relying on its predictions.
The earlier material on context of use and model risk established that the intended decision matters: risk depends on the consequence of a decision being wrong and on how much the model can influence that decision. The matrix does not replace that assessment. It turns the resulting risk judgment into an organized plan for examining evidence in four domains: code, calculations, biological assumptions, and prediction performance. It also shows where evidence is missing, weak, or inconsistent.
The matrix must be tied to a specific model version, intended use, and decision. The same PBPK model may warrant different evidence for two uses: exploratory ranking of compounds and selection of a clinical dose are not interchangeable claims. A matrix assembled without naming the decision can look complete while asking the wrong questions.
Matrix Structure and Risk Tiers
Use risk tiers such as low, moderate, and high only as qualitative summaries of the assessed risk. They are not universal categories with fixed meanings. The team should state why a tier was assigned, using the decision consequence and model influence for the context of use. A high-consequence decision for which model predictions are decisive generally warrants deeper evidence than a screening analysis that only helps prioritize subsequent experiments.
The columns represent distinct questions. Code evidence asks whether the implemented code represents the intended model equations and logic. Calculation evidence asks whether numerical operations and computational settings produce reliable results. Biological-assumption evidence asks whether the model structure and physiological inputs are plausible and supported for the intended population and conditions. Prediction-performance evidence asks how well predictions compare with relevant observations, including whether the evidence is relevant to the use and whether uncertainty is represented appropriately.
| Risk tier | Code | Calculations | Biological assumptions | Prediction performance |
|---|---|---|---|---|
| Low | Basic traceable implementation checks and tests of key functions | Check solver settings and key balances for the quantities used | Document principal assumptions and their rationale | Compare with available relevant observations; state the scope of the comparison |
| Moderate | Add independent review and tests covering important branches and inputs | Test convergence and numerical sensitivity for decision-relevant outputs | Assess important assumptions and input ranges against evidence for the use | Use relevant data not solely used to tune the model; examine predictive discrepancies |
| High | Require systematic independent review, traceability, and tests of critical pathways | Demonstrate numerical stability and conservation where applicable under relevant conditions | Provide strong, use-specific support for influential biological assumptions | Seek stringent, decision-relevant predictive evaluation and explicit uncertainty analysis |
These cells describe a gradient of evidence intensity, not a mandatory checklist. For example, a low-risk use can still require intensive numerical checking if the solver behaves poorly in the relevant regime. A risk tier does not excuse a known failure in any domain. Instead, it guides where additional evidence is especially important and how much confidence a decision can reasonably place in existing evidence.
Why Evidence Depth Should Increase With Risk
The justification is decision-focused. If a prediction has little influence on an action, a prediction error may have little effect on the action. If the model strongly influences a consequential choice, the same size error can matter much more. Therefore, as either the consequence of error or the model’s influence increases, it becomes more important to test the links between implementation, computation, biology, and prediction. This is a qualitative prioritization principle; it does not imply that a universal numerical product of consequence and influence determines the evidence required.
The domains are connected but not interchangeable. Agreement with observations cannot by itself establish that the code is implemented correctly: compensating errors or calibration may produce a close fit. A code review cannot establish that the physiology is relevant to the target population. A plausible biological rationale cannot show that predictions are sufficiently accurate for a particular decision. A matrix prevents one attractive evidence source from silently standing in for all the others.
Specify the context of use, model version, population, intervention or exposure, and decision the prediction is intended to inform.
Document the decision consequence and the model’s expected influence, then assign a justified qualitative tier.
For each cell, specify the evidence sought, evidence observed, and any unresolved gap. Do not write only “adequate” without a reason.
Identify which gaps could affect the intended decision and whether the available evidence supports the claim at the stated risk level.
Worked Example: Low-Risk Compound Ranking
Hypothetical use. A team uses a PBPK model to rank four candidate compounds for which to run additional in vitro experiments. The ranking only determines experiment order; it does not select a dose or end a development program. The team judges both the consequence of a mistaken ranking and the model’s influence on a final decision to be low, because further experiments will follow. It assigns a low risk tier.
For the code cell, suppose the model has 12 unit tests for the absorption, distribution, metabolism, and excretion functions used in this ranking, and all 12 pass. The cell records the test coverage and code version; it does not claim that 12 passing tests prove correctness. For calculations, suppose the simulated mass-balance residual is 0.08% for each of the four compounds under the reported run settings. The team records the calculation and confirms that changing solver tolerance does not change the rank order.
For biological assumptions, the team documents the assumed enzyme pathway and uses hypothetical in vitro inputs measured for all four compounds. It flags that a transporter contribution is not represented. For prediction performance, it compares predicted and observed exposure summaries from a small, relevant data set: three of four compounds are within two-fold, and the fourth is 2.4-fold from its observed value. The comparison is informative for prioritization, but the small data set and the outlying compound limit the claim.
The resulting matrix is proportionate to an exploratory use: basic code checks, a numerical check, documented biological assumptions, and a relevant but limited performance comparison. The conclusion is not that the model is generally qualified. It is that the evidence may support this restricted ranking use, with the transporter omission and the 2.4-fold discrepancy visible as limitations.
Worked Matrix Applications
Worked Example: High-Influence Dose Decision
Hypothetical use. A PBPK prediction informs whether a 100 mg dose should be reduced in a target population. If the predicted exposure increase is large, the model could materially affect the dose decision, and an incorrect decision could cause clinically important harm. The team assigns high risk. In this hypothetical analysis, the predicted exposure ratio is 1.60 relative to the reference population, with a prediction interval from 1.30 to 1.90. A decision rule under consideration flags ratios above 1.25. Both the point prediction and the interval’s lower limit exceed 1.25, so the prediction has a direct potential to influence the action; this calculation does not establish that the model is credible.
The high-risk matrix therefore calls for more than a general statement that the model was tested. In the code cell, the team requests independent review of the implementation and traceability from model equations to code, with tests of the pathways affecting the target population. In the calculation cell, it requires solver convergence checks and numerical sensitivity analyses for the predicted ratio and interval, not merely for a convenient baseline simulation.
In the biological-assumption cell, the team identifies the assumptions most likely to affect the ratio, such as the representation of the relevant physiological change and the activity of the elimination pathway. It seeks support applicable to this population and documents remaining uncertainty. In the performance cell, it seeks comparisons against relevant observations that were not used solely to tune the model, examines whether predictions are systematically biased, and evaluates uncertainty in the decision-relevant exposure ratio.
The matrix does not automatically recommend a dose reduction. It specifies the evidence needed before relying on a high-influence prediction, and it makes clear that a prediction above a decision threshold is not itself evidence that the prediction is accurate. If, for example, the independent code review is complete but population-specific performance evidence is absent, that gap remains consequential even if the other cells contain substantial evidence.
Worked Example: Moderate-Risk Performance Evidence
Hypothetical use. A model is used to plan a follow-up study, but the study design remains subject to review and the model does not determine treatment. The consequence and influence are assessed as moderate. The team compares predictions with eight held-out observations from a relevant hypothetical data set. Six predictions are within two-fold of observation, and two are outside that range.
The descriptive calculation is \(6/8=0.75\), so 75% of these eight observations are within two-fold; \(2/8=0.25\), or 25%, are outside. These proportions describe this small set only. They are not universal performance thresholds, and they do not by themselves establish acceptable prediction performance. The matrix records the number and relevance of observations, the comparison method, and the two discrepancies, including their direction and whether they concern the study conditions of interest.
For this moderate-risk use, the team also records code tests and calculation checks for the model version that generated the predictions. It reviews the biological assumptions that could explain the two larger discrepancies and notes whether the study population is represented. If the two outliers occur under conditions central to the planned study, that is a more important gap than two outliers under conditions outside the use. Thus, cell interpretation depends on relevance, not only on a count.
Diagnostics, Alternatives, and Limitations
A matrix is going wrong when it gives every model the same evidence burden, regardless of decision risk; when a cell contains only a label such as “validated”; or when evidence from one domain is treated as proof for another. Other warning signs include an unspecified model version, no link between evidence and the intended use, missing treatment of contradictory results, and a high-risk claim supported mainly by evidence from a different population or prediction task.
A flat checklist is the main alternative. It can improve consistency by ensuring that recurring evidence types are considered, but it does not show which checks deserve greater depth for a high-consequence decision. Requiring exhaustive evidence for every use is another alternative. It may be disproportionate for exploratory work and can divert effort from the uncertainties that matter most. The risk-informed matrix preserves a common structure while scaling the evidence plan and making limitations visible. It should not be mistaken for an algorithm that converts completed cells into a pass.
The matrix also has limitations. Risk tiers and cell descriptions require judgment, and different reviewers may disagree unless the rationale is documented. Evidence availability can constrain what is feasible; a gap does not disappear because it is hard to fill. A compact table may hide important detail, so each cell should point to methods, results, model versions, or reports that support its summary. Finally, the matrix organizes credibility evidence but does not itself establish acceptance criteria, define an applicability domain, or resolve discordant evidence. Those judgments require context-specific reasoning.
Check Your Understanding
Answer each question using the matrix principles above. Brief solutions follow the questions.
- Why is a risk tier not a numerical score that can certify a model?
- A model has extensive code tests but no evidence comparing predictions with relevant observations. Which matrix distinction does this illustrate?
- In a hypothetical comparison, 9 of 12 held-out predictions are within two-fold of observation. Calculate the proportion and state one limitation of interpreting it.
- Why might two prediction errors outside a chosen range deserve different attention in a moderate-risk use?
Solutions. (1) The tier qualitatively summarizes decision consequence and model influence; it does not measure credibility or prove that evidence is sufficient. (2) Code implementation evidence and prediction-performance evidence address different questions, so strong evidence in one does not substitute for the other. (3) \(9/12=0.75\), or 75%. This proportion describes only those 12 observations and is not, by itself, a universal pass criterion. (4) Relevance differs: an error under a condition central to the intended use may expose an important gap, whereas an error under an irrelevant condition may have less bearing on that use.