Credibility Assessment for Mechanistic Models

1. What Is Model Credibility?

Model credibility is the degree of justified trust in a model's ability to produce sufficiently reliable predictions for a particular intended use. The important phrase is for a particular intended use.

A mechanistic model is not simply "credible" or "not credible" in the abstract. A model may provide useful evidence for one question while requiring additional evidence before it can support a different, higher-consequence question.

This distinction is especially important for quantitative systems pharmacology (QSP), physiologically based pharmacokinetic (PBPK), PK/PD, disease-progression, and other mechanistic models. Such models can integrate biological knowledge and experimental data to make predictions in situations where direct experimentation may be difficult or impossible.

2. Why Does Credibility Matter?

Mechanistic models are often used precisely because the relevant biological system cannot be completely observed. A model may combine information from molecular experiments, preclinical studies, clinical data, literature, physiological databases, and prior knowledge.

The resulting predictions can therefore influence decisions even when the model itself is not directly observable. Credibility assessment provides a structured way to ask whether the available evidence is sufficient for the decision being considered.

The current ICH M15 General Principles for Model-Informed Drug Development, finalized in June 2026, provides harmonized principles for planning, evaluating, and documenting MIDD evidence. It emphasizes the importance of defining the intended use of a model and documenting the evidence supporting that use.

The appropriate credibility activities depend on the role of the model, the consequences of being wrong, and the evidence available from other sources.

3. State the Question of Interest

The first step is to state the question of interest. This is the scientific question, decision, or concern that the broader modeling and evidence-generation program is intended to address.

The question should be specific enough to identify what decision is being informed, but it may be broader than the model itself. Other evidence—such as clinical, nonclinical, biomarker, or experimental data—may also contribute to answering the question.

Weak formulation

"Use the QSP model to understand the disease."

More useful formulation

"Determine whether the proposed mechanism of action can plausibly explain the observed biomarker and clinical response over the studied dose range, and identify which biological processes most influence the predicted response."

The second formulation makes the scientific objective clearer and provides a basis for defining what the model must actually demonstrate.

4. Define the Context of Use

The context of use (COU) describes the specific role and scope of the model in addressing the question of interest. It is one of the central concepts in a risk-informed credibility assessment.

A useful COU specifies:

• the population or biological system being modeled;
• the intervention, perturbation, or exposure being evaluated;
• the quantities of interest;
• the prediction horizon;
• the intended decision or application;
• the data and other evidence that will be considered alongside the model; and
• important boundaries on how the model will be used.

A single mechanistic model can have multiple contexts of use. For example, the same PBPK model might be used to support a drug-drug interaction question in adults and separately to inform an initial pediatric dose. Those uses require separate credibility considerations because the populations, predictions, and decision consequences differ.

5. Assess Model Risk

Model risk is the possibility that reliance on the model or its predictions could contribute to an incorrect decision and an adverse consequence.

A useful risk-informed framework considers two major dimensions:

1. Model influence: how much the model contributes to the overall decision.
2. Decision consequence: what could happen if the decision informed by the model is wrong.

These dimensions are assessed for the particular context of use. A model used as minor supportive evidence may have lower model influence than a model whose prediction is being used in the absence of directly relevant clinical evidence.

The purpose of this assessment is not to assign a permanent label to the model. It determines the amount and type of credibility evidence that should be generated for the specific use.

6. Verification and Validation Are Different

Two words are often used together but answer different questions: verification and validation.

Verification can establish that software correctly implements the equations. It cannot by itself establish that those equations are appropriate for the biological question.

Conversely, a model can appear to predict observed data reasonably well while containing implementation errors that happen to be masked by parameter adjustments or compensating errors.

7. Verify the Computational Implementation

Mechanistic models can contain hundreds or thousands of equations, parameters, initial conditions, logical rules, and numerical operations. Software implementation is therefore itself an important source of uncertainty.

Useful verification activities may include:

• software quality assurance and version control;
• independent review of equations and implementation;
• unit testing;
• testing limiting or analytically solvable cases;
• comparison with independently implemented calculations;
• mass-balance or conservation checks;
• dimensional consistency checks;
• numerical convergence testing;
• solver tolerance and stability assessment; and
• testing of parameter bounds and unusual input conditions.

Example: mass balance

Suppose a model represents a conserved quantity distributed across several compartments. If there are no external sources or sinks, the total amount should satisfy:

A failure of this relationship may indicate an implementation error, numerical problem, or incorrect model specification.

Verification should be proportional to the model's complexity and intended use. High-consequence applications generally warrant more extensive and documented verification.

8. Verify Numerical Solutions

Even when model equations are correct and software is implemented correctly, numerical methods can introduce error.

Many mechanistic models are represented by systems of ordinary differential equations:

where \(\mathbf{x}\) represents model states, \(\mathbf{u}\) represents inputs, and \(\boldsymbol{\theta}\) represents model parameters.

The numerical solution is an approximation to the mathematical solution. Important checks include:

• solver convergence;
• sensitivity to integration tolerances;
• time-step dependence;
• stability under relevant parameter combinations;
• comparison against analytical solutions when available; and
• comparison with an independent numerical implementation when practical.

For example, if a numerical solution changes materially when solver tolerances are tightened, the prediction may be driven partly by numerical error rather than by the biological model.

9. Assess the Model Form

Model-form credibility concerns whether the equations, mechanisms, relationships, and structural assumptions are appropriate for the intended use.

Mechanistic models often contain assumptions that cannot be tested directly in every individual component. Examples include:

• which biological pathways are represented;
• which pathways are omitted;
• how compartments or physiological systems are connected;
• which relationships are linear or nonlinear;
• which biological processes operate on relevant time scales;
• how feedback mechanisms are represented; and
• whether parameters remain constant or vary dynamically.

Model-form assessment therefore combines empirical evidence with biological knowledge, prior literature, mechanistic plausibility, and sensitivity analysis.

10. Choose Validation Data That Test the Intended Prediction

Validation is strongest when the comparator data directly challenge the model in the domain relevant to the COU.

For mechanistic models, useful validation datasets may include:

• independent clinical studies;
• independent preclinical experiments;
• dose levels not used for model calibration;
• time points outside the calibration window;
• different physiological states;
• perturbation experiments;
• biomarker data;
• genetic or pharmacological interventions; and
• observations from related but appropriately comparable systems.

The key question is not simply whether the model was compared with data. It is whether the comparison provides evidence about the model's ability to make the prediction required by the COU.

Calibration versus validation

Suppose a model has parameters \(\boldsymbol{\theta}\) estimated using dataset \(D_{\mathrm{cal}}\). Predictions for the same dataset are not equivalent to predictions for an independent dataset \(D_{\mathrm{val}}\).

Using the same observations for both extensive calibration and apparent validation can make model performance look stronger than its true predictive performance.

11. Compare Predictions With Observations

Prediction-versus-observation comparisons should be tailored to the quantities of interest.

Graphical assessments are often especially informative because they can reveal systematic patterns that a single summary statistic may hide.

Depending on the model and COU, useful displays may include observed-versus-predicted plots, prediction intervals, time-course overlays, residual plots, visual predictive checks, calibration plots, and distributional comparisons.

12. Quantify and Communicate Uncertainty

A mechanistic model can produce a precise numerical prediction even when substantial uncertainty exists. Credibility assessment therefore requires attention not only to the model's central prediction but also to the uncertainty surrounding it.

Important sources include:

• Parameter uncertainty: uncertainty in estimated or literature-derived parameter values.
• Input uncertainty: uncertainty in model inputs such as physiological or experimental quantities.
• Structural uncertainty: uncertainty about the model form or omitted mechanisms.
• Measurement uncertainty: uncertainty in experimental or clinical observations.
• Numerical uncertainty: error introduced by numerical approximation.

A simple representation is:

where \(\boldsymbol{\theta}\) and \(\mathbf{x}\) may themselves be uncertain.

For a prediction \(Y\), uncertainty propagation can be expressed conceptually as:

where the Jacobian terms describe sensitivity of the prediction to uncertain parameters and inputs. In nonlinear models, simulation-based uncertainty propagation is often more appropriate than a simple linear approximation.

13. Use Sensitivity Analysis to Identify What Matters

Sensitivity analysis examines how model outputs change when inputs, parameters, or assumptions are changed.

For a model output \(Y\) and parameter \(\theta_i\), a local sensitivity can be represented as:

Scaled sensitivity measures are often easier to compare across parameters:

Sensitivity analysis can help identify:

• parameters that strongly influence the quantity of interest;
• assumptions that materially affect the decision;
• parameters for which additional experiments would be informative;
• mechanisms that dominate model predictions; and
• regions where model extrapolation may be particularly uncertain.

Sensitivity analysis is not a substitute for validation. A parameter can be highly influential without being correct, and a poorly identified parameter can sometimes have little influence on a particular prediction.

14. Assess Applicability to the Context of Use

A central question in credibility assessment is whether the available validation evidence is relevant to the actual intended application.

Consider a QSP model developed using adult clinical data. If the model is subsequently used to predict treatment response in pediatric patients, several questions arise:

• Are the relevant biological mechanisms expected to operate similarly?
• Are physiological parameters appropriately represented?
• Are disease mechanisms comparable?
• Is the treatment exposure range within a supported domain?
• Were the relevant mechanisms independently tested?
• Does the validation dataset challenge the model in the same dimensions as the proposed prediction?

Applicability therefore connects validation evidence to the COU. Evidence can be technically rigorous yet have limited relevance if it does not test the prediction that matters.

15. Build a Credibility Assessment Plan

A credibility assessment is stronger when the evidence-generation strategy is planned before the model is used to support the decision.

A practical plan can include the following elements:

1. Question of interest. State the scientific or decision question.
2. Context of use. Define exactly how the model will contribute.
3. Model influence. Describe the model's role relative to other evidence.
4. Decision consequence. Describe the implications of an incorrect decision.
5. Model risk. Determine the overall risk associated with the COU.
6. Credibility goals. Define what evidence would be considered sufficient.
7. Verification activities. Specify software, numerical, and calculation checks.
8. Validation activities. Specify comparator datasets and predictive assessments.
9. Uncertainty analysis. Identify relevant sources and propagation methods.
10. Applicability assessment. Explain why the validation evidence supports the COU.
11. Documentation. Record assumptions, datasets, methods, results, limitations, and deviations.

16. Worked Example: A QSP Model for Dose Selection

Consider a hypothetical QSP model for an inflammatory disease. The model describes target engagement, downstream biomarker signaling, and a clinical response variable. The development team wants to use the model to inform selection of doses for a Phase II study.

Step 1: Question of interest

Question: Which dose range is expected to provide biologically meaningful target engagement and response while remaining within the exposure range supported by available safety information?

Step 2: Context of use

The model will simulate biomarker and clinical-response trajectories over the proposed dose range in adults with the target disease. Model predictions will be used as one component of the evidence informing Phase II dose selection.

Step 3: Model influence

Suppose clinical PK and early biomarker data are available, along with preliminary clinical response data. The QSP model provides an integrative mechanistic interpretation rather than the sole source of evidence for dose selection.

Step 4: Decision consequence

An incorrect dose-selection decision could result in an inefficient Phase II study, inadequate exposure, or unnecessary exposure to a dose that does not provide sufficient benefit. The consequence therefore needs to be explicitly considered when setting the credibility goals.

Step 5: Verification

The team verifies:

• equation implementation against the model specification;
• mass-balance relationships;
• units and dimensional consistency;
• parameter transformations;
• solver convergence;
• special cases with known behavior; and
• independent reproduction of selected calculations.

Step 6: Calibration

Model parameters are estimated or constrained using appropriate PK, biomarker, and clinical data. Calibration datasets are documented separately from datasets reserved for predictive assessment.

Step 7: Independent predictive assessment

The model is evaluated against observations not used for the primary calibration. For example, the model may predict a biomarker trajectory at a dose level that was not used to estimate key parameters.

The analysis examines not only average prediction error but also whether there are systematic discrepancies across dose, time, or patient subgroup.

Step 8: Sensitivity and uncertainty

Suppose the predicted clinical response is particularly sensitive to two parameters describing target turnover and downstream pathway activity. The team performs uncertainty propagation over plausible values for these parameters and identifies how much the dose-selection conclusion changes.

Step 9: Credibility conclusion

The final conclusion should describe the evidence and its limitations rather than simply declaring that the model is "validated." For example, the report might state that the verification activities were completed, specified validation datasets were evaluated, predictions were consistent with predefined performance expectations, and remaining uncertainty is concentrated in particular biological parameters.

17. Define Credibility Goals and Acceptance Criteria

Credibility goals should be connected to the scientific purpose of the model. Where quantitative acceptance criteria are appropriate, they should be defined before evaluating the final results when feasible.

Examples of possible criteria include:

The exact numerical thresholds should be justified scientifically. A generic rule such as "the model must predict within 10%" is not automatically appropriate for every biological quantity.

18. Common Credibility Assessment Mistakes

Mistake 1: Treating fit as validation

A model can fit the calibration data extremely well while making poor predictions outside the calibration setting.

Mistake 2: Validating against the same data used for calibration

Using the same observations to estimate parameters and demonstrate predictive performance can produce optimistic assessments.

Mistake 3: Ignoring model influence

The evidence needed for a model used as supplementary interpretation may differ from the evidence needed when the model provides a major component of a consequential decision.

Mistake 4: Ignoring structural uncertainty

Uncertainty analysis that varies parameters while keeping the model structure fixed does not fully address uncertainty about whether the mechanisms and equations themselves are appropriate.

Mistake 5: Validating irrelevant outputs

A model may reproduce plasma concentration well while providing little evidence for its prediction of a downstream clinical response. Validation should focus on the quantities of interest in the COU.

Mistake 6: Treating credibility as permanent

Changes to the model, population, intervention, prediction horizon, or intended use can change the credibility assessment.

Mistake 7: Overstating what the model proves

A mechanistic model provides evidence conditional on its assumptions and supporting data. It does not automatically establish that a proposed biological mechanism is true simply because the model can reproduce observed observations.

19. Credibility Assessment in QSP Models

QSP models present particular credibility challenges because they often integrate multiple biological scales and data types.

A QSP model may contain:

• molecular binding and target engagement;
• cellular signaling pathways;
• biomarker turnover;
• immune-cell dynamics;
• disease progression;
• drug exposure;
• feedback mechanisms; and
• clinical endpoints.

Validation therefore needs to be considered across multiple levels of the model.

A strong QSP credibility argument therefore does not necessarily require every parameter or every pathway to have the same degree of empirical support. Instead, the evidence should be connected to the predictions and decisions for which the model will be used.

20. Credibility and Model-Informed Drug Development

Model-informed drug development increasingly uses quantitative models to integrate evidence and inform development decisions. The current ICH M15 guidance provides general principles for planning, model evaluation, and documentation of MIDD evidence.

The risk-informed credibility concept complements this approach by emphasizing that the required evidence depends on the intended use and consequences of relying on the model.

For a regulatory-facing model, documentation should make it possible for reviewers to understand:

• the question being addressed;
• the precise context of use;
• the model's role relative to other evidence;
• the assumptions and limitations;
• the verification activities performed;
• the validation datasets and their relevance;
• the predictive performance;
• uncertainty and sensitivity analyses;
• the applicability of the evidence to the COU; and
• how the final credibility assessment supports the intended use.

The FDA's June 2026 final ICH M15 guidance describes a harmonized framework for MIDD evidence and explicitly addresses model planning, evaluation, documentation, and regulatory interactions.

21. What Should a Credibility Report Contain?

A practical credibility report can be organized around the following structure:

1. Scientific question and decision.
2. Context of use.
3. Model description.
4. Model assumptions.
5. Data sources and provenance.
6. Parameter estimation and calibration.
7. Software verification.
8. Numerical verification.
9. Validation strategy.
10. Validation results.
11. Uncertainty analysis.
12. Sensitivity analysis.
13. Applicability assessment.
14. Known limitations and residual uncertainty.
15. Credibility assessment and intended use.

Traceability is particularly important. The report should make it possible to connect each important conclusion to the underlying model version, data, analysis, and computational implementation.

22. A Practical Credibility Assessment Workflow

1. State the question of interest.
2. Define the context of use.
3. Assess model influence and decision consequence.
4. Determine model risk.
5. Define credibility goals.
6. Plan verification and validation activities.
7. Generate or identify appropriate evidence.
8. Quantify uncertainty and evaluate sensitivity.
9. Assess applicability to the COU.
10. Integrate the evidence into an overall credibility assessment.
11. Document limitations and residual uncertainty.
12. Update the assessment when the model or intended use changes.

This process is iterative rather than strictly linear. A validation result may reveal a structural limitation, which may lead to model revision, new calibration, additional validation, or a narrower COU.

Where to Go Next

A natural progression is to study model verification and validation in QSP in greater detail, followed by parameter uncertainty, sensitivity analysis, predictive qualification, external validation, and model updating.

The next tutorial can build directly on this framework by examining verification, validation, and qualification of QSP models and showing how different evidence streams contribute to confidence in mechanistic predictions.

References