Why a Reduced Model Must Be Tested Across Conditions
A reduced QSP model is useful only insofar as it preserves predictions relevant to its intended use. Agreement with the full model at one dose and one baseline condition is not enough: elimination of a state or mechanism may have little effect there but become consequential at a higher dose, a shifted baseline, or a pathway perturbation. Testing therefore means a structured comparison of both models under matched conditions, with errors calculated for outputs that matter to the scientific or decision question.
The full model serves as a reference for this comparison, not as proof of biological truth. A reduced model that closely reproduces it has demonstrated agreement with that reference over the tested scenarios; it has not thereby been validated against every patient, experiment, or untested intervention. The conditions and endpoints used for comparison should reflect the intended scope of use.
For example, endpoints might include a biomarker concentration at a specified time, peak response, area under the curve (AUC), or a threshold-crossing time. They do not answer interchangeable questions. AUC can agree while peak magnitude differs, and a concentration at one time can agree while the preceding time courses diverge. Select endpoints before inspecting the comparison results, and include the outputs needed for the model's intended use.
Comparison Rules and Error Measures
Let \(y_{F,k,s}\) be endpoint \(k\) predicted by the full model in scenario \(s\), and \(y_{R,k,s}\) the corresponding reduced-model prediction. Both must refer to the same defined quantity and units. The signed error identifies direction: a positive value means the reduced model predicts a larger endpoint, while a negative value means it predicts a smaller one.
A relative error expresses the absolute discrepancy as a fraction of the reference prediction. It is useful when proportional discrepancy is meaningful and the reference endpoint is not zero or close to zero.
When the reference endpoint can be near zero, relative error can become arbitrarily large or undefined. A more stable assessment uses a tolerance \(T_k>0\), chosen in the endpoint's units and representing an acceptable discrepancy for the intended application. The tolerance-scaled error is the absolute error divided by that tolerance.
The tolerance should be set using the endpoint's role and meaningful scale, rather than chosen after seeing which value makes the model pass. If endpoints have different units or consequences, they generally need different tolerances. Relative error may also be reported, but it should not replace an interpretable, prespecified tolerance when values can approach zero.
Designing the Scenario Grid
A useful test grid varies the conditions that could expose the simplifications in the reduced model. At minimum, consider the intended dose range, relevant baseline conditions, and mechanistic perturbations that alter the processes affected by reduction. A perturbation could, for instance, inhibit or remove a pathway component that was lumped, eliminated, or fixed in the reduced representation. Include combinations when the intended use involves them: a perturbation at baseline does not automatically establish agreement for that perturbation at high dose.
For each pair of models, make inputs, initial conditions, parameter values or mapping rules, and endpoint definitions consistent. If the reduced model has been refit, distinguish the scenarios used for fitting from those used to evaluate agreement. Reusing only calibration conditions can make a reduced model appear reliable without testing whether it preserves predictions beyond those conditions.
The comparison assumes that the full model is an appropriate reference for the reduction question, that both implementations represent the stated scenario, and that numerical solution errors are small compared with the tolerances. If a reduced model changes the definition of an output, the comparison is not matched until a scientifically justified mapping to the same endpoint has been specified. These checks are part of interpreting an error, not optional housekeeping.
Name the prediction task, endpoints, time windows, dose range, baseline conditions, and perturbations that define the intended scope.
Choose an acceptable discrepancy for each endpoint in meaningful units, including a rule for endpoints near zero.
Apply the same inputs and comparable initial conditions to both models, then compute the same endpoints.
Report signed errors and tolerance-scaled errors, and locate which scenarios pass or exceed the criterion.
Worked Comparisons
Worked Example: A Dose Comparison with Two Endpoints
Suppose two hypothetical models are compared at a dose of 10 mg/kg under the same baseline condition. The full model predicts a biomarker AUC of 2,400 ng·h/mL and a peak concentration of 40 ng/mL. The reduced model predicts an AUC of 2,280 ng·h/mL and a peak of 43 ng/mL. Prespecified tolerances are 200 ng·h/mL for AUC and 2 ng/mL for peak concentration.
For AUC, the signed error is \(2{,}280-2{,}400=-120\) ng·h/mL, so the reduced model underpredicts AUC by 120 ng·h/mL. Its absolute and tolerance-scaled errors are
Thus, the AUC discrepancy is within its tolerance. For peak concentration, the signed error is \(43-40=3\) ng/mL and
The peak discrepancy exceeds its tolerance. The reduced model therefore does not meet the criterion for both endpoints, even though its AUC agrees within tolerance. The signed errors also show that it predicts a lower AUC but a higher peak.
Worked Example: Baseline Agreement Does Not Guarantee Perturbation Agreement
In a hypothetical pathway model, the endpoint is percent inhibition of a downstream response at a fixed time. At baseline and a low dose, the full and reduced models predict 20% and 19%, respectively. At a high dose, they predict 60% and 45%. Under a pathway-blocking perturbation at the high dose, they predict 35% and 20%. Use an absolute tolerance of 5 percentage points for each comparison.
At baseline and low dose, the absolute discrepancy is \(|19-20|=1\) percentage point, and the scaled error is \(1/5=0.20\). That comparison passes. At high dose without the perturbation, the discrepancy is \(|45-60|=15\) percentage points, giving \(15/5=3.00\), which fails. Under the high-dose pathway perturbation, the discrepancy is \(|20-35|=15\) percentage points and its scaled error is also \(3.00\), so it fails as well.
The low-dose baseline result alone would suggest close agreement, but it does not support use at high dose or under pathway blockade. In the latter comparisons, the reduced model predicts substantially less inhibition than the full model. The conclusion is limited to these tested scenarios and endpoints; it is not a claim that every dose or perturbation will fail.
Worked Example: Handling an Endpoint Near Zero
Consider a hypothetical adverse biomarker increment for which the full model predicts 0.20 units and the reduced model predicts 0.30 units under one perturbation. Let the prespecified acceptable discrepancy be 0.50 units. The absolute error and scaled error are
This comparison passes the stated tolerance. The relative error against the full prediction is \(0.10/0.20=0.50\), or 50%. That percentage sounds large, but it does not overturn the pass under the prespecified absolute criterion. If the reference prediction were exactly zero, the relative-error formula would be undefined. This example shows why the error measure must match the endpoint's scale and scientific meaning.
Justification, Diagnostics, and Alternatives
Tolerance scaling gives a simple way to compare heterogeneous endpoints without treating their raw numerical magnitudes as comparable. For \(T_k>0\), dividing both sides of the absolute-tolerance condition by \(T_k\) gives
Thus the scaled error does not create a new biological criterion; it expresses the chosen absolute criterion on a common scale. A value of 0.5 uses half the allowed discrepancy, while a value greater than 1 exceeds it. The maximum \(q_{k,s}\) across required comparisons can summarize the worst case, but it should be reported alongside the endpoint and scenario that produced it. An average can hide an important failure.
When results are unexpected, first check that the two simulations used the intended dose, perturbation, initial state, units, and time window. Then inspect signed errors across the scenario grid. A smoothly growing error with dose may indicate loss of a dose-dependent mechanism; a large error only after a perturbation may indicate that a removed or aggregated process becomes important in that condition. A sudden endpoint discrepancy can also arise from different event definitions or from numerical integration settings. Tightening solver tolerances and checking whether the discrepancy changes helps distinguish numerical error from a model-reduction effect.
| Comparison approach | What it reveals | Main limitation |
|---|---|---|
| Endpoint-specific errors across scenarios | Whether defined outputs meet tolerances under particular doses, baselines, and perturbations | Can miss disagreement in unselected aspects of the trajectory |
| Pointwise trajectory comparison | When and how predicted time courses diverge | Requires a time-grid or trajectory-level criterion and can obscure which endpoint matters |
| One aggregate score across all conditions | A compact summary of overall discrepancy | May conceal a serious error in one endpoint or perturbation |
These approaches complement rather than replace one another. Endpoint-specific comparison is interpretable for a defined use, while trajectory comparison can expose timing or transient differences that selected endpoints miss. An aggregate score is convenient for screening, but it should not be the only evidence for agreement. None of these comparisons establishes agreement outside the tested range, and close agreement with the full model does not establish that either model matches observed biology.
Common Pitfall and Scope
- Wrong reasoning: “The models agree at baseline, so the reduction is valid.” Correction: Agreement supports only the tested endpoint and condition. Check doses, baseline variation, and mechanistic perturbations relevant to intended use.
- Wrong reasoning: “The average error is small, so every condition is acceptable.” Correction: Inspect endpoint-scenario errors individually; a large, consequential discrepancy can be diluted by many small errors.
- Wrong reasoning: “A large relative error always means failure.” Correction: Near zero, relative error can exaggerate or become undefined. Use a meaningful prespecified tolerance and report units.
- Wrong reasoning: “The full model is the truth.” Correction: It is the reference for assessing the reduction. Agreement does not substitute for evaluation against data or justify extrapolation to untested conditions.
The result of testing is therefore conditional: the reduced model either meets stated tolerances for specified endpoints over a specified scenario set, or it does not. If it fails, the pattern of errors helps identify where predictive validity is lost. If it passes, the tested range and comparison criteria define the scope of the evidence; broader use requires additional relevant scenarios.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- The full model predicts an endpoint of 80 units and the reduced model predicts 74 units. If the tolerance is 10 units, calculate the signed error and scaled error, and state whether the comparison passes.
- Why can a low average error across dose conditions be inadequate evidence of agreement?
- The full model predicts an endpoint of 0 units and the reduced model predicts 0.2 units. Why is relative error unsuitable, and what comparison can be used if the tolerance is 0.5 units?
- Name two checks that can help determine whether a large discrepancy reflects a mismatched simulation setup or a genuine reduction-related difference.
Solutions. 1. The signed error is \(74-80=-6\) units; the scaled error is \(|-6|/10=0.60\), so it passes. 2. An average can hide a large error in a particular dose, perturbation, or endpoint; inspect the individual comparisons and worst case. 3. Relative error divides by the full-model prediction of zero and is undefined. The absolute error is 0.2 units, so the scaled error is \(0.2/0.5=0.40\), which passes. 4. Check matched dose, perturbation, initial conditions, units, and endpoint definition; also tighten numerical solver tolerances to assess whether numerical error contributes.