Why Compare Plausible Mechanism Sets?
A QSP model represents a selected set of biological mechanisms within a defined boundary. When a mechanism is uncertain or its relevance is unresolved, a single model structure can make the decision appear more definite than the underlying knowledge warrants. Comparing plausible structures asks a focused question: does the decision conclusion persist when the model includes or excludes a scientifically defensible mechanism?
The comparison is about decision stability, not about declaring one structure true. A prediction can change numerically without changing the decision, or change enough to reverse the decision classification. The distinction matters when model outputs are used to assess target attainment, compare interventions, or evaluate a safety criterion.
Let \(j=1,\ldots,J\) index a finite set of candidate structures. Each structure must represent the same intervention or comparison, population, time horizon, and decision context. Let \(Y_j\) be its prediction on a common decision-relevant scale, and let \(c\) be a specified threshold. For a rule in which reaching or exceeding the threshold counts as attainment, the structure-specific decision is \(Y_j\geq c\). Define the margin \(m_j=Y_j-c\); its units are the same as those of \(Y_j\).
Set Up a Fair Structural Comparison
Before comparing predictions, specify what changes across structures and what stays fixed. A useful comparison changes the uncertain mechanism set while keeping the question and its interpretation aligned. If one structure predicts a concentration and another predicts a percentage response, those outputs cannot be compared directly until each is mapped to a common, justified decision quantity.
Parameter values also need a defensible treatment. For each structure, parameters should be estimated, constrained, or varied in a way appropriate to that structure and the available evidence. Reusing a parameter value that has a different meaning after a mechanism is added can make the comparison misleading. Conversely, refitting each structure without a consistent data target can confound structural differences with differences in calibration. The goal is not to force identical parameter values, but to make the model predictions comparable for the stated decision.
The candidate set should be explicit. It may include a base structure and alternatives that add or remove a feedback, pathway, or exchange process judged uncertain but plausible. It should not be treated as a catalogue of every imaginable biological story. The conclusions apply only to the structures examined; they do not prove that an unconsidered mechanism could not change the result.
This result depends on the stated inequality. A prediction exactly at the threshold is classified as attainment because the rule uses \(\geq\), not \(>\). Thus, whether structures agree is determined by their classifications, rather than by whether their numerical margins have the same sign under an informal convention that treats zero as neither positive nor negative.
Worked Examples
Worked Example: Does a Response Target Remain Attainable?
Hypothetical illustration. A team evaluates whether a dose is predicted to produce at least 60% biomarker reduction at a specified time. Three plausible structures differ in whether an uncertain feedback mechanism is included. Their predictions are 62%, 58%, and 64%. For this decision, attainment means a predicted reduction of at least 60%.
The threshold is \(c=60\%\). The structure-specific margins are:
The first and third structures classify the dose as attaining the target; the second classifies it as not attaining the target. Since at least one margin is negative and at least one is nonnegative, the decision is not robust across these three structures. This does not establish which structure is correct. It identifies that this decision depends on the structural assumption represented by the alternatives.
Worked Example: Interpreting a Prediction Exactly at the Threshold
Hypothetical illustration. A model-based safety rule classifies a regimen as acceptable when its predicted safety margin is at least zero. Two plausible structures predict margins of \(-0.03\) and \(0.00\), respectively, in the same dimensionless margin units.
The classifications follow directly from the rule. The first margin is negative, so that structure predicts the regimen is not acceptable. The second margin equals zero, so that structure predicts it is acceptable under the stated “at least zero” rule. The structures therefore disagree.
In this case, \(\min_j m_j=-0.03<0\) and \(\max_j m_j=0.00\geq0\), which meets the disagreement condition. It would be incorrect to say the structures agree because neither margin is strictly positive. The decision rule treats zero as attainment, so classifications—not informal labels for the signs—settle the comparison.
Worked Example: Does the Preferred Intervention Change?
Hypothetical illustration. An intervention comparison uses predicted improvement in a common clinical-response scale. For each structure, define \(\Delta_j=Y_{A,j}-Y_{B,j}\), where \(Y_{A,j}\) and \(Y_{B,j}\) are the predicted improvements under interventions A and B. The decision rule selects A when \(\Delta_j>0\), and B when \(\Delta_j<0\). A tie at zero is reported as no predicted preference.
Three plausible structures predict \(\Delta_1=4.0\), \(\Delta_2=1.5\), and \(\Delta_3=-0.8\) response-scale units. The first two favor A because their differences are positive. The third favors B because its difference is negative. Therefore the preferred intervention changes across these structures; the ranking is not structurally robust.
The range of these predictions is \(4.0-(-0.8)=4.8\) response-scale units, but range alone does not define robustness. The key decision information is that the predictions cross zero, the preference boundary. If instead every \(\Delta_j\) were positive, the predicted preference for A would be stable even if the numerical differences varied substantially.
How to Compare and Diagnose Structural Results
A compact comparison table can prevent a wide range of predictions from obscuring the decision itself. Report, for each structure, the mechanism difference, the common output, its threshold margin, and the resulting classification. Also report the range of predictions when its magnitude is relevant. A range summarizes spread; it does not replace the decision rule or explain why predictions differ.
Specify the population, intervention or contrast, time point or horizon, output, and decision threshold or ranking rule.
Describe which uncertain mechanisms differ and why each included or excluded mechanism is scientifically plausible for this comparison.
Use a common output definition and consistent scenarios. Check that units, time points, and parameter interpretation match across structures.
Calculate margins or pairwise contrasts and classify each structure, including predictions exactly on a threshold.
State whether the classifications agree, how much predictions vary, and which candidate structures the conclusion covers.
Several diagnostics help determine whether an apparent structural effect is interpretable. First, verify that each simulation completed successfully and that conservation, positivity, and other model-specific checks relevant to the simulated system have not been violated. A failed or numerically unstable simulation is not a valid structural prediction. Second, compare trajectories or intermediate outputs as well as the final decision quantity when feasible. This can reveal whether a difference arises from the mechanism being tested, an unintended change in an initial condition, or inconsistent implementation.
A useful comparison changes one structural feature at a time when the scientific question permits it. If several mechanisms change together, the result can still show that the broader alternative structure affects the decision, but it cannot isolate which change caused the effect. This is a diagnostic principle, not a requirement that all plausible structures be simple nested variants.
Robustness Is Not Model Averaging
A single preferred structure gives one model-conditional decision. Comparing several structures shows whether that decision survives the alternatives considered. These are different summaries: agreement across structures is evidence of structural stability within the candidate set, while disagreement identifies a decision that is structure-sensitive.
One could calculate a weighted average of predictions only if the weights have a defensible interpretation for the intended analysis. Without justified weights, an average can hide meaningful disagreement. For example, predictions on both sides of a threshold may average to exactly the threshold even though the individual structures imply different classifications. Such an average is not evidence that the decision is robust.
Structural robustness is also distinct from parameter uncertainty. A model can be stable across mechanism sets at one selected parameterization but sensitive to parameter variation within each structure. Or it can be sensitive to structure even when parameter uncertainty within each structure is small. These sources of variation should not be conflated: the present comparison asks specifically whether conclusions persist across the plausible structures chosen for examination.
Limitations and Common Pitfalls
- Overclaiming from agreement. If every examined structure reaches the same decision, the conclusion is robust only across that examined set. It does not show that the structures are correct or that all plausible mechanisms were considered.
- Calling numerical similarity decision stability. Predictions can be close yet fall on opposite sides of a sharp threshold. Apply the rule to each prediction rather than judging robustness by visual similarity alone.
- Ignoring the boundary convention. “At least” and “greater than” are not interchangeable. State whether equality meets the criterion and classify zero margins accordingly.
- Comparing incompatible outputs. A difference in units, time horizon, population, or outcome definition can masquerade as a structural effect. Align the estimand before interpreting model differences.
- Treating disagreement as proof of failure. Disagreement does not by itself invalidate the models. It signals that the decision depends on a structural assumption within the candidate set and that this dependence should be made visible.
The practical value of this analysis is therefore not to eliminate uncertainty by selecting a convenient structure. It is to tell decision-makers whether the conclusion is invariant to specified, plausible changes in the mechanistic boundary, and to identify when it is not.
Check Your Understanding
For each problem, apply the stated rule and show whether the decision is robust across the candidate structures.
- A target is attained when predicted response is at least 30%. Three structures predict 31%, 30%, and 28%. Are the classifications stable?
- A safety decision is acceptable when the safety margin is nonnegative. Four structures predict margins of \(0.2\), \(0.0\), \(0.05\), and \(0.1\). What is the conclusion?
- For an intervention comparison, the rule favors A when \(\Delta=Y_A-Y_B>0\), favors B when \(\Delta<0\), and reports no preference at zero. Predictions are \(2.0\), \(0\), and \(-1.0\) units. Do the structures give one common preference?
- All candidate structures predict target attainment, but their predictions range from 41% to 75% for a threshold of 40%. What does agreement establish, and what does it not establish?
Solutions. (1) The margins are \(31\%-30\%=1\) percentage point, \(30\%-30\%=0\) percentage points, and \(28\%-30\%=-2\) percentage points. The first two meet the threshold; the third does not. Classifications are not stable. (2) Every margin is nonnegative, including the margin equal to zero. All four structures classify the decision as acceptable, so the classification is stable across this set. (3) The positive value favors A, zero gives no preference, and the negative value favors B. The structures do not give one common preference. (4) Every prediction exceeds 40%, so attainment is stable across the examined structures. The 34-percentage-point range shows that the numerical predictions differ; agreement does not establish that the structures are true, that predictions are accurate, or that unexamined structures would also attain the target.