Why Compare Models Side by Side?
A development decision can depend on predictions from several plausible models. For example, two exposure-response models may predict nearly the same average treatment benefit while placing that average on opposite sides of a go/no-go threshold. Conversely, models may predict noticeably different average benefits but still recommend the same action because both predictions meet the decision criterion. Reporting only an average prediction can hide either situation.
The purpose of a side-by-side comparison is to show what each model implies under a common decision question. The comparison is informative only if the models refer to the same target population, outcome definition, time horizon, interventions, and decision rule. Otherwise, apparent differences may reflect different questions rather than different model implications.
Set Up a Fair Comparison
Let \(m\) index a plausible model, \(a\) a candidate action, and \(Y\) a decision-relevant outcome. For example, \(a\) might specify a dose and \(Y\) might be a clinical benefit at a defined follow-up time. Each model implies a predictive distribution for \(Y\) under each action. A comparison should state which summary of that distribution is relevant: a mean, a probability of meeting a target, a quantile, or an expected utility.
For an outcome-based threshold rule, let \(\mu_m(a)\) be the model-specific predicted mean outcome under action \(a\), and let \(c\) be a prespecified threshold. If higher outcomes are desirable, a simple rule selects “go” when \(\mu_m(a)\) is at least \(c\), and “no-go” otherwise. For decisions among several actions, a utility rule selects the action with the highest expected utility under that model.
Here, \(\mathcal{A}\) is the set of available actions, \(U(a,Y)\) is the utility assigned to action \(a\) and outcome \(Y\), and \(E_m\) denotes expectation under model \(m\). The decision is conditional on the model and the specified utility. If a simpler threshold policy is used, it should be stated just as explicitly. Use the same utility or threshold for all models; changing the rule between models makes the action comparison uninterpretable.
Before comparing results, check that each model has been evaluated for the same population and scenario, with the same definitions and units. A model may have a different structure or parameterization, but the displayed quantity must answer the same question. State which model is being evaluated and show its prediction and action separately. Do not silently combine model-specific predictions into a single result.
Similar Means Can Produce Different Actions
A decision rule can change action abruptly at its threshold, even when predictions change only slightly. The following calculation explains why agreement in average outcomes does not guarantee agreement in action.
The predicted means differ by only \(0.10\), but they fall on opposite sides of the threshold. More generally, if the threshold \(c\) lies between the two means, then one model meets the mean-based criterion and the other does not. The action map is therefore not a smooth measure of predictive agreement: small changes in a prediction near the boundary can change the selected action. Far from the boundary, even larger prediction differences may leave the action unchanged.
Worked Example: Nearly Equal Benefits, Different Go/No-Go Decisions
Suppose two plausible models predict the mean improvement in a clinical score at 12 weeks for the same target population. A development policy says “go” if the mean improvement is at least \(1.75\) points. Model A predicts \(1.80\) points and Model B predicts \(1.70\) points. Their respective central 90% predictive intervals for an individual patient's improvement are \(0.70\) to \(2.90\) points and \(0.60\) to \(2.80\) points.
The predictions differ by \(1.80-1.70=0.10\) points. Applying the common rule gives:
The models agree closely in their predicted mean improvement, yet disagree in the selected action. The intervals overlap substantially, but that overlap does not remove the disagreement: the stated policy is based on the model-specific mean, not on whether the predictive intervals overlap. Under each model, the central 90% predictive interval describes a range for individual outcomes under that model's assumptions; it is not an interval for the mean or a direct measure of agreement between models.
A clear report would show both model-specific means, both intervals, the \(1.75\)-point threshold, and the resulting actions. It should also flag that the decision is sensitive to which plausible model is used. The values are hypothetical and illustrate communication, not evidence from an actual study.
Different Predictions Can Support the Same Action
The reverse case matters too. Two models can predict different means or uncertainty distributions and still lead to the same action if their predictions remain on the same side of a threshold. That is agreement in action, not evidence that the models are interchangeable. The decision may be stable under the chosen rule while other uses of the predictions remain sensitive to model choice.
Worked Example: Different Benefits, Same Threshold Decision
Suppose a policy selects “go” when the mean improvement at 12 weeks is at least \(1.00\) point. Under the same target population and outcome definition, Model C predicts \(1.40\) points with a central 90% predictive interval of \(0.20\) to \(2.60\) points. Model D predicts \(1.90\) points with an interval of \(0.50\) to \(3.30\) points.
The difference in predicted means is \(1.90-1.40=0.50\) points. Applying the stated policy:
The models agree on the action but not on the predicted magnitude or the predictive interval. That distinction matters if the prediction is later used for planning, dose selection, or another decision with a different threshold. It would be inaccurate to report that the models “agree” without specifying that they agree on this particular threshold-based action.
The intervals here describe individual outcomes under their respective models. They overlap, but interval overlap is not the reason the actions agree; both means exceed the \(1.00\)-point criterion. The example is hypothetical.
Compare Actions When There Are Several Options
For choices among multiple doses or regimens, a single predicted outcome may not be enough. A utility function can encode how benefits and harms are valued, and the expected utility under each model can then be compared across actions. The utility definition and scale should be stated so that readers can understand what the selected action optimizes.
Worked Example: Two Models Rank Candidate Doses Differently
Consider two candidate doses, 10 mg and 20 mg, and two plausible models. Expected net benefit is summarized on a common utility-point scale, with larger values preferred. Model A estimates expected utility as \(6.2\) points for 10 mg and \(5.8\) points for 20 mg. Model B estimates \(5.9\) points for 10 mg and \(6.4\) points for 20 mg. Suppose the model-specific 90% uncertainty intervals for these expected utilities are, respectively, \(5.4\) to \(7.0\), \(4.9\) to \(6.7\), \(5.0\) to \(6.8\), and \(5.5\) to \(7.3\) points.
| Model | Expected utility, 10 mg | Expected utility, 20 mg | Selected dose |
|---|---|---|---|
| A | 6.2 points (90% interval: 5.4–7.0) | 5.8 points (90% interval: 4.9–6.7) | 10 mg |
| B | 5.9 points (90% interval: 5.0–6.8) | 6.4 points (90% interval: 5.5–7.3) | 20 mg |
Under Model A, the expected-utility difference is \(6.2-5.8=0.4\) points in favor of 10 mg, so 10 mg is selected. Under Model B, the difference is \(6.4-5.9=0.5\) points in favor of 20 mg, so 20 mg is selected. The action rankings reverse, even though the model-specific uncertainty intervals overlap.
The intervals do not by themselves determine which dose is preferred: the stated rule selects the dose with the larger expected utility under each model. Their overlap signals uncertainty in the estimates, but is not a test of equality or proof that the dose rankings are identical. A decision report should make clear that the dose choice depends on model structure under this utility rule. All numbers are hypothetical.
Diagnostics and Communication Choices
A side-by-side display is most useful when it makes the source and practical consequence of disagreement visible. Include the model label, target population, outcome and time point, candidate action, prediction summary and units, uncertainty summary, decision rule, and selected action. If the decision turns on a threshold, show the threshold alongside the predictions. For a utility comparison, report the utility scale and enough values to see why one action wins under each model.
Check the comparison for these common failure modes:
- Misaligned estimands. Predictions at different time points or for different populations cannot be read as a direct model comparison. Recompute them for a shared target or explain the mismatch.
- Mixed decision rules. Applying a mean threshold to one model and a probability threshold to another creates a procedural difference, not a fair comparison. Apply one prespecified rule consistently.
- Confusing interval overlap with action agreement. Intervals can overlap while model-specific actions differ, or fail to overlap while the chosen action is the same. Apply the actual rule to the relevant summaries.
- Hiding model-specific uncertainty. Reporting only point predictions obscures how uncertain each model's prediction is. Show suitable model-specific uncertainty summaries and identify what they describe.
- Interpreting agreement too broadly. The same action under one rule does not establish similar predictions, agreement under other rules, or equivalence of the models.
Comparing models side by side differs from choosing one model as the sole basis for a decision: the comparison preserves the consequences of each plausible specification. It also differs from combining model predictions into one summary. A combined prediction answers a different question and should not replace the model-specific results in this comparison. In particular, an action comparison should not conceal disagreement by reporting only a central prediction across models.
The method has limits. It does not establish which model is correct, and it cannot compensate for models that are implausible or poorly supported. Agreement is conditional on the selected models, scenario, summaries, and decision rule. When predictions lie near a decision boundary, small modeling differences may change the action; when all models are far from it, action agreement may be reassuring for that specific decision but does not demonstrate predictive equivalence. Model-specific uncertainty and between-model disagreement should therefore be described separately.
Check Your Understanding
Answer each question using the stated rule, then compare your reasoning with the solutions.
- A “go” decision requires a mean benefit of at least 2.0 points. Model P predicts 2.05 points and Model Q predicts 1.95 points. What do the models imply about action, and how far apart are their means?
Solution. Model P implies go because \(2.05 \geq 2.0\); Model Q implies no-go because \(1.95 < 2.0\). The means differ by \(2.05-1.95=0.10\) points, so similar means do not ensure the same action near a threshold.
- Two models predict means of 1.3 and 1.8 points. A go decision requires a mean of at least 1.0 point. Do the models agree on outcome magnitude? Do they agree on the action?
Solution. They do not agree on predicted magnitude: the means differ by \(1.8-1.3=0.5\) points. They do agree on this action, because both means are at least 1.0 point and therefore both imply go.
- For two doses, Model R predicts expected utilities of 7.1 and 6.8 points; Model S predicts 6.5 and 7.0 points. Under the rule “select the dose with the larger expected utility,” which dose does each model select?
Solution. Model R selects the first dose because \(7.1>6.8\). Model S selects the second dose because \(7.0>6.5\). The selected action depends on the model.
- Two model-specific predictive intervals overlap, but their predicted means fall on opposite sides of the stated go/no-go threshold. Is overlap enough to declare agreement in action?
Solution. No. If the rule is based on the mean, apply it to each model's mean. Opposite sides of the threshold imply different actions, regardless of whether the predictive intervals overlap.