Adjudicating Conflicting Models Before a Development Decision

Why Conflicting Models Matter

A development decision can be sensitive to how the exposure–response relationship is represented. Two models may both describe the available data reasonably well yet imply different doses for reaching a target response. The disagreement is important, but it does not by itself establish that one model is wrong or that the dose must be chosen by averaging the predictions. The decision depends on the purpose of the analysis, the assumptions behind each model, and the consequences of being wrong.

This case uses hypothetical data from a Phase II trial. The aim is to choose among 100, 125, and 150 mg once daily, using two fitted models for the probability of a clinical response at week 12. For illustration, the development team has specified a target response probability of at least 0.65 and a maximum acceptable probability of a dose-limiting toxicity of 0.08. These thresholds are decision criteria for this example, not universal standards.

The Hypothetical Evidence and Competing Models

The trial randomized participants to dose groups. The observed response proportions below are simple descriptive summaries; the models also use participant-level information. Both models are assumed to have been judged plausible for this case. Their differences concern how the response curve is represented and which information is used to translate dose into response.

Model 1 is a saturating dose-response curve fitted to randomized dose and response data. Let \(D\) be dose in mg. Its predicted response probability is

Model 2 uses measured steady-state exposure, adjusted for baseline severity and then standardized to the trial population. For this example, exposure is assumed proportional to dose, so define the normalized exposure \(x=D/10\). The model uses a steeper, sigmoidal response shape:

These are illustrative fitted curves, not claims that dose and exposure are interchangeable in general. Model 1 relies on randomization to support comparisons by assigned dose. Model 2 additionally relies on the exposure measurement and dose-to-exposure mapping, and on assumptions that support interpreting its adjusted, standardized exposure-response prediction for a dose decision. In particular, exposure-response associations can be affected by factors related to both exposure and outcome. The mapping and adjustment assumptions need to be credible before Model 2 is used to recommend a dose.

Finding the Dose Each Model Implies

For a continuous dose, the target dose is the smallest dose at which the model’s predicted response probability is at least 0.65. Solving the equations makes clear that the recommendation difference arises from the curve shapes, rather than from using different target response probabilities.

A short justification helps explain why the models can separate despite fitting the same general trend. Model 1 increases smoothly toward its upper limit, with an approximately linear increase at low dose. Model 2 has a squared exposure term: at low exposure, the numerator grows as \(x^2\), so the curve rises more slowly at first and then more sharply through its middle range. The models can therefore predict similarly around 100 mg but diverge at higher doses. Sparse or noisy information about the shape between dose groups can leave both curve forms plausible.

Adjudicating the Difference

The first task is to determine what is actually being compared. Both models here predict a week-12 response probability, but Model 1 is indexed by randomized assigned dose, whereas Model 2 predicts after adjustment and standardization using measured exposure. Before treating their outputs as alternatives for the same decision, the team must check that the response definition, analysis population, time point, and population to which predictions apply are aligned. A difference in estimand or population can look like a difference in model shape.

Next, examine how each model obtains information and what must be true for its prediction to apply. Randomized assignment supports estimating dose-group effects, subject to trial conduct and outcome ascertainment. Exposure is measured after treatment starts; an exposure-response association is not automatically a causal effect of changing dose. Model 2’s relevance to a dose recommendation depends on a sufficiently credible dose-to-exposure relationship and on adjustment that addresses important differences between people with different exposures. If those conditions are weak, its apparent precision or steep curve should not settle the choice.

The observed proportions provide context but do not select the model mechanically. For example, at 150 mg the observed rate is \(35/50=0.700\). Model 1 predicts 0.6571, while Model 2 predicts 0.7228. The discrepancy is a reason to understand the data and assumptions, not a stand-alone test that proves either model invalid. With small groups, observed proportions vary; predictions also have parameter and structural uncertainty.

Making the Decision Robust Without Hiding Uncertainty

In this case, selecting 150 mg is robust to the choice between the two specified efficacy models under the stated central-prediction rule. It is not automatically a final development recommendation. The rule has not accounted explicitly for uncertainty around the model predictions, variation in individual responses, uncertainty in the toxicity curve, or the possibility that neither model captures the true relationship. A decision-maker should examine whether the conclusion remains acceptable when these uncertainties and reasonable variations in the assumptions are considered.

There are several legitimate approaches when models conflict. A team may select one model if there is a compelling scientific and evidential reason to regard it as more suitable for the context of use. It may average predictions across models when the weights are defensible and the resulting average answers the decision question. Or it may use a robust rule, such as requiring the criteria to hold under every model retained as credible. These approaches answer different questions: choosing one model privileges it, averaging represents a weighted combination, and a robust rule protects against disagreement within a defined model set. None removes the need to justify that set or the decision thresholds.

If 150 mg’s added exposure and toxicity burden is consequential, the team could also consider whether a targeted study or additional analysis would resolve the decision-relevant uncertainty. The useful information would be information likely to change the dose choice, such as data that distinguish the curve shapes in the dose range where their predictions differ. Collecting more data at doses where both models make nearly identical predictions might add little to adjudication.

Diagnostics, Failure Modes, and Limitations

Model disagreement should prompt a focused review rather than an automatic preference for the more complex model or the model that gives the desired dose. Check that the models use compatible endpoint definitions and populations; that the exposure scale and units are correct; and that predictions at observed dose levels are not being confused with extrapolation beyond them. Examine whether the main conclusion depends on an assumption such as proportional dose-to-exposure mapping or on a particular response-curve shape. Large systematic differences between predictions and observed group outcomes would also weaken confidence, although no single observed proportion is a definitive diagnostic.

A common mistake is to say, “Model 2 reaches the efficacy target at 125 mg, so 125 mg is the recommended dose.” That reasoning ignores the credible Model 1 prediction, which is below target at 125 mg, and overlooks the fact that dose selection uses both efficacy and safety criteria. The correction is to state the decision rule first, evaluate every retained model against it, and report the conditions under which the selected dose passes.

The approach has limits. A robust decision depends on which models are considered credible; omitting a plausible model can make apparent robustness misleading. Conversely, requiring a decision to satisfy every conceivable model can be so conservative that it is not useful. The model set, target population, thresholds, and treatment of uncertainty should therefore be made explicit. In this example, 150 mg passes across the two specified models using point predictions, but a decision based on uncertainty distributions could differ.