From a Model Output to an Action
A model can produce a numerical result without specifying what a development team should do with it. For example, a model may predict a 62% probability of a target response under a candidate regimen. Whether that prediction supports advancing the regimen depends on the decision being made, the available actions, and the consequences of choosing incorrectly. A model output is evidence for a decision; it is not the decision rule itself.
To make the connection explicit, define the intended population and outcome, the decision options, and the rule that maps the output to an action. The same predicted probability might support advancement when the cost of stopping a useful regimen is high, but not when the cost of advancing an ineffective regimen is high. The cutoff is therefore part of a decision policy, not a universal property of the model.
A useful rule is operational: two people given the same relevant output can apply it and identify the same action. “Advance if the model looks promising” is not operational. “Advance if the model-predicted probability of target response by week 8 is at least 0.60” is operational, provided the population, regimen, response definition, and action are also clear.
Deriving a Probability Cutoff From Decision Consequences
A cutoff can be justified by comparing the expected consequences of the available actions. The simple derivation below assumes two possible states: the candidate has a meaningful effect or it does not. Let \(q\) be the model-predicted probability of a meaningful effect for the specified candidate and target population. Assume the model output is being interpreted as that probability, the consequences of each action in each state have been specified, and the only actions under consideration are advance and stop. These assumptions are deliberately simple; real development decisions may involve more actions and outcomes.
Suppose, for illustration, that advancing a candidate with no meaningful effect has a loss of 12 decision units, while stopping a candidate that has a meaningful effect has a loss of 8 units. Assume the other two action-state combinations have zero loss for this comparison. The units are illustrative, not money, patient outcomes, or a universal measure of value. If losses are averaged using the model-predicted probabilities, the expected loss of each action is:
The first expression uses \(1-q\), the predicted probability of no meaningful effect; the second uses \(q\), the predicted probability of a meaningful effect. Choose the action with lower expected loss. Advancing is preferred when:
Thus, under these illustrative losses, the rule is to advance when \(q>0.60\), stop when \(q<0.60\), and treat \(q=0.60\) as a tie in expected loss. A project could specify a tie-breaking action in advance. Changing the relative losses changes the cutoff: a different cutoff does not mean the model changed; it means the decision policy gives different weight to the consequences.
Worked Example: Applying a Cutoff Derived From Losses
A development team is considering whether to advance a hypothetical regimen. For its specified target population, the model predicts \(q=0.68\) for the probability of a meaningful response. Use the illustrative losses of 12 units for advancing a regimen with no meaningful effect and 8 units for stopping one with a meaningful effect. Assume the model output has the probability interpretation just defined, and that these are the losses the team intends to use for this decision.
The expected loss of advancing is:
The expected loss of stopping is:
Since \(3.84<5.44\), advancing has the lower expected loss under the stated policy. Equivalently, \(0.68>0.60\), so the derived rule selects advance. This conclusion is conditional on the hypothetical probabilities and losses; it does not establish that the regimen is effective or that the same cutoff should be used for another decision.
Different Outputs Can Support Different Decisions
A rule depends on what the model reports. A predicted mean outcome and the predicted probability of meeting a target are different summaries. They can lead to different actions because they answer different decision questions. Neither summary is automatically the right one: the output should correspond to the outcome the action policy is intended to address.
Worked Example: Mean Change Versus Target Response
Suppose a hypothetical model predicts week-8 change in a symptom score for a target population. Lower scores are better, so a negative change indicates improvement. For illustration, assume the model predicts a change of \(-4.00\) points for 38% of the population and \(-1.42\) points for the remaining 62%. Consider two separately specified advancement policies: (A) advance if mean improvement is at least 2 points, or (B) advance if at least 50% of the population is predicted to improve by 3 or more points. Assume both outputs refer to the same regimen, population, and time point.
Under policy A, first calculate the predicted mean change:
The mean predicted improvement is about 2.40 points, which exceeds the policy’s 2-point target. Policy A therefore selects advance. Under policy B, the predicted proportion improving by at least 3 points is 0.38, because only the group with a 4-point improvement meets that target. Since \(0.38<0.50\), policy B selects do not advance.
These decisions differ because the policies ask different questions. The mean includes improvements of less than 3 points, whereas policy B counts only people meeting the 3-point target. The model output alone does not decide which policy is appropriate; the intended development objective must determine that choice. The two policies should not be presented as though they were interchangeable ways of calculating the same criterion.
More than one output can also be written into a single rule. For example, a team might require both a minimum predicted response probability and a maximum predicted serious-toxicity probability. This is a conjunctive rule: both conditions must be met. Stating the combination matters. A rule requiring both conditions is not equivalent to one that accepts either condition, nor does it automatically say how to rank two candidates that both pass.
Worked Example: Combining Efficacy and Safety Outputs
A hypothetical team compares two regimens for the same population and follow-up period. It specifies in advance that a regimen qualifies for further development only if its predicted probability of target response is at least 0.60 and its predicted probability of serious toxicity is no more than 0.20. Assume the two model outputs refer to the stated response and toxicity outcomes in the same target population.
For regimen A, the model predicts a response probability of 0.64 and a serious-toxicity probability of 0.12. It passes both conditions: \(0.64\geq0.60\) and \(0.12\leq0.20\). Regimen A qualifies under this rule.
For regimen B, the predicted response probability is 0.76 and the predicted serious-toxicity probability is 0.23. It passes the response condition because \(0.76\geq0.60\), but fails the toxicity condition because \(0.23>0.20\). Regimen B does not qualify under the stated conjunctive rule, despite its higher predicted response probability.
The rule therefore identifies regimen A as qualifying and regimen B as not qualifying. It does not prove that regimen A is safer or more effective in practice, nor does it specify what to do if neither regimen passes. Those are separate policy questions that need explicit answers if they are part of the decision.
Writing and Checking the Rule
Before using a rule, make its components explicit. In particular, confirm that the output is defined on the same population, outcome, and time horizon as the decision. A response probability for one endpoint cannot silently stand in for a probability of a different endpoint. Likewise, a model output for one regimen should not be compared with a cutoff intended for another setting unless the policy justifies that use.
State the action choices in concrete terms, such as advance, stop, or select a regimen.
Specify what the model reports, including the outcome, target population, and time point relevant to the action.
Give the threshold or comparison rule, including how equality and multiple criteria are handled.
Ask what action follows on each side of the threshold and whether the rule reflects the decision’s stated priorities.
A threshold rule is attractive because it is simple to communicate and apply. An expected-loss rule can make trade-offs explicit when probabilities and consequence weights are available. A multi-criterion rule can preserve separate requirements, such as minimum efficacy and maximum toxicity, but may leave several passing options without a ranking. These approaches are alternatives in policy design, not competing model-fitting methods. A more complicated rule is not necessarily better if its inputs or consequences are unclear.
Diagnostics, Pitfalls, and Limitations
The first check is logical: can the stated output actually be compared with the stated threshold? If the rule says “advance when response probability is at least 0.60,” the output must be a probability for the specified response, not a mean score change or a probability for a different endpoint. The second check is operational: is the action defined for values below, above, and exactly at the cutoff? With several criteria, check every combination that could occur.
Also examine how close the output is to the cutoff. A value just above a threshold and one far above it receive the same action under a simple binary rule, but that does not mean the decisions are equally robust to changes in assumptions or estimates. This observation is a reason to document the rule and consider what would happen under plausible alternative outputs; it is not a reason to quietly move the cutoff after seeing the result.
- Mistake: treating a model output as an action. “The predicted response probability is 0.58, so the regimen should advance” omits the rule that makes 0.58 sufficient. Correction: state the criterion and show how the output meets or fails it.
- Mistake: treating a cutoff as universal. A 0.60 threshold is not inherently correct for every endpoint, population, or development choice. Correction: state the decision purpose and justify the policy for that purpose.
- Mistake: changing the output without changing the rule. A cutoff for a target-response probability cannot be applied directly to a predicted mean change. Correction: define the output and the matching criterion together.
- Mistake: reading a rule as proof. Meeting a decision threshold does not establish that the model is correct or that an outcome is certain. Correction: describe the result as the action selected under the specified model output and policy.
A decision rule has limitations even when it is clearly stated. Its result depends on the chosen output, thresholds, and consequences; a different policy can produce a different action from the same model result. A rule can also omit considerations that matter to the actual decision. Making these choices visible improves interpretation, but does not by itself establish that the model output is sufficiently reliable for the use.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A policy says to advance if the predicted probability of target response is at least 0.55. The model reports 0.52. What action follows, and what does that conclusion mean?
- Using losses of 10 units for advancing when there is no meaningful effect and 5 units for stopping when there is a meaningful effect, derive the cutoff for \(q\). What action is preferred at \(q=0.70\)?
- A regimen has predicted target-response probability 0.70 and predicted serious-toxicity probability 0.22. A rule requires response probability of at least 0.60 and toxicity probability no more than 0.20. Does it qualify?
- Why might a policy based on mean improvement select a different action from one based on the probability of achieving a specified improvement?
Solutions. 1. The rule selects do not advance because \(0.52<0.55\). This is the action under that stated policy; it does not prove that the regimen cannot work. 2. The expected losses are \(L(\text{advance})=10(1-q)\) and \(L(\text{stop})=5q\). Advance when \(10(1-q)<5q\), so \(10<15q\) and \(q>10/15\approx0.667\). At \(q=0.70\), advance is preferred. 3. It does not qualify: it passes the response condition but fails the toxicity condition because \(0.22>0.20\). 4. A mean uses the size of outcomes across the population, while a target-probability rule counts the fraction meeting a specified threshold. They summarize different features and can therefore lead to different actions.