Set the Decision Before Comparing Models
A candidate model is not “best” in the abstract. For a development decision, its usefulness depends on the question it must answer, the data supporting that answer, and the consequences if its recommendation is wrong. This workshop uses a fictional case in which a team must decide whether a proposed regimen is expected to achieve a response target. The numbers are hypothetical and are for learning only.
Suppose the context of use is to predict the mean response at a specified exposure for a defined patient population, then recommend whether the regimen meets a target. Response is measured in points on a prespecified scale. The decision threshold is 30 points: the regimen qualifies if the predicted mean response is at least 30 points. In an actual development setting, the population, endpoint, exposure measure, prediction target, and decision rule would all need to be stated precisely.
The rule is simple; supporting it is not. A model may predict well on average but give an unreliable answer at the intended exposure. Conversely, a model with slightly larger overall prediction errors might be more credible around the threshold if its data better represent the intended population and use. The earlier material on decision rules, validation, applicability domains, and decision-sensitive uncertainty provides the vocabulary for examining those issues.
Worked Comparison: Predictions at the Decision-Relevant Exposures
Three candidate models have been developed to predict response from exposure. The team has held out observations from the same target population and measured response at exposures close to the proposed regimen. The table gives the observed mean response at each exposure and each model’s prediction. For this exercise, treat those held-out means as the validation reference, while recognizing that observed means are not error-free truths.
| Exposure | Observed mean response | Model A prediction | Model B prediction | Model C prediction |
|---|---|---|---|---|
| 8 exposure units | 26 points | 25 points | 27 points | 24 points |
| 10 exposure units | 30 points | 32 points | 29 points | 31 points |
| 12 exposure units | 32 points | 39 points | 30 points | 35 points |
For the proposed regimen, \(x^*=10\) exposure units and \(T=30\) points. Thus Model A recommends qualifying the regimen because 32 is at least 30; Model B recommends not qualifying it because 29 is below 30; Model C recommends qualifying it because 31 is at least 30. The models disagree precisely where the decision matters, so a comparison of their errors and the relevance of their evidence is needed.
Worked Example: Calculate and Interpret Validation Errors
Calculate the mean error and root mean squared error (RMSE) for each candidate. Define the error at validation point \(i\) as prediction minus observed response. A positive mean error indicates overprediction on average for these validation points; RMSE summarizes the typical magnitude of error, giving larger errors more influence because they are squared.
For Model A, the three errors are \(25-26=-1\), \(32-30=2\), and \(39-32=7\) points. Its mean error is \((-1+2+7)/3=2.667\) points. Its RMSE is:
For Model B, the errors are \(1,-1,-2\) points. Its mean error is \((1-1-2)/3=-0.667\) points, and its RMSE is:
For Model C, the errors are \(-2,1,3\) points. Its mean error is \((-2+1+3)/3=0.667\) points, and its RMSE is:
On these three validation points, Model B has the smallest RMSE and the smallest absolute mean error. That supports better predictive performance in this particular validation set. It does not by itself establish that Model B is the right model for the decision: it predicts 29 points at the proposed regimen, one point below the target, and the data and uncertainty near that target still need scrutiny.
Why the Error Summary Is Not the Decision
RMSE is a useful summary, but it treats errors as distances from observed values, not as decision consequences. A derivation makes one limitation clear. For \(m\) validation points, let \(e_i\) be prediction minus observation, and let \(\bar e\) be their mean. The average squared error can be written as:
To verify the identity, substitute \(e_i=(e_i-\bar e)+\bar e\), expand the square, and sum. The cross-product is zero because \(\sum_{i=1}^{m}(e_i-\bar e)=0\). Thus squared error combines squared average bias with the average squared variation of errors around that bias. RMSE compresses both features into one number; it does not show where errors occur, whether a model is biased near the threshold, or whether a particular error changes the action.
In the worked comparison, Model A’s largest error is at 12 exposure units, while its prediction at the decision exposure is above the target. Model B has the best overall RMSE but recommends against qualification. A decision-focused review therefore asks which predictions matter most for the stated use, not just which model has the smallest aggregate error.
Check Relevance Before Rewarding Performance
Validation performance is informative only to the extent that the validation data represent the intended prediction task. Relevant checks include whether the population, endpoint definition, measurement process, exposure range, and clinical setting match the proposed use. Also check whether the validation observations were kept separate from the data used to develop or tune the candidate model. Reusing development data as if they were independent validation can make performance look better than it is on new data.
The range matters especially for a threshold decision. In the first example, the proposed exposure of 10 lies among validation exposures of 8, 10, and 12 units. That is encouraging for local relevance, but three points alone provide limited evidence about performance. If a model’s predictions are sensitive to a small change in exposure, the sparse points may not resolve that behavior. Nor does validation in the target population automatically support use in a different population.
Worked Example: Do Not Extrapolate a Validation Score
Suppose a second proposed regimen is expected to produce an exposure of 14 units, but the validation observations still cover only 8, 10, and 12 units. Assume a new candidate, Model D, has an RMSE of 1.2 points over those three validation exposures and predicts 34 points at 14 units. The threshold remains 30 points.
The arithmetic rule gives \(34 \geq 30\), so Model D recommends qualifying the regimen. But 14 exposure units lies outside the validation range, and the low RMSE describes performance only over the tested range. It does not establish that the prediction at 14 is accurate. Model D’s recommendation therefore relies on extrapolation beyond the evidence described; the validation score alone cannot justify treating 34 points as a reliable threshold comparison.
The appropriate conclusion is not that Model D must be wrong, nor that the regimen must fail. It is that this evidence does not adequately establish the prediction for the stated use. The team would need relevant evidence supporting the model’s behavior at higher exposures, or would need to narrow the intended use to a range with adequate support. A favorable fit within the data range cannot fill that evidence gap.
Bring Consequences and Uncertainty Into the Comparison
The consequences of crossing the threshold incorrectly may be asymmetric. A false positive occurs when a model recommends qualifying a regimen that does not in fact meet the target; a false negative occurs when the model recommends against a regimen that does meet it. The relative importance depends on the decision and its context. A model comparison should state these consequences rather than silently treating every prediction error as equally costly.
A point prediction just above or below the threshold deserves particular care. If plausible changes in assumptions, data, or model structure move the prediction across the threshold, then the recommendation is fragile. A prediction well separated from the threshold may be less sensitive to the same degree of uncertainty. The question is not merely whether uncertainty exists, but whether it could change the decision and what that change would mean.
Worked Example: Compare Recommendations Under Unequal Consequences
Consider a separate fictional review with two models for the same threshold decision. Model P predicts 31 points and estimates a 0.20 probability that the regimen’s true mean response is below 30. Model Q predicts 29 points and estimates a 0.10 probability that the true mean response is at least 30. These probabilities are hypothetical, assumed to be calibrated for this example, and are not probabilities that either model is correct.
Suppose the decision team assigns a loss of 10 units to a false positive and 2 units to a false negative. If Model P’s recommendation is to qualify, its expected loss from a false positive is \(0.20\times10=2.0\) loss units. If Model Q’s recommendation is not to qualify, its expected loss from a false negative is \(0.10\times2=0.20\) loss units. Under these stated probabilities and loss values, the no-qualification recommendation has the lower expected loss.
This calculation does not make Model Q universally preferable. It shows that a decision can rationally weigh the probability and consequence of each type of error, rather than choosing solely by point prediction or global RMSE. If the loss values or calibrated probabilities changed, the comparison could change. The team should also examine how robust the recommendation is to uncertainty in those inputs.
These examples use different hypothetical candidate sets to isolate different issues: validation error, support at the intended exposure, and unequal consequences. They are not a single dataset from which to rank all models. In a real comparison, the evidence should be organized around one clearly stated use and decision, with consistent validation data where feasible.
A Practical Selection Sequence
A disciplined comparison keeps the intended decision visible at every step. It does not replace scientific judgment with a score; it makes the basis for judgment explicit.
Name the population, predicted quantity, intended exposure, threshold, and action attached to each side of the threshold.
Ask whether development and validation data reflect the population, measurements, and prediction range required for this use.
Summarize prediction errors, inspect their direction and location, and pay particular attention to errors that could change the threshold decision.
Determine whether plausible uncertainty or changes in assumptions move predictions across the threshold, and describe the consequences of each possible error.
State which model is better supported for the specified use, what evidence supports that judgment, and what use or conclusion remains unsupported.
The main alternatives are not simply “choose the model with the lowest RMSE” versus “choose the most complex model.” A single overall score offers a compact performance summary but can obscure threshold-relevant failures. A decision-focused comparison gives greater attention to predictions and consequences near the action boundary, but depends on a clear and justified decision rule. A model-class argument based on scientific knowledge can explain plausible behavior, but cannot substitute for relevant predictive evidence. These approaches can complement one another; none removes the need to check whether the evidence supports the intended use.
There are important limitations. Small validation sets make performance summaries unstable; observed responses can be noisy; and a threshold rule may simplify a decision that actually involves several outcomes or constraints. A model comparison is also conditional on the candidate models, data, assumptions, and intended use considered. It does not prove that the selected model is true, and a favorable comparison does not automatically justify extrapolation to a new population, endpoint, or exposure range.
Common Pitfall
Wrong reasoning: “Model B has the lowest RMSE, so it is the right choice and its recommendation is reliable.” The correction is that RMSE describes average error on specified validation data; it does not establish data relevance, accuracy at the decision threshold, or acceptable consequences of the recommendation. In the worked comparison, Model B’s lowest RMSE does not remove the need to assess why it predicts below the target, how uncertain that prediction is, and whether its validation evidence adequately supports the use.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A model predicts 28 points when the threshold is 30. Under the rule in this workshop, what recommendation does it make?
- For errors of \(-2\), \(0\), and \(4\) points, calculate the mean error and RMSE. What does the sign of the mean error indicate?
- A model has low RMSE on validation data from exposures of 5 to 9 units and predicts at 12 units. What can and cannot be concluded from that RMSE?
- Why might a model with slightly higher RMSE be preferable for a threshold decision?
Solutions. (1) It recommends not qualifying the regimen because \(28<30\). (2) The mean error is \((-2+0+4)/3=0.667\) points. The RMSE is \(\sqrt{((-2)^2+0^2+4^2)/3}=\sqrt{20/3}=2.582\) points. The positive mean error indicates average overprediction on these three points. (3) The score summarizes performance on the tested range; it does not establish accuracy at 12 units, which is outside that range. Further relevant evidence or a narrower intended use is needed. (4) Its evidence might be more relevant to the target use, its predictions near the threshold might be more reliable, or its recommendation might better account for the consequences of errors. The answer depends on the stated use and supporting evidence, not RMSE alone.