Why Fit and Decision Fitness Are Different
A model may describe the data used to build it very closely and still be unreliable for its intended use. The data may not represent the population or conditions relevant to the decision, the model may predict poorly in new cases, or it may predict an outcome that does not answer the decision question. A model can also be asked to predict beyond the range supported by its data. In each case, close agreement with observed data is not enough to establish that the model is fit for purpose.
The key distinction is between fit and decision fitness. Fit describes agreement between model predictions and observed values in a specified data set. Decision fitness asks whether the model, its evidence, and its uncertainty are adequate for a particular context of use and the decision rule linked to it. The same model can be useful for describing one data set but inadequate for predicting a treatment outcome in a different population.
A fit statistic is not a universal certificate. Its interpretation depends on what data were used, what outcome was evaluated, and what errors matter for the decision. For example, an average error near zero can occur when overpredictions and underpredictions cancel. A small error in a clinically important region may matter more than a larger error far from the decision threshold. Thus, the context of use and decision rule determine what performance evidence is relevant.
What a Fit Statistic Can—and Cannot—Show
One common summary is the root mean squared error (RMSE). It is the square root of the average squared residual and is expressed in the same units as the outcome. Squaring prevents positive and negative residuals from cancelling, while giving larger errors more influence.
RMSE reports the typical scale of prediction errors in the evaluated data, with greater weight on large errors. It does not, by itself, show whether predictions are biased in a particular subgroup, whether the data represent the intended population, whether the model predicts well in new data, or whether the modeled outcome is the one needed for the decision.
Worked Example: Excellent Development-Data Fit, Larger Later Errors
In a hypothetical development data set, a model predicts an outcome measured in percentage points. The observed and predicted values are 11 and 10, 19 and 20, 31 and 30, and 39 and 40. Assume the predictions and observations refer to the same outcome and conditions, and that the four records are the data set being summarized.
The residuals are observed minus predicted: \(11-10=1\), \(19-20=-1\), \(31-30=1\), and \(39-40=-1\) percentage points. Therefore,
The model has small errors in this data set. Now consider four hypothetical later observations, with observed values 18, 26, 22, and 30, and predictions 22, 22, 25, and 27 percentage points. The residuals, again observed minus predicted, are \(18-22=-4\), \(26-22=4\), \(22-25=-3\), and \(30-27=3\) percentage points. Their mean is zero, but the RMSE is
The zero mean residual does not mean that predictions were accurate: positive and negative errors cancel in the mean, while the RMSE captures their size. The later-set RMSE is larger than the development-set RMSE. That difference is a reason to investigate performance in the relevant conditions, not by itself proof of why performance changed or proof that every future prediction will have this error.
Sampling and Prediction Must Match the Intended Use
A model learns from the data available to it. If the data contain only a narrow or selected part of the population, a good fit can be genuine for that part and uninformative about people or conditions not represented. This is a sampling problem when the observed sample does not adequately represent the target population or the features that matter to the outcome. It is not enough to report that a model fits its sample: the evidence must also support the population and conditions named in the context of use.
A useful justification for this distinction follows from the decomposition of prediction error. Suppose, for a target case with covariates \(x\), the outcome is \(Y=f(x)+\epsilon\), where \(f(x)\) is the conditional mean and the random error \(\epsilon\) has mean zero and variance \(\sigma^2(x)\). Suppose the fitted prediction \(\hat f(x)\) is random across possible development samples. Its expected squared prediction error at \(x\) can be written as
To see why, write \(Y-\hat f(x)=\epsilon+f(x)-\hat f(x)\), square, and take expectations. The cross-term involving \(\epsilon\) has expectation zero if \(\epsilon\) has conditional mean zero and is independent of the fitted prediction. For the remaining prediction term, use \(E[(\hat f-f)^2]=\operatorname{Var}(\hat f)+(E[\hat f]-f)^2\). The three terms represent squared bias, variability in the fitted prediction, and outcome variation not captured by the model. This identity is for a fixed target \(x\); performance averaged over a population also depends on which \(x\) values that population contains. A low training-set error can therefore coexist with larger target-population error if the model is biased there, uncertain there, or evaluated over a different distribution of cases.
Worked Example: A Selected Sample Hides a Relevant Difference
In a hypothetical study, the outcome is response improvement in percentage points. Let \(X\) denote drug exposure and let \(Z\) indicate a disease feature: \(Z=0\) for the lower-inflammation group and \(Z=1\) for the higher-inflammation group. For illustration, suppose the true relationship is \(Y=10+10X+20Z\), with no random error. The development sample includes only \(Z=0\), with \(X=1,2,3\). The observed outcomes are therefore 20, 30, and 40 percentage points.
A model \( \hat{Y}=10+10X \) predicts those three observations exactly: at \(X=1,2,3\), its predictions are 20, 30, and 40. Its development-data RMSE is \(0\) percentage points. Suppose, however, that the intended use includes patients with \(Z=1\). At \(X=3\), the model predicts \(10+10(3)=40\) percentage points, while the stipulated relationship gives \(10+10(3)+20(1)=60\) percentage points. The prediction is 20 percentage points too low for that target case.
The example assumes the stated relationship applies to both groups; the development data cannot establish that assumption because they contain no \(Z=1\) patients. The excellent fit is real for the sampled group, but it does not establish performance in the missing group. A model assessment should therefore check whether the development data cover the features that define the intended population and whether those features could change predictions in a decision-relevant way.
Predicting the Wrong Outcome Is Not Enough
A model can predict its chosen endpoint accurately while failing to answer the development question. The modeled outcome must connect to the decision rule. For example, a model of target engagement may be useful for describing or predicting a biomarker, but a treatment-selection decision may also depend on clinical benefit and safety. If the decision rule requires both a minimum benefit and an acceptable risk, accurate biomarker predictions alone cannot establish that both conditions are met.
Worked Example: Accurate Biomarker Prediction Does Not Resolve a Benefit–Risk Decision
Suppose, hypothetically, a model predicts a biomarker response at two regimens with errors no greater than 1 percentage point. The decision rule, however, is to select a regimen only if expected clinical benefit is at least 30 percentage points and the probability of a serious adverse event is below 10%. The model predicts biomarker responses of 80% and 82% for the two regimens.
Those predictions indicate similar biomarker responses, but neither is a prediction of clinical benefit or adverse-event risk. Even if the biomarker predictions were exact, they do not supply the two quantities required by the rule. If separate hypothetical evidence estimated serious adverse-event risks of 4% and 18%, respectively, the second regimen would fail the risk condition, but the biomarker model could not make that distinction from its output. The issue is not that the biomarker model fits poorly; it is that its output alone is not sufficient for this decision.
This conclusion assumes the stated decision rule is the one to be applied and that the model has been used only to predict the biomarker endpoint. If the biomarker had been validated as a suitable surrogate for the relevant outcomes in this context, that would be additional evidence to assess—not something established by good biomarker fit alone.
Extrapolation Adds Assumptions
A prediction is an extrapolation when it applies a model outside the range or conditions directly supported by the development data. A model may fit within its observed domain while different plausible relationships give different predictions outside it. The farther the intended use extends beyond the evidence, the more the conclusion depends on structural assumptions rather than direct information from the data.
Worked Example: Two Exact Fits, Different High-Dose Predictions
Consider hypothetical response data at doses of 20 and 30 mg: the observed responses are 50 and 60 units. Assume the intended decision at 60 mg is whether response reaches at least 80 units. Over the observed doses, both of the following models fit exactly:
where \(D\) is dose in mg and \(Y\) is response in units. At 20 mg, the predictions are \(30+20=50\) and \(100(20)/(20+20)=50\) units. At 30 mg, they are \(30+30=60\) and \(100(30)/(30+20)=60\) units. Both have zero error at these two dose levels.
At 60 mg, however, the first model predicts \(30+60=90\) units, while the second predicts \(100(60)/(60+20)=75\) units. The first prediction meets the 80-unit threshold and the second does not. The observed data cannot choose between these predictions because 60 mg lies beyond the observed dose range. This hypothetical calculation does not establish that either model is biologically correct; it shows that fit at 20 and 30 mg alone cannot resolve the decision at 60 mg. Additional evidence or a justified basis for the model form would be needed.
How to Assess Fitness for the Intended Decision
The relevant checks follow from the purpose of the model. They do not replace evaluation of fit; they put fit in context. A useful review asks whether the data, model outputs, and performance evidence align with the stated population, conditions, and decision rule.
Look for important population features, treatment conditions, and outcome ranges that are absent or sparsely represented. A large data set does not compensate for missing target groups or conditions.
Examine performance in the target population and in regions relevant to the threshold or action, not just overall agreement. Where appropriate, assess performance on data not used to fit the model.
Map each model output to the quantity required by the decision rule. If an outcome is only indirect evidence for the required quantity, state the additional assumptions and evidence needed to use it.
Identify predictions made outside observed conditions and determine whether the decision depends on assumptions that have little direct support. Treat apparent precision from a model calculation as distinct from strength of evidence.
Diagnostics should be interpreted in relation to the specific failure mode. Residual patterns across groups or dose levels can reveal systematic over- or underprediction that an overall RMSE hides. A difference between development and validation performance can signal weak transport to new data, but its cause needs investigation. A mismatch between the modeled endpoint and the decision rule is a problem of relevance even when predictive diagnostics are favorable. Extrapolation is a concern when the decision rests on predictions beyond the evidence, even if the model fits every observed point.
Common Pitfall and Practical Comparison
Wrong reasoning: “The model fits the observed data very well, so it is reliable for the decision.” This treats fit as if it established representative sampling, accurate prediction in the target setting, relevance of the output, and validity outside the observed domain. None of those follows automatically from a fit statistic.
Correction: State what data the fit describes and evaluate whether those data and predictions support the intended use. Fit summaries are useful for describing in-sample agreement. Predictive checks and validation address how predictions compare with relevant observations, including data not used for fitting when available. Decision-focused evaluation asks whether errors, uncertainty, and output relevance could change the decision. These approaches answer related but different questions; no single metric substitutes for all of them.
Decision fitness is also not a claim that a model is universally good or bad. It is specific to a context of use. A model may be adequate for describing a biomarker within a studied dose range, yet inadequate for selecting a regimen in an unstudied population. Limits should be stated plainly: which population and conditions are supported, what outcomes are predicted, and where further evidence is needed. The decision consequence matters as well: a prediction just on one side of a threshold may require more scrutiny than a similar-sized error far from it.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- A model has residuals of \(2,-2,2,-2\) units in its development data. Explain why its mean residual is not enough to describe prediction error, and calculate its RMSE.
- A model fits a sampled population that excludes patients with a feature expected to modify treatment response. What can the close fit establish, and what can it not establish?
- A decision requires both a minimum efficacy and a maximum safety risk. Why might an accurate model of a biomarker still be insufficient?
- Two models fit the observed dose range equally well but predict opposite sides of a decision threshold at a higher dose. What is the central evidence limitation?
Solutions. 1. The positive and negative residuals cancel, giving mean residual \(0\), but their magnitude remains. The RMSE is \(\sqrt{(2^2+(-2)^2+2^2+(-2)^2)/4}=\sqrt{4}=2\) units. 2. The fit describes agreement in the sampled population; it does not establish performance in the excluded group, especially if the feature changes response. 3. A biomarker prediction does not provide the efficacy and safety quantities required by the rule unless additional evidence supports that link. 4. The data do not distinguish the models’ predictions outside the observed range; the decision depends on extrapolation and assumptions not resolved by in-range fit.