Why a Model Comparison Can Be Misleading
A comparison table can make two candidate models look directly comparable even when their reported results come from different evidence. One model may have been tested on an easier sample, evaluated against a different target, or assessed using a different error metric. In those circumstances, a higher score does not by itself show that the model is better.
The central question is not merely whether the numbers are calculated correctly. It is whether they answer the same question using comparable data and procedures. This is particularly important when results are summarized for a decision-maker: an apparently small difference can be an artifact of the evaluation design rather than a real difference in predictive performance.
A fair comparison does not require that candidate models use identical algorithms. The models may differ in their predictors, structure, or fitting method. The requirement is that the evidence used to judge them is aligned. A comparison may be useful even when exact alignment is not possible, but the mismatch must be disclosed and the results should not be presented as a direct ranking.
What Must Match
Start by checking four sources of comparability: the evaluation sample, the target, the preprocessing, and the metric. For classification, also check the classification threshold when reporting measures based on predicted classes.
- Evaluation sample: Were the same records, time period, and sampling rules used? Different samples can have different levels of difficulty or different mixes of cases.
- Target definition: Do both models predict the same event or quantity over the same time horizon? “Claim paid within 90 days” and “ultimate claim cost” are different targets, even if both are related to claims.
- Preprocessing: Were missing-value treatment, transformations, category handling, and other data preparation steps defined consistently? Any step estimated from data should be learned from training data, not from held-out evaluation outcomes.
- Metric: Do the reported measures assess the same performance objective and use the same units? Accuracy, AUC, MAE, and RMSE answer different questions.
- Decision rule: For confusion-matrix measures, were the same threshold and class-label conventions used? A change in threshold can change sensitivity, specificity, and accuracy without changing the underlying predicted probabilities.
For numerical predictions \(\hat{y}_i\) and observed values \(y_i\), a useful error summary is root mean squared error. For \(n\) evaluation cases with the same target definition and units, it is
RMSE is expressed in the units of the target. It gives greater influence to large errors because the errors are squared. If one model’s RMSE is calculated in dollars and another’s in thousands of dollars, the figures cannot be compared without putting them on a common scale. Even after unit conversion, the target and evaluation cases must still match.
A Short Justification for Using the Same Cases
Using the same cases helps separate differences in model predictions from differences in case difficulty. Suppose models A and B predict the same \(n\) observed outcomes \(y_i\). Define their squared errors for case \(i\) as \((\hat{y}_{Ai}-y_i)^2\) and \((\hat{y}_{Bi}-y_i)^2\). The difference in their average squared errors is
Each bracket compares the two models on the same observed outcome. A difficult case that produces a large error for both models is therefore not mistaken for a model-specific disadvantage simply because it appears in only one model’s evaluation sample. Since the square-root function is increasing for nonnegative values, the model with the lower average squared error also has the lower RMSE when both are evaluated on these same cases.
This argument does not establish that the observed difference will persist in future data. It explains why common evaluation cases make the comparison more interpretable. The size and uncertainty of a measured difference remain relevant, especially when the evaluation sample is small.
Worked Examples
Worked Example: Classification Results From Different Test Samples
A hypothetical insurer reports that Model A has AUC \(0.82\) on a test sample of 400 claims, while Model B has AUC \(0.85\) on a different test sample of 120 claims. The positive-event prevalence is \(10\%\) in A’s sample and \(30\%\) in B’s. A manager asks whether Model B is therefore the better candidate. Explain what can and cannot be concluded, then compare the models on a shared toy benchmark using a common threshold of \(0.50\).
Method. First assess whether the reported AUC figures represent the same evaluation evidence. Then, for the supplied shared benchmark, calculate the confusion-matrix counts and accuracy for each model.
The reported AUC values were calculated on different samples. AUC measures ranking performance, but different samples can contain different case mixes and different levels of difficulty. The prevalence difference alone does not prove that the AUCs are incomparable, nor does it make AUC an accuracy measure. However, without evaluation on a common or otherwise demonstrably comparable sample, the difference \(0.85-0.82=0.03\) is not sufficient evidence to rank the models.
For a transparent check, use the following hypothetical common benchmark. The observed outcomes, in order, are \(1,1,1,0,0,0,1,0\). At the stated threshold, Model A predicts \(1,1,0,0,1,0,1,0\), and Model B predicts \(1,0,1,0,0,0,1,1\).
For Model A, the first, second, and seventh cases are true positives, so \(TP=3\). The third positive case is missed, so \(FN=1\). Among the four negative cases, the fifth is predicted positive, giving \(FP=1\), and the fourth, sixth, and eighth are true negatives, giving \(TN=3\). Thus, accuracy is
For Model B, the first, third, and seventh cases are true positives, so \(TP=3\); the second positive case is missed, so \(FN=1\). Of the negative cases, the eighth is a false positive, so \(FP=1\), and the fourth, fifth, and sixth are true negatives, so \(TN=3\). Its accuracy is
Check. For each model, \(TP+FN=4\) observed positives and \(FP+TN=4\) observed negatives, accounting for all eight cases. On this toy benchmark and at this threshold, both models have accuracy \(0.7500\). This does not establish equal AUC or equal performance on a larger population; it demonstrates why the reported AUC difference on separate samples does not resolve their comparison.
Common mistake avoided: Declaring Model B superior simply because \(0.85\) is greater than \(0.82\). The correction is to obtain comparable evidence, such as predictions from both models on a shared evaluation set, and then calculate the relevant metrics under stated conditions.
Worked Example: Different Target Definitions and Units
A hypothetical reserving team reports Model A with RMSE \(1.4\) thousand dollars for claim payments made within 90 days, and Model B with RMSE \(1.6\) thousand dollars for ultimate claim cost. The team then supplies a common benchmark of four claims, with ultimate costs and both models’ predictions all stated in thousands of dollars: observed costs are \(10,20,30,40\); Model A predicts \(12,18,33,37\); Model B predicts \(11,22,28,41\). Determine what the original figures support and calculate both models’ RMSEs on the common benchmark.
Method. Check the target before comparing reported metrics. Then calculate each model’s errors against the common ultimate-cost observations, square the errors, average them over the four cases, and take the square root.
The original figures use different targets: payment within 90 days is not the same quantity as ultimate cost. Therefore, the reported RMSE values \(1.4\) and \(1.6\) do not support a direct ranking. On the common benchmark, both prediction vectors and observed values refer to ultimate cost, in thousands of dollars.
For Model A, the prediction errors \(\hat{y}_i-y_i\) are \(12-10=2\), \(18-20=-2\), \(33-30=3\), and \(37-40=-3\). The squared errors are \(4,4,9,9\), with sum \(26\). The mean squared error is \(26/4=6.5\), so
For Model B, the errors are \(11-10=1\), \(22-20=2\), \(28-30=-2\), and \(41-40=1\). The squared errors are \(1,4,4,1\), with sum \(10\). The mean squared error is \(10/4=2.5\), so
Check. Both calculations use four observations and the same target and units. On this common benchmark, Model B has the lower RMSE. The square roots are rounded to four decimal places; in dollars, these RMSEs are approximately \(\$2{,}549.51\) and \(\$1{,}581.14\), respectively.
Common mistake avoided: Comparing \(1.4\) directly with \(1.6\) and selecting Model A. Those figures concern different targets. The common-benchmark calculation supports a comparison for ultimate cost on these four cases, not a conclusion about all future claims.
Worked Example: Metric Choice Can Reverse the Ranking
A hypothetical equipment-risk team compares two models on the same five evaluation cases. The observed repair costs, in hundreds of dollars, are \(10,10,10,10,10\). Model A predicts \(10,10,10,10,6\), while Model B predicts \(9,9,9,9,9\). Calculate MAE and RMSE for both models, then explain why a report must identify the metric used to choose between them.
Method. For MAE, average the absolute errors. For RMSE, square the errors, average them, and take the square root. Both metrics use the same five observations and the same target units.
Model A’s errors are \(0,0,0,0,-4\). The absolute errors sum to \(4\), so its MAE is \(4/5=0.8\) hundred dollars. The squared errors sum to \(16\), so its RMSE is
Model B’s errors are \(-1,-1,-1,-1,-1\). Its absolute errors sum to \(5\), giving MAE \(5/5=1\) hundred dollars. Its squared errors also sum to \(5\), giving
Check. Model A has lower MAE, \(0.8<1\), while Model B has lower RMSE, \(1<1.7889\). The ranking reverses because RMSE gives a larger penalty to Model A’s one error of \(4\) than MAE does. A decision-maker should select the metric that matches the intended error objective, rather than treating one model as unconditionally better.
Common mistake avoided: Reporting a model as “more accurate” without naming the metric. State the metric and its meaning; when the choice affects the decision, show more than one relevant measure or explain why one is the primary measure.
A Practical Audit Before Reporting a Ranking
A concise audit helps turn a set of model scores into a defensible comparison. It also makes clear when the available evidence is insufficient for a direct ranking.
Specify the prediction target, time horizon, intended use, and whether the decision concerns ranking, numerical error, or a threshold-based classification outcome.
Check that each candidate is evaluated on the same records or on samples whose differences are understood and adequately addressed. Record any differences in time period, sampling, or case mix.
Confirm identical outcome definitions and units. Confirm that imputation, scaling, feature selection, and other data-dependent preprocessing were learned from training data and applied consistently to evaluation data.
Use the same metric definitions and, for threshold-based results, the same stated threshold-selection procedure. Explain what the selected metric emphasizes.
Describe what the common evidence supports and avoid extending a result from a particular sample, target, or metric to a broader claim than it warrants.
Preprocessing deserves particular attention. If a transformation or imputation rule is estimated using the full data set before the evaluation split, information from evaluation records can influence the model-development process. The reported evaluation may then no longer represent a clean assessment of how the modeling procedure would perform on data not used in development. When comparing candidates, preprocessing should be part of each candidate’s training procedure and should be applied using information available at the appropriate stage.
A useful report can still show results that are not directly comparable, provided it labels the differences clearly. For example, separate results may describe how each model performed in its own setting. The report should then say that the evidence does not establish which candidate performs better under a common evaluation, rather than presenting the largest score as a winner.
Check Your Understanding
For each problem, decide what the evidence supports and complete the requested calculation or audit.
- A hypothetical fraud screen reports 94% accuracy for Model A on one sample and 91% for Model B on another sample. The samples cover different months and have different event prevalence. Can the figures establish that A is better? Name two pieces of evidence needed for a fair comparison. (Answer: No; compare both models on the same or demonstrably comparable evaluation cases, with the same target and metric, and disclose the threshold if using accuracy.)
- In a hypothetical portfolio of 1,000 accounts, 50 have the event. A classifier predicts “no event” for every account. Calculate its accuracy and sensitivity, and explain why accuracy alone is insufficient here. (Answer: Accuracy \(=950/1000=0.9500\); sensitivity \(=0/50=0.0000\); accuracy alone hides that the model detects none of the events.)
- On four common evaluation cases, Model A has errors \(0,0,0,5\), and Model B has errors \(2,2,2,2\), in the same target units. Calculate each MAE and RMSE and state which model is preferred by each metric. (Answer: A: MAE \(=5/4=1.25\), RMSE \(=\sqrt{25/4}=2.5\). B: MAE \(=8/4=2\), RMSE \(=\sqrt{16/4}=2\). MAE favors A; RMSE favors B.)
- A candidate model’s imputation values and scaling parameters were calculated using all records before a train/test split. What is the concern, and what should the comparison procedure do instead? (Answer: Evaluation data influenced data-dependent preprocessing, so the evaluation may be contaminated; estimate preprocessing using training data within the modeling procedure, then apply it to held-out data.)
- Two models are evaluated on the same records, but one predicts paid amount within 30 days and the other predicts total cost over the full life of a claim. What must be resolved before their RMSE values can support a direct ranking? (Answer: They must be evaluated against a common target definition and time horizon, with consistent units and the same cases; otherwise their RMSE values do not measure error on the same outcome.)