Why Model Comparisons Need a Common Basis
A comparison can sound decisive while answering the wrong question. One candidate may have a lower error on one test sample, while another is evaluated on a different sample or at a different decision threshold. Even when the calculations are correct, those results do not support a fair direct comparison. A useful report first makes clear that the candidates were assessed under comparable conditions, then explains what the differences mean for the intended decision.
The relevant comparison depends on the predictive question and intended action. For numeric predictions, a reduction in error may matter, but its size and units matter too. For classification, accuracy alone may hide the consequences of false positives and false negatives. A model can therefore be better on one metric and worse on another. The aim is not to crown a model in the abstract; it is to describe evidence about the candidates against decision-relevant criteria.
“Same basis” does not mean that every comparison must use one metric or one threshold. It means that each reported difference has a clear, consistent interpretation. If models have different operational requirements, those differences should be reported as trade-offs rather than concealed or folded into an arbitrary overall score.
Specify the Comparison Before Interpreting It
A report should identify what was held constant and what may differ. At minimum, clarify the population and outcome being evaluated, the evaluation data and time period, the metrics, and whether each metric is calculated on held-out observations. If model settings or thresholds were chosen after examining those outcomes, the reported performance may be optimistic; selection and evaluation should be separated where possible.
For example, if the purpose is to predict claim cost, a useful error measure might be mean absolute error or root mean squared error. These use different definitions and emphasize different aspects of prediction error. Root mean squared error gives greater influence to large errors because errors are squared before averaging. Whichever measure is selected, the report should name it, state its units, and keep those choices consistent across candidates.
For classification, a threshold converts a predicted probability into a class or action. If thresholds are tuned, describe the selection procedure and ensure they are selected using data separate from the final evaluation data. It can be appropriate to compare each model at a separately chosen threshold when that reflects the planned use, but then the comparison concerns the resulting model-and-threshold procedures, not just the prediction algorithms. A shared threshold answers a different question. State which question is being answered.
Worked Example: Reporting a Reduction in Numeric Prediction Error
A hypothetical insurer compares two claim-cost prediction models on the same 100 held-out claims. The observed cost and both predictions are recorded in dollars. The sum of squared prediction errors is $1,600 for Model A and $1,444 for Model B. Calculate and interpret the root mean squared error (RMSE) for each model.
Method. For \(n\) evaluated observations with prediction errors \(e_1,\ldots,e_n\), RMSE is \(\sqrt{\sum_{i=1}^n e_i^2/n}\). A smaller RMSE indicates smaller average squared error on this evaluation set, with the result expressed in the outcome’s units.
For Model A, the mean squared error is \(1{,}600/100=16\) dollars squared, so its RMSE is \(\sqrt{16}=\$4.00\). For Model B, the mean squared error is \(1{,}444/100=14.44\) dollars squared, so its RMSE is \(\sqrt{14.44}=\$3.80\).
The absolute RMSE difference is \(\$4.00-\$3.80=\$0.20\). Relative to Model A, the reduction is \(0.20/4.00=0.05\), or 5%. Thus Model B had 5% lower RMSE on this held-out sample. This is a sample-based comparison, not proof that Model B will have lower error in every future population. The method avoids the common mistake of comparing the raw sums of squared errors without accounting for the number of evaluated claims.
Check. Squaring the reported RMSE values gives \(4.00^2=16\) and \(3.80^2=14.44\), matching the two mean squared errors.
Translate Metric Differences Into Decision Consequences
A numerical gain matters only insofar as it supports the intended use. A report should connect a metric to the problem it measures, then state the practical scale of the difference. “RMSE fell by 5%” is more informative than “Model B performed better,” but the decision-maker may also need to know whether a 20-cent reduction in error is material, what costs implementation would add, and whether large errors remain a concern.
For a decision involving classification, explicitly consider the consequences of each type of error. A false negative may mean a missed opportunity to intervene; a false positive may trigger an unnecessary review. Their consequences need not be equal. The following example makes the cost assumptions explicit so the result is about a stated hypothetical decision, not a universal ranking.
Worked Example: Comparing Classification Decisions by Cost
A hypothetical service team uses a risk flag to offer an early review. On the same 200 held-out cases, 40 cases ultimately require review and 160 do not. At separately selected thresholds chosen using training data, Model A flags 28 of the 40 review cases and 20 of the 160 non-review cases. Model B flags 32 review cases and 32 non-review cases. For this illustration, assume each missed review case has a cost of $100 and each unnecessary review has a cost of $20. Ignore all other costs. Which model has the lower total error cost on this test set?
Method. Count false negatives among the 40 cases requiring review and false positives among the 160 not requiring review. Apply the same stated cost per error type to both models.
For Model A, false negatives are \(40-28=12\); false positives are 20. Its missed-review cost is \(12\times\$100=\$1{,}200\), and its unnecessary-review cost is \(20\times\$20=\$400\). Total cost is \(\$1{,}200+\$400=\$1{,}600\).
For Model B, false negatives are \(40-32=8\); false positives are 32. Its missed-review cost is \(8\times\$100=\$800\), and its unnecessary-review cost is \(32\times\$20=\$640\). Total cost is \(\$800+\$640=\$1{,}440\).
Model B’s cost is \(\$1{,}600-\$1{,}440=\$160\) lower on this sample, a reduction of \(\$160/\$1{,}600=10\%\) relative to Model A. This does not establish that Model B is universally preferable: the conclusion depends on the stated costs and this evaluation sample. It also omits possible implementation costs. The method avoids treating a larger number of flags, or a higher detection rate alone, as sufficient evidence of a better decision rule.
Check. The counts reconcile: Model A has 28 detected and 12 missed review cases; Model B has 32 detected and 8 missed. Each model’s non-review cases similarly divide into flags and non-flags: \(20+140=160\) for A and \(32+128=160\) for B.
Quantify Uncertainty in the Difference
A performance difference measured on finite data is uncertain. Reporting only two point estimates can make a small, unstable difference sound conclusive. Where feasible, describe uncertainty in the difference, not just uncertainty around each model’s separate score. When models are evaluated on the same observations, their errors are paired: the same case can be easy for both, difficult for both, or difficult for one and easy for the other.
For a simple paired analysis of a numeric error measure, define \(d_i\) as the error for Model A on observation \(i\) minus the error for Model B on that same observation. A positive mean difference favors B for a measure where smaller error is better. The standard error of the sample mean difference is \(s_d/\sqrt{n}\), where \(s_d\) is the sample standard deviation of the paired differences. This formula describes sampling uncertainty under an independent, representative sample of paired observations; it does not correct for a biased evaluation sample or data leakage.
A confidence interval or other uncertainty summary should state the method and assumptions. Small samples, clustered observations, repeated observations from the same policyholder, and time-dependent data may invalidate a simple independent-observation calculation. A resampling method can be useful, but must preserve the dependence structure and evaluation design. Whatever method is used, uncertainty intervals describe variability under their assumptions; they do not guarantee future performance.
Worked Example: Interpreting a Paired Error Difference
On eight hypothetical held-out claims, the absolute errors in dollars are as follows. Model A’s errors are \(8,5,7,4,10,6,5,3\); Model B’s errors for those same claims are \(6,6,6,4,7,8,4,3\). Calculate the average absolute error for each model and a paired 95% \(t\)-interval for the mean difference \(d_i=|e_{Ai}|-|e_{Bi}|\). Use \(t_{0.975,7}=2.365\). Treat this small calculation as an illustration of the method, not as strong evidence for deployment.
Method. Pair errors by claim, calculate each difference, and use the sample standard deviation of those differences. Positive differences indicate lower absolute error for B.
The sum of A’s absolute errors is \(8+5+7+4+10+6+5+3=48\), so its mean is \(48/8=\$6.00\). The sum for B is \(6+6+6+4+7+8+4+3=44\), so its mean is \(44/8=\$5.50\). The observed mean difference is therefore \(\bar d=\$6.00-\$5.50=\$0.50\).
The paired differences are \(2,-1,1,0,3,-2,1,0\). Their deviations from \(0.5\) are \(1.5,-1.5,0.5,-0.5,2.5,-2.5,0.5,-0.5\). The squared deviations sum to \(2.25+2.25+0.25+0.25+6.25+6.25+0.25+0.25=18\). Thus \(s_d=\sqrt{18/7}\approx\$1.6036\), and \(SE(\bar d)=1.6036/\sqrt{8}\approx\$0.5669\).
The interval is \(0.50\pm2.365(0.5669)\). The margin is approximately \(\$1.340\), giving \((-\$0.84,\$1.84)\) after rounding to cents. The interval includes zero, so this small sample does not clearly establish which model has the lower mean absolute error. The calculation avoids a common mistake: treating the two sets of errors as unrelated when each pair comes from the same claim.
Check. The average of the eight paired differences is \((2-1+1+0+3-2+1+0)/8=4/8=\$0.50\), agreeing with the difference between the two mean absolute errors.
Justification for the Paired Comparison
The reason to calculate paired differences is that the target is the average advantage of one model over the other on the same evaluated cases. For each case, \(d_i\) records that case’s contribution to the comparison. Averaging those contributions gives
The equality follows by distributing the sum across subtraction. Therefore, the paired mean difference equals the difference between the two sample mean absolute errors, while retaining the observation-by-observation pairing needed to estimate variability in that difference. Ignoring the pairing discards information about which observations favor which model and can give an inappropriate uncertainty estimate.
Present Trade-Offs, Not a Universal Winner
A decision-ready comparison states what each candidate does well, where it is weaker, and what is uncertain. The appropriate conclusion may be conditional: one model has a modest error advantage, while another has a lower operational burden; or a model improves a costly error type while increasing a less costly one. The decision-maker can then weigh these facts against the actual use case rather than being asked to accept an unsupported overall ranking.
Keep the claims proportional to the evidence. A lower score on one test sample supports a statement about that sample and metric. It does not, by itself, prove that the difference is statistically reliable, that it will persist over time, or that the model is fair across relevant groups. Those are distinct questions that require appropriate evaluation. Explain the scope and limitations of the comparison in plain language.
Name the target, population, evaluation period, held-out data, metric, and any decision threshold or cost assumptions.
Give both candidates’ results and the absolute difference; add a relative change only when its reference value is clear and meaningful.
Explain what the metric difference means for the intended action and identify relevant trade-offs, such as missed cases versus unnecessary reviews.
Describe uncertainty, assumptions, and material limitations. Recommend a model only in relation to stated decision criteria.
For a professional report aligned with the SOA Exam PA (Predictive Analytics) syllabus, a fair comparison is not just a contest between scores. It is a traceable explanation of what was compared, how large the differences were, what they imply for the intended decision, and what remains uncertain.
Check Your Understanding
For each question, compare candidates only on the stated basis and interpret the result in context.
- Two numeric prediction models are evaluated on the same 50 observations. Their sums of squared errors are 1,250 and 1,125. Calculate each RMSE and the relative RMSE reduction for the second model. (Answer: RMSEs are 5.00 and approximately 4.743; the reduction is approximately 5.1%.)
- On a common test set, Model A has 6 false negatives and 18 false positives; Model B has 4 false negatives and 28 false positives. If a false negative costs $75 and a false positive costs $10, calculate each total error cost and identify the lower-cost model on this test set. (Answer: A costs $630; B costs $580; B is lower by $50 under the stated costs.)
- For paired observations, the differences \(d_i=|e_{Ai}|-|e_{Bi}|\) are \(1,2,-1,0\). Calculate the mean difference and state which model has the lower mean absolute error in this sample. (Answer: \(\bar d=0.5\); Model B has lower mean absolute error by $0.50 if errors are in dollars.)
- A report says Model B’s accuracy is higher, but it was measured on a different test population and at a different threshold from Model A. Give two reasons the comparison is not yet fair and one remedy. (Answer: The evaluation populations and thresholds differ, so the scores may reflect different case mixes and decision rules; evaluate on a common held-out population and state a consistent, appropriately selected threshold procedure.)
- A paired error comparison favors Model B, but its uncertainty interval for the mean difference includes zero. What conclusion is justified, and what conclusion is not justified? (Answer: The observed sample favors B, but the evidence does not clearly establish a nonzero mean advantage under the interval’s assumptions; claiming that B is definitively superior is not justified.)