Why Benchmark Against Current Practice?
A new model may appear useful compared with a simple baseline, such as always predicting the most common class or always predicting the historical average. But that comparison does not answer the operational question: does the model improve on what the organization already does? If an existing model, rule, or professional workflow is already used, it is often the more relevant benchmark.
A fair comparison requires more than naming an incumbent. The candidate and benchmark must be evaluated on the same cases, against the same outcome definition and time horizon, using the same metric. Otherwise, a difference in reported performance may reflect different data or measurement choices rather than better predictions.
For a set of \(n\) evaluation cases, let \(y_i\) denote the observed outcome for case \(i\), \(\hat{y}_i^M\) the candidate model's prediction, and \(\hat{y}_i^B\) the benchmark prediction. For numeric predictions, one possible measure is mean absolute error (MAE):
For binary classifications, accuracy is the fraction of cases whose predicted class matches the observed class. Accuracy is appropriate only when it matches the stated evaluation objective; it should not be chosen merely because it is easy to calculate. The earlier material on operational success and simple baselines provides context for defining the objective and choosing a metric.
What Makes a Comparison Fair?
Before calculating performance, specify the comparison so that both systems are assessed on equal terms. A benchmark may be an incumbent statistical model, a fixed rule, or a structured human process. If a human process is the comparator, define what it produces—such as a recorded risk category or an estimated amount—and when that output is made. A vague description such as “what staff usually do” is not enough to reproduce or evaluate the benchmark reliably.
Define the outcome and the prediction horizon. For example, specify whether a prediction concerns a claim during the next 12 months or during the next 30 days.
Use the prediction or decision the current process would actually have produced at the relevant decision time. Do not reconstruct it using information that was unavailable then.
Match candidate and benchmark outputs to the same observed outcomes and apply the same metric to each.
A benchmark comparison describes performance on the evaluated cases and under the stated process. It does not, by itself, establish that a change in policy will improve future outcomes.
The same-case requirement has a direct mathematical justification. Define \(L_i^M\) and \(L_i^B\) as the candidate's and benchmark's losses on case \(i\), using the same loss definition. Their average loss difference is
This identity holds because both averages use the same \(n\) cases. Comparing separate groups may mix model performance with differences in case composition. For instance, a benchmark assessed on mostly simple cases cannot be fairly compared with a candidate assessed on mostly complex cases. Paired evaluation does not eliminate every source of bias, but it avoids this basic mismatch.
Worked Comparisons
Worked Example: Comparing Two Classification Systems
A hypothetical insurer wants to predict whether a submitted application will require additional documentation. On eight held-out applications, the outcome \(y_i=1\) means documentation was required and \(y_i=0\) means it was not. The current rule and a candidate model produce the following class predictions. Compare them using accuracy.
| Application | Observed outcome | Current rule | Candidate model |
|---|---|---|---|
| 1 | 1 | 0 | 1 |
| 2 | 0 | 0 | 0 |
| 3 | 0 | 0 | 0 |
| 4 | 1 | 1 | 1 |
| 5 | 0 | 0 | 0 |
| 6 | 1 | 1 | 1 |
| 7 | 0 | 0 | 1 |
| 8 | 0 | 0 | 0 |
Method. Count exact matches between each prediction column and the observed outcome column, then divide each count by eight. The evaluation cases and the metric are identical for both systems.
Current rule. Its predictions match the outcomes for applications 2, 3, 4, 5, 6, 7, and 8: seven correct out of eight. Its accuracy is \(7/8=0.875\), or 87.5%.
Candidate model. Its predictions match the outcomes for applications 1, 2, 3, 4, 5, 6, and 8: also seven correct out of eight. Its accuracy is \(7/8=0.875\), or 87.5%.
Check. The candidate corrects the current rule's error on application 1 but makes an error on application 7, where the rule was correct. The two changes offset in the accuracy count. The candidate has not improved accuracy on these cases. The common mistake this comparison avoids is claiming an improvement simply because the candidate fixes one visible error while overlooking the error it introduces.
Worked Example: Benchmarking Numeric Predictions
A hypothetical claims team compares an incumbent estimate of claim severity with a candidate model. The target is the final claim amount in dollars, and both systems are evaluated on the same five claims. Calculate each system's MAE.
| Claim | Final amount | Incumbent estimate | Candidate estimate |
|---|---|---|---|
| 1 | $100 | $120 | $100 |
| 2 | $200 | $180 | $220 |
| 3 | $300 | $330 | $260 |
| 4 | $400 | $360 | $380 |
| 5 | $500 | $550 | $520 |
Method. For each claim, calculate the absolute difference between the estimate and final amount. Add the five errors and divide by five.
Incumbent errors. The absolute errors are \(|100-120|=\$20\), \(|200-180|=\$20\), \(|300-330|=\$30\), \(|400-360|=\$40\), and \(|500-550|=\$50\). Their sum is \(20+20+30+40+50=\$160\). Therefore, the incumbent MAE is \(\$160/5=\$32.00\).
Candidate errors. The absolute errors are \(|100-100|=\$0\), \(|200-220|=\$20\), \(|300-260|=\$40\), \(|400-380|=\$20\), and \(|500-520|=\$20\). Their sum is \(0+20+40+20+20=\$100\). Therefore, the candidate MAE is \(\$100/5=\$20.00\).
Check. On these five claims, the candidate's MAE is \(\$32.00-\$20.00=\$12.00\) lower. This is a direct comparison with the incumbent on the same claims, not evidence by itself about performance on all future claims. The common mistake avoided is comparing dollar errors from different claim sets or using different target definitions.
Worked Example: When Current Decisions Shape the Benchmark
A hypothetical retention team makes renewal offers using its existing process and records the process's predicted renewal probabilities. A candidate model is intended to forecast renewal under that same offer process. On four evaluation cases, the observed renewal outcomes are \(1,0,0,0\). The incumbent probabilities are \(0.7,0.6,0.4,0.3\), and the candidate probabilities are \(0.8,0.4,0.2,0.2\). Compare MAE, and also calculate the error for a simple constant baseline equal to the observed evaluation-set average.
Incumbent MAE. The four absolute errors are \(|1-0.7|=0.3\), \(|0-0.6|=0.6\), \(|0-0.4|=0.4\), and \(|0-0.3|=0.3\). Their sum is \(0.3+0.6+0.4+0.3=1.6\), so the incumbent MAE is \(1.6/4=0.400\).
Candidate MAE. The four absolute errors are \(|1-0.8|=0.2\), \(|0-0.4|=0.4\), \(|0-0.2|=0.2\), and \(|0-0.2|=0.2\). Their sum is \(0.2+0.4+0.2+0.2=1.0\), so the candidate MAE is \(1.0/4=0.250\).
Simple constant baseline. The observed mean is \((1+0+0+0)/4=0.25\). Predicting \(0.25\) for each case gives absolute errors \(0.75, 0.25, 0.25, 0.25\), with sum \(1.50\) and MAE \(1.50/4=0.375\).
Check and interpretation. On these cases, the candidate MAE of \(0.250\) is below both the incumbent MAE of \(0.400\) and the simple-baseline MAE of \(0.375\). But the incumbent is the more operationally relevant comparator if the model is intended to improve the current renewal process. The outcome was observed under that process; a different offer policy could change the outcomes. This calculation does not estimate what would have happened under a different policy. The common mistake avoided is treating a comparison with a simple constant prediction as proof that a proposed change to current decisions will improve renewal outcomes.
Common Benchmarking Traps
- Using only a simple baseline. A constant prediction can show that a model learns more than a basic rule, but it may not show that the model improves on current practice.
- Comparing different cases or periods. Differences in case mix or time period can affect the metric. Compare both systems on the same eligible cases whenever possible.
- Changing the target or horizon. A prediction of claims over 12 months is not directly comparable with a benchmark for claims over six months. Define the same outcome window for both.
- Reconstructing the incumbent with future information. The benchmark should reflect what the process could have produced at the decision time, not a later version informed by outcomes or subsequent records.
- Confusing prediction performance with policy impact. Historical outcomes reflect the decisions made in that setting. A better prediction score does not, by itself, demonstrate the effect of replacing or changing those decisions.
- Reporting only a favorable comparator. If current practice is the operational alternative, report the candidate's comparison with current practice rather than highlighting only an easier baseline.
Check Your Understanding
For each problem, identify the appropriate comparison and calculate the requested result where data are provided.
- On six applications, observed outcomes are \(1,0,1,0,1,0\). Current-rule predictions are \(1,0,0,0,1,0\); candidate predictions are \(1,1,1,0,0,0\). Calculate both accuracies and state which is higher. (Answer: Current rule \(5/6=83.33\%\); candidate \(4/6=66.67\%\); the current rule is higher.)
- Final amounts for three claims are \(\$80,\$120,\$200\). Incumbent estimates are \(\$100,\$100,\$230\); candidate estimates are \(\$90,\$150,\$180\). Calculate each MAE and the candidate's reduction in MAE. (Answer: Incumbent MAE \(\$70/3=\$23.33\); candidate MAE \(\$60/3=\$20.00\); reduction \(\$3.33\), rounded to cents.)
- A candidate model is intended to replace a current staff process that assigns each case a recorded low, medium, or high risk category. The historical file contains the staff category and the eventual outcome for each case. What must be specified to make the staff process a usable benchmark? (Answer: Define the category output and decision time, outcome and horizon, eligible cases, and evaluation metric; compare with candidate results on the same cases.)
- For four cases, observed outcomes are \(1,0,0,1\). Incumbent probabilities are \(0.8,0.6,0.2,0.3\); candidate probabilities are \(0.7,0.3,0.1,0.8\). Calculate both MAEs. (Answer: Incumbent errors sum to \(0.2+0.6+0.2+0.7=1.7\), so MAE \(0.425\); candidate errors sum to \(0.3+0.3+0.1+0.2=0.9\), so MAE \(0.225\).)
- A historical customer file records renewal outcomes under an existing offer policy. A proposed model would select a different group of customers for offers. Why is comparing its historical prediction score with the incumbent not enough to establish the effect of the new offer policy? (Answer: Historical outcomes were observed under the existing policy, not the proposed one; prediction-score comparison alone does not establish outcomes under changed decisions.)