Why Put the Comparison in a Decision Table?
A model comparison often contains several kinds of evidence: validation performance, the ease of explaining the model, the resources needed to run it, and known limitations. These considerations do not all have the same units, so reporting them as one unexplained score can conceal the actual trade-off. A decision table keeps the evidence visible and makes the selection rule easier to audit.
The table does not make the decision automatically. Its purpose is to show which candidates satisfy the requirements, where they differ, and why the recommended model is appropriate for the stated use. A defensible recommendation connects the decision to the use case: a small performance gain may matter greatly in one setting and little in another.
Set the Comparison Rules Before Filling the Table
A table is only useful if its entries answer the same question. Before comparing candidates, state the prediction target, the population or period the decision concerns, and the evaluation data used. Performance entries should be based on the same cases, outcome definition, and evaluation procedure. If a metric depends on a classification threshold, report the threshold or make clear that the same decision rule was used for all candidates.
For each metric, state its direction: for example, lower RMSE is better, while higher AUC is better. Do not put a number in the table without its units or context. A value of 0.82 could be an AUC, a proportion, or something else; the heading should make the meaning clear. Earlier material on evaluation metrics and model comparison provides the detailed interpretation of individual measures. Here the emphasis is on presenting them consistently for a decision.
The non-performance columns should also be concrete. “Easy to use” is not a useful operational assessment unless it is tied to the deployment context. Prefer a description such as “scoring code already supported by the existing system” or “requires a separate model-serving process.” Interpretability should describe what a user can reasonably understand from the model, not imply that the model establishes causes.
Identify what action the model comparison is meant to support and the practical requirements that matter.
Use the same evaluation cases, target definition, and metric calculation for all candidates.
Show performance, interpretability, operational cost, and limitations in separate, clearly labelled columns.
Check hard requirements first, then explain any remaining trade-off and the recommendation it supports.
Why Consistent Evaluation Cases Matter
Suppose candidate models \(A\) and \(B\) are scored on the same \(n\) validation cases. Let \(L_{iA}\) and \(L_{iB}\) be their losses on case \(i\), with lower loss representing better performance. The difference between their mean losses can be written as the mean of their case-by-case loss differences:
This equality holds because both averages use exactly the same \(n\) cases, so the sums can be subtracted term by term. It gives a direct comparison of model losses case by case. If the candidates are instead evaluated on different case sets, a difference in mean loss may reflect differences in the cases as well as differences in the models. In that situation, the comparison does not isolate model performance.
For squared-error comparisons, RMSE is the square root of MSE. Because the square-root function is increasing for nonnegative values, two candidates evaluated on the same cases have the same ranking by MSE and RMSE. The numerical difference in RMSE is not, however, the same as the difference in MSE; report the metric that matches the decision and label it accurately.
Worked Example: Comparing Two Models on the Same Validation Cases
A hypothetical insurer compares two models for predicting a claim amount, in dollars, on four validation cases. The actual amounts are $100, $200, $300, and $400. Model A predicts $110, $180, $330, and $380; Model B predicts $120, $190, $310, and $390. Calculate MAE and RMSE for each model, then identify the performance conclusion the decision table should report.
Method. For each case, calculate the residual as prediction minus actual amount. MAE is the mean absolute residual, and RMSE is the square root of the mean squared residual. Both models use the same four cases and the same dollar outcome.
Model A. Its residuals are \(110-100=10\), \(180-200=-20\), \(330-300=30\), and \(380-400=-20\) dollars. The absolute residuals sum to \(10+20+30+20=80\), so \(\mathrm{MAE}_A=80/4=\$20.00\). The squared residuals sum to \(10^2+(-20)^2+30^2+(-20)^2=100+400+900+400=1{,}800\). Thus \(\mathrm{MSE}_A=1{,}800/4=450\) dollars squared, and \(\mathrm{RMSE}_A=\sqrt{450}=\$21.21\), rounded to two decimal places.
Model B. Its residuals are \(120-100=20\), \(190-200=-10\), \(310-300=10\), and \(390-400=-10\) dollars. The absolute residuals sum to \(20+10+10+10=50\), so \(\mathrm{MAE}_B=50/4=\$12.50\). The squared residuals sum to \(20^2+(-10)^2+10^2+(-10)^2=400+100+100+100=700\). Thus \(\mathrm{MSE}_B=700/4=175\) dollars squared, and \(\mathrm{RMSE}_B=\sqrt{175}=\$13.23\), rounded to two decimal places.
Decision-table entry and check. Both measures favor Model B on these four cases: its MAE is $7.50 lower and its RMSE is $7.98 lower after rounding. The square roots also check against the calculated MSE values: \(\sqrt{450}\) is about 21.21 and \(\sqrt{175}\) is about 13.23. The table should label these as validation results from four cases, not as a guarantee of future performance.
Common mistake avoided. Do not compare Model A on one set of cases with Model B on another and attribute the whole difference to the model. Also do not label 450 as Model A’s RMSE: it is its MSE in dollars squared; taking the square root gives RMSE in dollars.
Choose Criteria That Can Change the Decision
A compact table is selective rather than exhaustive. Include the measures and operational facts that could reasonably affect the choice. A typical structure is:
| Candidate | Validation performance | Interpretability | Operational cost | Limitations |
|---|---|---|---|---|
| Model A | Named metric, value, and evaluation context | What can be explained to the intended users | Scoring time, infrastructure, or maintenance needs | Important evidence gaps or known risks |
| Model B | Same metric and evidence basis | Same type of description | Same type of description | Limitations stated in context |
If a metric or a limitation is unknown, mark it as unknown rather than quietly treating it as acceptable. If a candidate fails a non-negotiable requirement, state that plainly. For example, a model that misses a required sensitivity level should not be described as the preferred option merely because it has the largest AUC.
It is useful to separate hard constraints from preferences. A hard constraint rules a candidate in or out, such as an approved maximum scoring time. A preference is a trade-off, such as favoring a simpler model when performance is practically similar. If a preference is expressed through weights or a composite score, the table should disclose how those values were chosen and show whether a reasonable change could alter the ranking. An unexplained total score can suggest a degree of objectivity that the decision process does not have.
Worked Example: A Small Performance Gain Versus Simpler Reporting
A hypothetical underwriting team evaluates two candidate classification models on the same 1,200 validation cases. The agreed minimum sensitivity is 0.70. The team has also stated in advance that an AUC increase smaller than 0.020, by itself, will not justify a substantial increase in explanation and operating burden. At the same classification threshold, the logistic regression model has AUC 0.812, sensitivity 0.70, and specificity 0.76. A tree-based model has AUC 0.824, sensitivity 0.73, and specificity 0.72. The logistic model takes 8 seconds per 10,000 cases to score, compared with 31 seconds for the tree-based model. Compare them in a decision table and recommend one.
Method. First check the hard requirement, then calculate the performance differences, and finally apply the team’s stated preference. The metrics are stipulated to use the same validation cases and the same threshold, so the comparisons are on a consistent basis.
| Candidate | AUC; sensitivity; specificity | Interpretability | Scoring time per 10,000 | Limitation to report |
|---|---|---|---|---|
| Logistic regression | 0.812; 0.70; 0.76 | Relatively direct to summarize | 8 seconds | Lower AUC and sensitivity than the tree-based model |
| Tree-based model | 0.824; 0.73; 0.72 | More involved to explain | 31 seconds | Lower specificity; explanation and operating burden are higher |
Check requirements and differences. Both models meet the sensitivity minimum: the logistic model is exactly at 0.70, and the tree-based model is 0.03 above it. The AUC gain for the tree-based model is \(0.824-0.812=0.012\), which is less than the stated 0.020 threshold. Its sensitivity gain is \(0.73-0.70=0.03\), while its specificity is lower by \(0.76-0.72=0.04\). Its scoring time is \(31-8=23\) seconds more per 10,000 cases.
Recommendation. Given the team’s stated rule, recommend logistic regression for this use. Both candidates meet the minimum sensitivity, and the tree-based model’s AUC increase is below the threshold the team set for accepting greater burden. The recommendation should still report the tree-based model’s higher sensitivity and lower specificity; those are material trade-offs, not details to omit.
Common mistake avoided. Choosing the candidate with the numerically highest AUC without checking the stated requirement would ignore the sensitivity constraint and the trade-off in specificity. The recommendation is conditional on the team’s declared practical threshold; it is not a general rule that logistic regression is always preferable.
Use Constraints to Make a Defensible Selection
When several dimensions matter, a simple sequence can make the reasoning transparent: remove candidates that fail hard constraints, compare the remaining candidates on the important performance measures, then consider interpretability and operating cost in light of the intended use. State any condition on which the choice depends. A recommendation is stronger when a reader can see what evidence would lead to a different choice.
Worked Example: Applying a Daily Scoring-Time Limit
A hypothetical claims team must score 50,000 records per day. It requires sensitivity of at least 0.75 at the chosen operating threshold, and its batch-processing budget is at most 600 seconds per day. Three candidate models have the following validation results and scoring times per 10,000 records: a GLM, sensitivity 0.68 and 12 seconds; a random forest, sensitivity 0.76 and 80 seconds; and a boosted model, sensitivity 0.79 and 170 seconds. All sensitivity values are measured on the same validation cases and threshold. Compare candidates and recommend one under these stated constraints.
Method. Test the sensitivity requirement, then scale the stipulated scoring times to the daily volume. This scaling assumes the scoring environment and batch conditions remain the same and that runtime is approximately proportional to the number of records.
The daily volume is \(50{,}000/10{,}000=5\) blocks of 10,000 records. The GLM’s estimated daily scoring time is \(12\times5=60\) seconds. The random forest’s is \(80\times5=400\) seconds. The boosted model’s is \(170\times5=850\) seconds.
| Candidate | Sensitivity | Estimated seconds per day | Meets sensitivity at least 0.75? | Meets time at most 600 seconds? |
|---|---|---|---|---|
| GLM | 0.68 | 60 | No | Yes |
| Random forest | 0.76 | 400 | Yes | Yes |
| Boosted model | 0.79 | 850 | Yes | No |
Recommendation and check. The GLM fails the sensitivity requirement despite meeting the time budget. The boosted model meets the sensitivity requirement but exceeds the time budget by \(850-600=250\) seconds per day. The random forest meets both constraints, so it is the only feasible candidate and is the recommendation under the stated rules. Its performance is not the highest of the three; feasibility, not a single performance ranking, determines the decision here.
Limitation and common mistake avoided. The time estimates rely on approximately proportional scaling and the stated operating conditions. Production runtime should be checked before implementation. Do not select the boosted model solely because its sensitivity is highest: it violates the explicit daily budget.
Write the Recommendation Next to the Evidence
The table should be followed by a short explanation that names the selected model, the requirements it satisfies, the most important trade-off, and any condition that could change the decision. Avoid a summary that merely repeats every cell. For example: “Select the random forest because it meets the sensitivity and scoring-time requirements; the boosted model has higher sensitivity but exceeds the time budget. Reassess the choice if the daily processing limit changes.” This links evidence to action without suggesting that a validation result guarantees future outcomes.
Limitations should be specific enough to guide a reader’s interpretation. Depending on the case, relevant limitations might include limited evaluation cases for a subgroup, a validation period that may not represent future conditions, dependence on a particular data-processing step, or uncertainty about production runtime. State only limitations supported by the available evidence. A decision table is not a substitute for investigating a concern that could make the comparison unreliable.
Check Your Understanding
For each situation, identify the appropriate decision-table conclusion. Answers are included so you can check your reasoning.
- Two regression models are evaluated on the same 300 cases. Model A has RMSE $42 and Model B has RMSE $39. Which has lower validation RMSE, and by how much? (Answer: Model B, by $3.)
- A candidate has AUC 0.86 but sensitivity 0.64. The stated minimum sensitivity is 0.70. Should it pass the hard requirement? (Answer: No; its sensitivity is 0.06 below the minimum.)
- Model A scores 15,000 records in 45 seconds and Model B scores 15,000 records in 75 seconds. Under approximately proportional runtime, estimate each time for 60,000 records. (Answer: Model A, 180 seconds; Model B, 300 seconds.)
- Two candidates have performance measured on different validation periods, and one has lower error. What must be resolved before using that difference to select a model? (Answer: They should be compared on consistent evaluation evidence, such as the same cases and outcome definition, or the difference may reflect the periods rather than the models.)
- A model has the best validation metric but fails a required maximum scoring time. Another model meets the time limit and all minimum performance requirements. Which is feasible under the stated constraints, and what should the recommendation say? (Answer: The second model is feasible; recommend it under the stated constraints and report that the higher-performing candidate fails the time limit.)