From Model Comparison to a Defensible Recommendation
A model comparison is not complete when one candidate has the best score in a table. A recommendation must explain which model is suitable for the specified use, what evidence supports that conclusion, and what qualifications remain. A small average gain may not justify added complexity; a serious diagnostic defect may outweigh an attractive aggregate metric; and a model that cannot meet a required operating constraint is not a viable choice for that use.
The task here is to make a comparative model recommendation, not to prescribe a downstream business action. Keep the intended predictive use explicit, distinguish validation evidence from operational facts, and state what the evidence does not establish. A recommendation written this way is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
A Structure for the Recommendation
A concise recommendation can be developed in four steps. The steps are not a scoring formula: a strong recommendation does not assign arbitrary points to interpretability, performance, and cost. Instead, it explains which evidence is important for the intended use and whether each candidate meets the necessary conditions.
Specify what the model is expected to predict and identify the metric or performance requirement that matters for that use.
Use the same validation partitions, target definition, preprocessing rules, and evaluation method for the candidates. Describe the size and consistency of any difference, not only which average is larger or smaller.
Ask whether the error pattern is acceptable for the intended use and whether each candidate meets relevant requirements for data availability, interpretation, latency, and maintenance.
Identify the preferred model, give the main reasons, and state material limitations that affect how broadly the recommendation can be relied upon.
Matched validation means that each candidate is evaluated on the same validation observations under comparable conditions. For a metric \(M\), let \(M_{Aj}\) and \(M_{Bj}\) be the values for models A and B on validation fold \(j\). The mean difference is
This identity follows by distributing the sum: \(\sum_j(M_{Bj}-M_{Aj})=\sum_j M_{Bj}-\sum_j M_{Aj}\), and dividing each sum by the same number of folds \(K\) gives the difference in the two means. Pairing makes the comparison transparent: each difference is calculated on the same fold. It does not, by itself, prove that the observed difference will persist in future data. Variation across folds and the broader validation design still matter.
The sign of the difference must be interpreted in context. Higher AUC or sensitivity may be desirable for a classification use, while lower RMSE or MAE is generally desirable for a regression use. No metric should be called “better” without identifying which direction is preferred and why it is relevant.
Worked Example: A Small Classification Gain With a Threshold Requirement
A hypothetical service team wants to rank applications for further review. It also requires sensitivity of at least 0.70 at a fixed operating threshold. Two classifiers are evaluated on the same five validation folds. The threshold was set before these results were examined. AUC measures ranking performance; sensitivity at the fixed threshold measures the proportion of actual positive cases classified as positive.
| Fold | Model A AUC | Model B AUC | Model A sensitivity | Model B sensitivity |
|---|---|---|---|---|
| 1 | 0.78 | 0.80 | 0.70 | 0.75 |
| 2 | 0.80 | 0.81 | 0.72 | 0.74 |
| 3 | 0.79 | 0.80 | 0.68 | 0.71 |
| 4 | 0.77 | 0.79 | 0.71 | 0.74 |
| 5 | 0.81 | 0.82 | 0.69 | 0.71 |
Method. Compare both metrics on the matched folds, then check the stated sensitivity requirement. For AUC, the fold-by-fold differences \(B-A\) are 0.02, 0.01, 0.01, 0.02, and 0.01. Their sum is 0.07, so the mean difference is \(0.07/5=0.014\). Model A has mean AUC \((0.78+0.80+0.79+0.77+0.81)/5=3.95/5=0.790\). Model B has mean AUC \((0.80+0.81+0.80+0.79+0.82)/5=4.02/5=0.804\). The arithmetic checks: \(0.804-0.790=0.014\).
For sensitivity, Model A's mean is \((0.70+0.72+0.68+0.71+0.69)/5=3.50/5=0.700\). Model B's mean is \((0.75+0.74+0.71+0.74+0.71)/5=3.65/5=0.730\). Model A is below 0.70 on two folds (0.68 and 0.69), while Model B is at or above 0.70 on all five.
Recommendation. If the required sensitivity must be met consistently across these validation folds, Model B is better supported for this use: it has higher mean AUC and meets the sensitivity requirement on each fold. The AUC gain is modest, so the evidence does not justify claiming a large or certain improvement. The assessment is limited to these validation folds and this fixed threshold; it does not establish performance under future population or operating changes. The common mistake avoided is choosing Model A because its mean sensitivity equals the requirement while ignoring that it falls below the requirement on two folds.
Combine Metrics With Diagnostic and Operational Evidence
Aggregate metrics summarize performance but can hide patterns that matter to the intended use. Earlier material on error profiles and residual diagnosis provides tools for detecting those patterns. In a recommendation, report the implication of a diagnostic finding rather than listing every diagnostic in isolation. For example, a model with a lower overall error may still be a poor choice if its errors are concentrated in a segment that the intended use must serve reliably.
Practical requirements also need to be explicit. Some are preferences, such as a desire for a simpler model; others are constraints, such as a maximum prediction time or required availability of input variables. A candidate that violates a necessary constraint should not be recommended for that use merely because it has the best validation metric. Where a requirement is a preference rather than a hard limit, explain the trade-off instead of treating it as an automatic exclusion.
Worked Example: Regression Metrics and an Error-Pattern Concern
A hypothetical maintenance group predicts repair cost in dollars to help plan workload. Two models are evaluated using the same five folds. The team values both squared-error performance and the typical size of an error. RMSE and MAE are reported in dollars; lower values are preferable. A residual review also flags possible underprediction for the highest-cost cases.
| Fold | GLM RMSE | Boosted model RMSE | GLM MAE | Boosted model MAE |
|---|---|---|---|---|
| 1 | 104 | 98 | 72 | 75 |
| 2 | 98 | 96 | 69 | 67 |
| 3 | 101 | 100 | 71 | 73 |
| 4 | 99 | 97 | 68 | 65 |
| 5 | 103 | 99 | 70 | 72 |
Method. Calculate the mean of each metric and compare the error scales. The GLM's mean RMSE is \((104+98+101+99+103)/5=505/5=\$101\). The boosted model's mean RMSE is \((98+96+100+97+99)/5=490/5=\$98\). The absolute reduction is \(\$101-\$98=\$3\). Relative to the GLM, this is \(3/101=0.0297\), or about 2.97%.
The GLM's mean MAE is \((72+69+71+68+70)/5=350/5=\$70\). The boosted model's mean MAE is \((75+67+73+65+72)/5=352/5=\$70.40\). Thus, RMSE favors the boosted model, but mean MAE is \(\$0.40\) higher for it. These two results are not contradictory: RMSE gives greater influence to larger errors than MAE does.
Recommendation. The boosted model has a modest advantage in mean RMSE, but it does not improve mean MAE, and the residual review raises a concern about underprediction among the highest-cost cases. The evidence does not support an unqualified recommendation based only on the RMSE ranking. For workload planning, the team should investigate whether the flagged high-cost errors are material to that use and compare the candidates on that segment before preferring the boosted model. The common mistake avoided is treating the lowest aggregate RMSE as conclusive while disregarding the other metric and the error pattern. No conclusion about the high-cost segment's future performance is warranted from the stated aggregate values alone.
Check Feasibility Before Preferring a Candidate
A model can perform well in validation and still be unsuitable for its stated use. Distinguish evidence about predictive performance from evidence about deployment feasibility. For instance, validation scores do not show that a required predictor will be available at prediction time; an operational test may be needed to establish that. If a candidate depends on an input that is unavailable for part of the intended population, state that restriction rather than quietly assuming complete availability.
Worked Example: A Better Average Score That Misses Operating Limits
A hypothetical equipment-inspection program predicts inspection cost, in dollars, for every incoming work order. Its requirements are prediction latency below 40 milliseconds and availability of all required inputs for every work order. Five matched validation folds give the RMSE values below. A separate operational check finds that Model B takes 65 milliseconds per prediction and requires an external input unavailable for 8% of incoming work orders. Model A takes 4 milliseconds, and its inputs are available for all work orders.
| Fold | Model A RMSE | Model B RMSE |
|---|---|---|
| 1 | 52 | 51 |
| 2 | 50 | 49 |
| 3 | 51 | 50 |
| 4 | 49 | 48 |
| 5 | 53 | 52 |
Method. Model A's mean RMSE is \((52+50+51+49+53)/5=255/5=\$51\). Model B's is \((51+49+50+48+52)/5=250/5=\$50\). The reduction is \(\$1\), or \(1/51=0.0196\), approximately 1.96% relative to Model A. The fold differences \(B-A\) are \(-1,-1,-1,-1,-1\), so the reduction is consistent across these folds.
Recommendation. Recommend Model A for the stated use. Model B's lower RMSE is a small validation advantage, but its 65-millisecond latency exceeds the 40-millisecond limit, and its required input is unavailable for 8% of work orders. Model A meets both operating requirements. This recommendation is specific to a use that requires coverage of every incoming work order; it does not establish that Model A is preferable for a different use with different requirements.
Common mistake and correction. The wrong calculation is to call Model B “about 2% better” and recommend it because \(1/50=2\%\), using the candidate's RMSE as the denominator while also ignoring the operating limits. For a reduction relative to Model A, the denominator is Model A's RMSE: \(1/51=1.96\%\). More importantly, even that correctly calculated gain does not override explicit feasibility constraints. The preferred model is the one fit for the specified use, not necessarily the one with the smallest validation error.
Write the Conclusion With Appropriate Qualifications
A defensible recommendation is specific, proportionate to the evidence, and bounded by the validation design. It should name the preferred candidate, identify the strongest reasons for preferring it, and acknowledge limitations that could change the conclusion. Avoid vague statements such as “Model B is best” or “the model is accurate.” State which use is under consideration and which evidence supports the preference.
A useful final check is to ask whether a reader can distinguish each of the following:
- Observed evidence: the metrics, diagnostics, and operational results actually reported.
- Interpretation: why those results matter for the specified use.
- Limitation: what the evaluation does not establish, such as performance under a different population or operating condition.
- Recommendation: which candidate is preferred for that use, and on what basis.
Do not convert a predictive comparison into a causal claim. A lower validation error supports a claim about predictive performance under the evaluation design; it does not show that a predictor causes the outcome or that changing the predictor would change the outcome. Similarly, a recommendation based on one validation design should not be described as universally optimal.
Check Your Understanding
For each hypothetical comparison, identify the defensible recommendation or the additional qualification needed. Treat stated requirements as applying to the intended use.
- Two classifiers are tested on the same four folds. Their AUCs are 0.76, 0.78, 0.77, 0.79 for Model A and 0.78, 0.79, 0.78, 0.80 for Model B. Calculate both means and the paired mean difference. What is the appropriately qualified comparative conclusion? (Answer: Model A mean AUC is 0.775; Model B mean AUC is 0.7875; the matched mean gain for B is 0.0125. B has higher AUC on every fold, but the gain is modest and does not alone establish a practically important or future-performance advantage.)
- For a regression use, Model A has fold RMSE values 20, 22, 21, and 21. Model B has 19, 22, 20, and 21. Calculate the means. A required diagnostic review has not been done. What conclusion is justified? (Answer: Model A mean RMSE is 21; Model B mean RMSE is 20.5. B has a 0.5-unit lower mean RMSE, but the gain is small and a comparative recommendation should be qualified pending relevant diagnostic review.)
- A classification model has sensitivity values 0.80, 0.78, 0.69, and 0.81 across matched validation folds. The stated requirement is sensitivity of at least 0.75 on every fold. Does the model meet the requirement, even if its mean sensitivity is high? (Answer: No. Its mean is \(3.08/4=0.770\), but the third fold's sensitivity, 0.69, is below 0.75.)
- Two models have mean RMSE values of \$84 and \$81 on matched folds. The second model requires an input unavailable for 12% of cases, while the intended use requires predictions for all cases. Which model is supportable for that use? (Answer: The \$84 model is supportable if its inputs and other requirements are met; the lower-RMSE model fails the stated all-cases input-availability requirement.)
- A candidate has lower average error overall but substantially larger errors in a subgroup that the intended use must serve. What should a recommendation do before naming it the preferred model? (Answer: Report and assess the subgroup error pattern in context; do not rely on the overall average alone. If the subgroup performance is unacceptable, qualify or withhold the preference until the concern is resolved.)