Comparing Candidate Models on a Common Criterion

Why Comparisons Need a Common Basis

A model comparison is meaningful only when the candidates are judged on the same task. If one model is evaluated on a different group of cases, against a different baseline, or using a different definition of success, a better-looking score may reflect the comparison rather than better predictions. In practice, a model can also be preferable for one operational goal and inferior for another.

The comparison should therefore make three choices explicit: which cases represent the target population, what baseline provides the reference point, and what operational outcome defines success. These choices belong to the evaluation design, not to the model that happens to score best.

Definition: A common-criterion comparison evaluates candidate models on the same target population, using the same baseline and the same success criterion. The criterion may be a prediction metric, such as mean absolute error, or an operational quantity, such as net value under a specified action policy. A model's preference is conditional on those choices.

For a numeric prediction task, a baseline might predict the same historical average for every case. For a classification task, it might be a defined existing policy, such as taking no action. A baseline is not automatically a weak model: it is a reference that helps answer whether a candidate improves on a simple or current approach.

The evaluation population must also be fixed. That means comparing predictions for the same eligible cases, with the same observed outcomes and the same inclusion rules. If cases are excluded from one model's evaluation but not another's, the reported scores do not answer a like-for-like question.

A Practical Comparison Framework

Before ranking candidates, record the comparison specification. State what counts as one case, what outcome is being predicted, which observations are included, what the baseline does, and how the result will be judged. For an operational criterion, also state the action policy and how the resulting outcomes are valued. The values and rules must be held fixed across the candidates.

Fix the target population.
Specify the cases to which the eventual decision will apply, then use the same evaluation cases and outcome definition for every candidate.

Set the baseline.
Define the reference prediction or policy before inspecting which candidate wins. Compare every candidate with that same reference.

Define success operationally.
Choose the quantity that answers the real question, such as error on a stated scale or net value under a stated action rule.

Apply the same calculation.
Use identical cases, units, weights, thresholds, and valuation assumptions for each candidate. Report the underlying quantities as well as the resulting comparison.

The framework does not imply that there is one universally best model. It means that a conclusion is interpretable: “Candidate B performs better under this population, baseline, and success criterion.” That is more precise than saying simply that B is “better.”

Comparing Numeric Predictions Against a Baseline

For numeric predictions, suppose a data set contains $n$ evaluation cases, with actual outcomes $y_1,\ldots,y_n$ and predictions $\hat{y}_1,\ldots,\hat{y}_n$. Under a mean absolute error criterion, the average absolute prediction error is:

\operatorname{MAE}=\frac{1}{n}\sum_{j=1}^{n}|y_j-\hat{y}_j|.

This formula is useful only when the cases and outcome scale are comparable. The baseline MAE must be calculated on those same cases and in those same units. If smaller error is the stated objective, a candidate with lower MAE than the baseline improves on it for this criterion; that conclusion does not automatically establish better performance under another objective.

Worked Example: A Common Numeric Baseline

A hypothetical service team forecasts the number of daily equipment checks needed. On four evaluation days, the actual counts are $10, 20, 30, 40$. A baseline predicts $25$ checks each day. Candidate A predicts $12, 18, 33, 37$, while Candidate B predicts $9, 23, 27, 41$. Compare the candidates using MAE on these same four days, and calculate improvement relative to the baseline.

Method. Compute absolute errors by day, then average them. Use the baseline on exactly the same four cases.

For the baseline, the absolute errors are $|10-25|=15$, $|20-25|=5$, $|30-25|=5$, and $|40-25|=15$. Their sum is $15+5+5+15=40$, so its MAE is $40/4=10$ checks.

For Candidate A, the absolute errors are $|10-12|=2$, $|20-18|=2$, $|30-33|=3$, and $|40-37|=3$. Their sum is $2+2+3+3=10$, so its MAE is $10/4=2.5$ checks.

For Candidate B, the absolute errors are $|10-9|=1$, $|20-23|=3$, $|30-27|=3$, and $|40-41|=1$. Their sum is $1+3+3+1=8$, so its MAE is $8/4=2$ checks.

The baseline improvement for A is $(10-2.5)/10=0.75$, or $75\%$. For B, it is $(10-2)/10=0.80$, or $80\%$. Candidate B has the lower MAE and the larger relative reduction from the common baseline. The calculation checks: both candidate MAEs are below the baseline MAE of 10, and $2<2.5$.

Common mistake avoided: comparing A on these four days with a baseline score calculated on a different set of days. A percentage improvement is meaningful here because the reference score and candidate score use the same cases and units.

From Classification Results to Operational Value

For a classification task, a prediction alone does not determine the operational result: a policy must specify what action follows a prediction. The confusion-matrix counts describe the outcomes after that rule is applied. To compare models using operational value, state the value assigned to each relevant outcome and the cost of the action, then apply the same rule and values to every candidate.

For example, if each correctly identified positive case yields benefit $b$, each case acted on incurs cost $c$, and there is no other benefit or cost, the total net value is:

\text{Net value}=b(\text{true positives})-c(\text{cases acted on}).

This formula follows by adding each case's contribution. A true positive that receives the action contributes $b-c$; a false positive that receives it contributes $-c$; a case that receives no action contributes zero under the stated assumptions. Summing across cases gives $b$ times the number of true positives, minus $c$ times the total number of actions. The formula does not apply unchanged if other outcomes have values or costs; those terms must then be included.

Worked Example: Comparing Policies by Net Value

A hypothetical review team evaluates two models on the same 1,000 accounts, of which 100 have the target outcome. Under the prespecified action rule, Model A flags 260 accounts, including 80 true positives. Model B flags 160 accounts, including 70 true positives. Assume each true positive that is acted on produces a benefit of $500, and every flagged account costs $100 to review. The baseline takes no action and has net value $0. Which policy has greater net value?

Method. Apply the stated value formula to each policy, without changing the population, action cost, or baseline.

For Model A, the benefit is $80\times \$500=\$40{,}000$. Review costs are $260\times \$100=\$26{,}000$. Its net value is $\$40{,}000-\$26{,}000=\$14{,}000$.

For Model B, the benefit is $70\times \$500=\$35{,}000$. Review costs are $160\times \$100=\$16{,}000$. Its net value is $\$35{,}000-\$16{,}000=\$19{,}000$.

Relative to the common no-action baseline of $0, A adds $14,000 and B adds $19,000. Model B is preferred under this operational criterion, even though it identifies 10 fewer true positives. Its 100 fewer reviews save $10,000, more than the $5,000 benefit lost from those 10 true positives. The check is $\$19{,}000-\$14{,}000=\$5{,}000$, matching that net trade-off.

Common mistake avoided: choosing the model with more true positives without accounting for the cost of the actions needed to obtain them. The conclusion is specific to the stated benefits, review costs, action rule, and target population.

Why a Model Preference Can Reverse

A preference reversal does not necessarily mean that a calculation is wrong. It may mean that the question has changed. A screening program that requires a minimum sensitivity is asking a different question from a review team seeking the greatest net value. Similarly, a change in the relative importance of errors changes the operational objective, even if the models and evaluation cases are unchanged.

The comparison remains transparent when the goal changes are stated explicitly. Do not report a ranking from one criterion as though it settled a different decision. If a requirement is a threshold, first check which candidates satisfy it; if the goal is to maximize a defined quantity, calculate that quantity for each candidate.

Worked Example: The Goal Changes the Preferred Model

On the same 1,000 hypothetical accounts, 120 have the target outcome. A prespecified rule produces these results: Model A identifies 90 positives and flags 390 accounts in total; Model B identifies 75 positives and flags 175 accounts. The baseline takes no action. Compare the models first for a screening program requiring sensitivity of at least $0.70$, then for net value if each true positive is worth $200 and each flag costs $20.

Method. Calculate sensitivity for the screening requirement, then calculate net value using the same account population and the stated dollar values.

Model A has 30 false negatives because $120-90=30$. Its sensitivity is $90/120=0.75$, so it meets the $0.70$ minimum.

Model B has $120-75=45$ false negatives. Its sensitivity is $75/120=0.625$, so it does not meet the minimum. Under the screening requirement, A is preferred because A qualifies and B does not.

For the net-value goal, A's benefit is $90\times \$200=\$18{,}000$, and its flagging cost is $390\times \$20=\$7{,}800$. Its net value is $\$18{,}000-\$7{,}800=\$10{,}200$.

B's benefit is $75\times \$200=\$15{,}000$, and its flagging cost is $175\times \$20=\$3{,}500$. Its net value is $\$15{,}000-\$3{,}500=\$11{,}500$. The no-action baseline is $0, so both policies exceed it, but B is preferred for net value. The difference is $\$11{,}500-\$10{,}200=\$1{,}300$.

The arithmetic checks against the trade-off: B gives up 15 true positives, worth $15\times \$200=\$3{,}000$, but avoids $390-175=215$ flags, saving $215\times \$20=\$4{,}300$. The net gain is $\$4{,}300-\$3{,}000=\$1{,}300$. Common mistake avoided: calling one model universally superior when it wins under one stated goal and loses under another.

Common Comparison Traps

Several seemingly small differences can invalidate a model ranking. Check the comparison design before interpreting the score.

Different evaluation cases: a model scored on easier cases can appear stronger even if its predictions are no better on the cases that matter.
Different baseline: improvements measured against different reference policies cannot be compared as if they were on a common scale.
Different outcome scale or units: error in dollars cannot be compared directly with error in thousands of dollars without converting units.
Different operational rule: if candidates use different action thresholds or follow-up rules, the comparison includes a policy difference, not only a model difference. State and justify that difference.
Changing the criterion after seeing results: selecting whichever measure favors a preferred candidate makes the conclusion difficult to interpret. Define the decision goal first.
Ignoring the stated population: a result for one population or subgroup need not describe another. A change in who is included can change the ranking.

A concise report should name the evaluation population, baseline, criterion, and winner. It should also say whether the winner satisfies any operational requirement. If more than one legitimate goal is under consideration, present the corresponding results separately rather than disguising a goal change as a single objective ranking.

Key takeaway: A defensible model comparison holds the target population, baseline, and operational success criterion constant. If the preferred model changes, identify which goal or comparison condition changed and explain the resulting trade-off.

Check Your Understanding

For each problem, compare candidates only under the stated population, baseline, and criterion.

A hypothetical outreach team evaluates two rules on 500 accounts, 50 of which have the target outcome. Rule A flags 110 accounts and captures 40 positives; Rule B flags 80 and captures 35. Each captured positive is worth $300 and each flag costs $50. Against a no-action baseline worth $0, calculate both net values and state which rule wins. (Answer: A: $40\times300-110\times50=\$6{,}500$; B: $35\times300-80\times50=\$6{,}500$; tie.)
On three common evaluation days, actual demand is $5,10,15$. The baseline predicts 10 each day; Model A predicts $6,9,14$, and Model B predicts $5,12,15$. Calculate MAE for all three and identify each model's percentage reduction from the baseline. (Answer: Baseline MAE $=10/3$; A MAE $=1$, a $70\%$ reduction; B MAE $=2/3$, an $80\%$ reduction; B has lower MAE.)
A screening evaluation includes 80 positive cases. Model A detects 60 and flags 240 cases; Model B detects 52 and flags 100. Which meets a minimum sensitivity of $0.70$? (Answer: A sensitivity $=60/80=0.75$, so A meets the requirement; B sensitivity $=52/80=0.65$, so B does not.)
A numeric prediction study has 80 urban cases and 20 rural cases. Model A has MAE 2 in urban cases and 10 in rural cases; Model B has MAE 4 in urban cases and 1 in rural cases. Calculate each overall MAE using the stated population weights. (Answer: A $=0.8(2)+0.2(10)=3.6$; B $=0.8(4)+0.2(1)=3.4$; B has lower population-weighted MAE.)
For the same errors as in Question 4, suppose the stated goal changes to giving urban and rural cases equal weight, regardless of their population shares. Calculate the average of the two segment MAEs for each model and identify the preferred model under that goal. (Answer: A $=(2+10)/2=6$; B $=(4+1)/2=2.5$; B is preferred. The criterion's weighting changed, so this is a different comparison.)

What You'll Learn