Choosing Between Accuracy, Interpretability, and Operating Cost

Model Choice Is a Decision, Not a Leaderboard

A candidate model with the best validation score is not automatically the best model to deploy. A model may require more time to score cases, more specialist effort to maintain, or more explanation than the business can provide. Its errors may also have consequences that are materially different from what an overall accuracy measure suggests. The practical question is therefore not just “Which model predicts best?” but “Which model meets the requirements, and what are the costs and consequences of using it?”

This distinction matters in actuarial work. For example, a hypothetical insurer might need a prediction to be available during a customer interaction, provide a reviewable reason for a referral, and fit within a fixed annual support budget. Those requirements can rule out a model with a better validation metric. Conversely, if a model’s errors lead to costly missed cases, a higher operating cost may be justified by lower expected error costs.

Definition: A hard constraint is a requirement a model must satisfy to be considered, such as a maximum scoring time or a requirement to provide an explanation for each decision. A preference is a factor used to choose among models that satisfy the hard constraints, such as lower annual cost or better ranking performance. The feasible set is the set of candidate models that meet every specified hard constraint.

Write down the constraints before comparing preferences. This avoids treating a mandatory requirement as something that can simply be traded away for a better metric. Among feasible candidates, state the decision rule: for example, choose the model with the best prespecified validation measure, or choose the model with the lowest estimated total annual cost. If more than one criterion matters, explain how the organization prioritizes them rather than concealing the trade-off in a single score.

Compare Practical Criteria Explicitly

A useful comparison considers criteria that are relevant to the intended use. These can include model performance on an appropriate validation set, explanation needs, scoring time, implementation and maintenance effort, and the consequences of errors. Each criterion needs context. A scoring time of 20 milliseconds might be acceptable in a nightly batch but unacceptable in a system that must respond immediately to a customer.

Criterion	Question to ask	How it can affect the choice
Predictive performance	Which prespecified validation measure matches the intended use?	May distinguish candidates that meet operational requirements.
Interpretability	Who needs to understand or review the prediction, and at what level?	A mandatory explanation can exclude a candidate.
Scoring and operating cost	How much time, computing capacity, and support does production use require?	Can make a higher-performing model impractical or more expensive.
Implementation effort	What work is required to build, integrate, test, and maintain the model?	One-time effort can be compared over a stated planning horizon.
Error consequences	What does each kind of incorrect prediction cause?	Unequal consequences can make total error count misleading.

Where criteria are numerical, check their direction: some are to be maximized, such as a performance measure, while others are to be minimized, such as cost or latency. For a simple dominance check, candidate A dominates candidate B if A is no worse on every relevant criterion and strictly better on at least one, using the correct direction for each criterion. The candidate that is worse is Pareto-dominated and need not be preferred over the candidate that dominates it. If each candidate has an advantage on some criterion, neither dominates the other; the choice then depends on constraints and stated priorities.

Worked Example: Applying Hard Constraints Before Ranking

A hypothetical claims team compares three models using the same validation data. Its production rule requires a case-level explanation and a 95th-percentile scoring time of no more than 15 milliseconds. Among models that meet both requirements, the team will choose the one with the highest area under the receiver operating characteristic curve (AUC). The results are:

Model L: AUC 0.79, scoring time 5 milliseconds, case-level explanations available.
Model T: AUC 0.77, scoring time 3 milliseconds, case-level explanations available.
Model B: AUC 0.84, scoring time 22 milliseconds, no case-level explanations.

Step 1: Apply the explanation requirement. Models L and T provide case-level explanations. Model B does not, so B is infeasible, even though it has the highest AUC.

Step 2: Apply the scoring-time requirement. L takes 5 milliseconds, which is at most 15 milliseconds. T takes 3 milliseconds, also at most 15 milliseconds. Both remain feasible.

Step 3: Apply the stated preference. Among feasible models, L has AUC 0.79 and T has AUC 0.77. Under the team’s stated rule, choose L. T is not selected despite its shorter scoring time because scoring time is a constraint already met by both candidates, not the tie-breaking preference.

Check. L satisfies both requirements and has the higher AUC among feasible candidates. The conclusion follows from the stated requirements and decision rule, not from a claim that L is best for every possible use. The common mistake is to select B by AUC alone, ignoring two unmet production requirements.

Translate Error Types Into Consequences

A count of incorrect predictions treats all errors as equal. That may be unsuitable when, for example, a missed high-cost case has a different consequence from an unnecessary review. If the consequences can be expressed in monetary terms, expected error cost provides a common basis for comparison. The amounts and error rates must refer to the same population and time period, and the cost assigned to an error should represent the consequence relevant to the decision.

Formula: Suppose each case falls into one of several mutually exclusive error categories $j$, with probability $p_j$ and cost $c_j$ per error. If $N$ comparable cases are assessed over the period, and the model has fixed operating cost $C_{\mathrm{run}}$ and implementation cost $C_{\mathrm{impl}}$ spread evenly over $H$ years, an estimate of annual total cost is $$ C_{\mathrm{annual}} = N\sum_j p_j c_j + C_{\mathrm{run}} + \frac{C_{\mathrm{impl}}}{H}. $$ This calculation assumes the category probabilities and per-error costs are applicable to the cases and period being assessed, the cost components are additive, and implementation cost is allocated evenly without discounting or salvage value. If implementation costs are not to be amortized, or if the financial comparison uses discounting, this particular annualization is not appropriate.

The error-cost term follows from expected value. For one case, let the cost be $c_j$ if error category $j$ occurs and zero otherwise. The expected cost is the sum of each possible cost multiplied by its probability, $\sum_j p_jc_j$. Across $N$ cases, expected total error cost is $N\sum_j p_jc_j$, by additivity of expectation. This multiplication does not require the cases to be independent; it does require the probabilities and costs to describe the cases being evaluated.

Worked Example: Different Error Counts, Different Error Costs

A hypothetical screening program expects 10,000 cases per year. A false negative costs an estimated $2,000, while a false positive costs $50. On the relevant validation sample, Model G produces 8 false negatives and 160 false positives per 10,000 cases. Model E produces 14 false negatives and 80 false positives per 10,000 cases. Annual operating costs are $8,000 for G and $15,000 for E. Compare their estimated annual total costs.

Step 1: Calculate Model G’s annual error cost. Its false-negative cost is $8 \times \$2{,}000 = \$16{,}000$. Its false-positive cost is $160 \times \$50 = \$8{,}000$. Therefore, its total annual error cost is $\$16{,}000+\$8{,}000=\$24{,}000$.

Step 2: Add Model G’s operating cost. Its estimated annual total is $\$24{,}000+\$8{,}000=\$32{,}000$.

Step 3: Calculate Model E’s annual error cost. Its false-negative cost is $14 \times \$2{,}000 = \$28{,}000$. Its false-positive cost is $80 \times \$50 = \$4{,}000$. Therefore, its total annual error cost is $\$28{,}000+\$4{,}000=\$32{,}000$.

Step 4: Add Model E’s operating cost. Its estimated annual total is $\$32{,}000+\$15{,}000=\$47{,}000$. Under this cost assumption, G has an estimated annual total cost $\$47{,}000-\$32{,}000=\$15{,}000$ lower than E.

Check and common mistake. G makes $8+160=168$ errors; E makes $14+80=94$. Choosing E just because it makes fewer errors would ignore their unequal consequences. The cost calculation correctly accounts for the more expensive false negatives and includes each model’s operating cost. The conclusion depends on the hypothetical costs being appropriate and sufficiently complete.

Include Implementation and Operating Effort

A model may require substantial initial work before it can score a single production case. For a planning comparison, a one-time implementation cost can be spread over an explicitly chosen horizon, as in the formula above. This is a simplifying accounting assumption, not a claim that costs occur evenly in practice. If the model is expected to be replaced quickly, using a long horizon understates annualized implementation cost. If one candidate’s deployment requires data or system changes that another does not, those differences should be included where they can be estimated.

Operating cost can include more than computing charges. It may include routine monitoring, support, retraining, review capacity, or specialist time. Avoid double-counting: for example, do not add a manual-review expense both as an operating cost and as a separate cost for the same false positive. Keep the comparison on a consistent basis: same expected case volume, time horizon, error-cost definitions, and scope of included expenses.

Worked Example: Adding an Amortized Implementation Cost

A hypothetical property-screening team has confirmed that two models meet its explanation and scoring-time requirements. On expected annual volume, Model S has an estimated error cost of $180,000 and annual operating cost of $25,000. Model E has an estimated error cost of $135,000 and annual operating cost of $42,000. S costs $60,000 to implement; E costs $120,000. The team uses a three-year planning horizon and spreads implementation cost evenly over those three years. Which model has the lower estimated annual total cost?

Step 1: Annualize implementation cost. For S, $\$60{,}000/3=\$20{,}000$ per year. For E, $\$120{,}000/3=\$40{,}000$ per year.

Step 2: Add S’s cost components. Its estimated annual total is $\$180{,}000+\$25{,}000+\$20{,}000=\$225{,}000$.

Step 3: Add E’s cost components. Its estimated annual total is $\$135{,}000+\$42{,}000+\$40{,}000=\$217{,}000$.

Step 4: Compare. E’s estimated annual total is lower by $\$225{,}000-\$217{,}000=\$8{,}000$. Under the stated assumptions and horizon, E is preferred on estimated annual total cost, even though its operating and implementation costs are higher.

Check. E’s $45,000 lower annual error cost exceeds its additional $17,000 operating cost and $20,000 annualized implementation cost. The net advantage is $\$45{,}000-\$17{,}000-\$20{,}000=\$8{,}000$, matching the total-cost comparison. The common mistake is to compare operating costs alone or to omit the stated implementation horizon.

Make the Recommendation Conditional and Auditable

A defensible choice states what was treated as mandatory, what was optimized among feasible candidates, and which assumptions could change the result. If an explanation requirement is mandatory, say which candidates fail it. If expected cost is the decision rule, show its components and identify the error-cost assumptions. If a model wins on one criterion but loses on another, state the trade-off rather than implying that one candidate is universally superior.

Also distinguish a measured result from a practical judgment. Validation performance is estimated from data; operating and error costs may rely on planning assumptions. A recommendation based on estimated annual costs should identify the volume, time horizon, and cost inputs that drive it. If those inputs change, the comparison may need to be repeated. A model that is feasible under current infrastructure may not be feasible after a change in scoring volume or explanation policy.

A frequent exam trap is to rank every candidate by one metric before checking whether it can be used. Another is to call the numerically highest metric “best” without stating why that metric represents the business objective. A sound sequence is: specify the use, apply the hard constraints, compare the remaining candidates using the stated preferences, and check the arithmetic and assumptions. This approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Define the use and requirements.
Specify who will use predictions, what explanations or response times are required, and which limits are mandatory.

Form the feasible set.
Remove candidates that fail any hard constraint before ranking by preferences.

Compare consequences and effort.
Use consistent validation evidence and, where appropriate, calculate error, operating, and annualized implementation costs.

State the decision rule and caveats.
Name the selected candidate, the criteria that determined the choice, and the assumptions that could alter it.

Key takeaway: Choose from models that satisfy the real requirements. Then compare feasible candidates using an explicit decision rule that reflects predictive performance, explanation needs, operating and implementation effort, and the consequences of errors.

Check Your Understanding

For each original scenario, apply the stated constraints or cost rule and check the calculation.

A service requires case-level explanations and scoring within 12 milliseconds. Model A has AUC 0.81, takes 8 milliseconds, and provides explanations. Model B has AUC 0.86, takes 25 milliseconds, and provides no case-level explanations. Which model is feasible? (Answer: Model A only; Model B fails both requirements.)
On the same annual volume, Model M has 6 false negatives and 90 false positives; Model N has 10 false negatives and 40 false positives. A false negative costs $1,000 and a false positive costs $30. Before operating costs, which model has lower error cost, and by how much? (Answer: M; its cost is $6,000 + $2,700 = $8,700, versus N’s $10,000 + $1,200 = $11,200, a difference of $2,500.)
Model P has annual error cost $18,000, operating cost $7,000, and implementation cost $30,000. Model Q has annual error cost $12,000, operating cost $12,000, and implementation cost $18,000. Under a three-year horizon with implementation cost spread evenly, which has lower estimated annual total cost? (Answer: Q, by $5,000; P totals $18,000 + $7,000 + $10,000 = $35,000, while Q totals $12,000 + $12,000 + $6,000 = $30,000.)
All candidates meet the operational requirements. A has AUC 0.80, annual operating cost $8,000, and provides the required explanations. B has AUC 0.83, operating cost $12,000, and provides no explanations. C has AUC 0.78, operating cost $10,000, and provides the required explanations. Which candidate is infeasible under the explanation requirement, and which feasible candidate is Pareto-dominated? (Answer: B is infeasible; C is Pareto-dominated by A, which has higher AUC, lower cost, and the same explanation capability.)
A model produces 5 false negatives and 100 false positives per 5,000 annual cases. A false negative costs $800 and a false positive costs $20; annual operating cost is $4,000. What is the estimated annual total cost, before implementation cost? (Answer: $10,000; error cost is $5 \times \$800 + 100 \times \$20 = \$6,000$, and adding $4,000 operating cost gives $10,000.)

What You'll Learn