Select a Model Under Interpretability and Runtime Constraints

Model Choice Is an Operational Decision

A claims model can predict well and still be unsuitable for deployment. A complex ensemble may score slowly, require specialist maintenance, or make it difficult to explain a particular referral. A simpler model may give up a modest amount of predictive accuracy while meeting the operation’s explanation and service requirements. The defensible choice depends on the case evidence and the actual constraints, not on a general preference for one algorithm.

This tutorial focuses on selecting among candidates after their predictive performance and operational characteristics have been assessed. Comparisons are meaningful only when the candidates address the same target and use comparable validation evidence. For example, a claim-frequency model evaluated by deviance should not be ranked against a severity model evaluated by RMSE as if the numbers measured the same thing. The discussion is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: A hard constraint is a requirement a candidate must satisfy to be eligible for use, such as a maximum scoring time or a mandated explanation format. A preference is a desirable characteristic used to choose among eligible candidates, such as somewhat lower error or less maintenance effort. A model-selection recommendation identifies the eligible candidate that best addresses the stated preferences and explains any trade-off.

Start by translating the case into decision rules. Separate requirements that cannot be relaxed from preferences that can be traded against one another. “Must produce a reviewable reason for each referral” is a constraint if the operation cannot approve a model without that capability. “Prefer lower deviance” is a preference unless a minimum predictive standard has been specified.

Confirm comparability.
Check that candidates predict the same target, use the same relevant validation population and period, and are compared with the same metric or clearly defined metrics.

List the operating requirements.
Write each hard constraint in a testable form, including its units and threshold. Distinguish an actual requirement from a stakeholder preference.

Remove infeasible candidates.
A candidate that fails a mandatory requirement is not rescued by excellent performance on another dimension.

Compare the feasible candidates.
Use dominance, an agreed scoring rule, or a direct discussion of trade-offs. State why the selected candidate fits this operation.

Qualify the recommendation.
State what evidence supports it and what conditions could change it, such as a tighter service limit or a different explanation requirement.

Constraints, Preferences, and Dominance

A constraint should be applied before preferences. Suppose scoring must finish within 10 milliseconds per claim. A model requiring 40 milliseconds is ineligible even if it has the lowest validation error. Conversely, if the requirement is only that scoring finish within 50 milliseconds, the same candidate remains eligible and its speed may simply be less attractive than a faster alternative.

To compare candidates more formally, orient each measure so its direction is clear. Lower prediction error, runtime, and maintenance effort are generally preferable; higher interpretability or explanation quality is generally preferable. An eligible candidate dominates another when it is at least as good on every relevant dimension and strictly better on at least one, with all dimensions measured consistently.

\text{Candidate A dominates Candidate B if } A_j \geq B_j \text{ for every benefit measure } j, \text{ and } A_k > B_k \text{ for at least one } k.

For measures where lower is better, such as error or runtime, reverse the comparison direction or first convert the measure to a benefit score. Dominance does not require assigning arbitrary weights to different considerations. It identifies a candidate that is no worse on every stated dimension and better on at least one.

Justification: If decision-makers prefer more of every benefit and less of every cost, a dominated candidate cannot be preferred on the listed dimensions: replacing it with the dominating candidate improves at least one dimension and worsens none. This conclusion depends on the measures being comparable and on there being no omitted consideration that favors the dominated candidate.

When no candidate dominates, there is a genuine trade-off. One model may be more accurate while another is faster and easier to explain. A transparent decision rule can help, but a numerical score is not automatically objective: its scales and weights encode stakeholder judgments. Use one only when those choices are stated and defensible.

Worked Example: Applying Mandatory Claims-Scoring Requirements

A hypothetical insurer is comparing three claim-frequency models for daily referral scoring. All candidates were evaluated on the same later-period validation set using mean deviance, for which lower is better. The operation requires mean deviance no greater than 0.835, scoring latency no greater than 10 milliseconds per claim, a usable explanation based on model terms, and no more than 5 staff-days of maintenance per quarter. The lasso model has 0.829 deviance, 6 milliseconds latency, term-based explanations, and 4 maintenance days. The GLM has 0.842, 3 milliseconds, term-based explanations, and 2 days. A random forest has 0.801, 42 milliseconds, no approved term-based explanation, and 8 days. Select a candidate.

Method. Apply each hard constraint as a pass-or-fail test before comparing preferences. The data are hypothetical; the decision uses only the stated case evidence.

GLM. Its latency passes because $3 \leq 10$, its explanation passes, and maintenance passes because $2 \leq 5$. Its deviance fails because $0.842 > 0.835$. Therefore, it is ineligible.

Lasso. Its deviance passes because $0.829 \leq 0.835$; latency passes because $6 \leq 10$; the explanation requirement passes; and maintenance passes because $4 \leq 5$. It satisfies all four requirements.

Random forest. Its deviance passes because $0.801 \leq 0.835$, but latency fails because $42 > 10$, its explanation does not meet the stated requirement, and maintenance fails because $8 > 5$. It is ineligible despite having the lowest deviance.

Recommendation and check. Select the lasso model: it is the only candidate satisfying all mandatory conditions. The result is consistent with each threshold. A common mistake is to choose the random forest because its deviance is smallest; the correction is to screen for feasibility before ranking predictive performance. This recommendation would need review if the latency limit or approved explanation standard changed.

When a Weighted Score Is Appropriate

If multiple candidates meet the hard requirements and none dominates, stakeholders may specify a weighted score. The score should use clearly defined scales and weights, and its result should be checked against the underlying evidence. An ordinal interpretability rating, for instance, is not a natural measurement scale; converting it to points is a decision convention, not a statistical discovery.

The following example uses an explicitly approved score for illustration. It assumes all three hypothetical models meet the operation’s hard requirements. The validation deviance is from the same cohort, and the baseline deviance is 1.00. The stakeholders define prediction points as the improvement over that baseline, scaled so a deviance of 0.80 earns 100 points. Interpretability is rated from 1 to 5, with 1 converted to 0 points and 5 to 100. Runtime and quarterly maintenance points are defined by the stated formulas.

P = 100\frac{1.00-D}{0.20}, \qquad I = 100\frac{R-1}{4}, \qquad T = 100\left(1-\frac{L}{100}\right), \qquad M = 100\left(1-\frac{Q}{20}\right)

Here $D$ is validation deviance, $R$ is the interpretability rating, $L$ is latency in milliseconds, and $Q$ is maintenance staff-days per quarter. The approved overall score is $0.50P+0.25I+0.15T+0.10M$. The score is intended for candidates within the stated ranges; if a value falls outside a scale, the convention must be agreed before applying it.

Worked Example: Trading Accuracy Against Explanation and Upkeep

A hypothetical claims operation compares a GLM with $D=0.92$, $R=5$, $L=5$, and $Q=2$; a random forest with $D=0.82$, $R=2$, $L=40$, and $Q=8$; and a boosting model with $D=0.80$, $R=1$, $L=60$, and $Q=12$. The approved weights and scoring scales are those stated above. Which candidate has the highest overall score?

Method. Calculate each component first, then apply the same weights to every candidate.

GLM. Its prediction points are $100(1.00-0.92)/0.20=100(0.08)/0.20=40$. Interpretability points are $100(5-1)/4=100$. Runtime points are $100(1-5/100)=100(0.95)=95$. Maintenance points are $100(1-2/20)=100(0.90)=90$. Its total is $0.50(40)+0.25(100)+0.15(95)+0.10(90)=20+25+14.25+9=68.25$.

Random forest. Prediction points are $100(1.00-0.82)/0.20=100(0.18)/0.20=90$. Interpretability points are $100(2-1)/4=25$. Runtime points are $100(1-40/100)=60$. Maintenance points are $100(1-8/20)=60$. Its total is $0.50(90)+0.25(25)+0.15(60)+0.10(60)=45+6.25+9+6=66.25$.

Boosting. Prediction points are $100(1.00-0.80)/0.20=100$. Interpretability points are $100(1-1)/4=0$. Runtime points are $100(1-60/100)=40$. Maintenance points are $100(1-12/20)=40$. Its total is $0.50(100)+0.25(0)+0.15(40)+0.10(40)=50+0+6+4=60$.

Recommendation and check. The GLM scores 68.25, the random forest 66.25, and boosting 60.00. Under this particular approved scoring rule, select the GLM. Its predictive score is lower, but its interpretability, speed, and maintenance scores compensate under the stated weights. The arithmetic is consistent with the three component totals. A common mistake is to treat the score as an objective fact; the correction is to disclose the chosen scales and weights and explain that a different stakeholder priority can change the ranking.

Translate the Requirement Into an Explanation Test

“Interpretable” is not a complete operating specification. A review team may need a short list of model terms, a sequence of decision rules, or a reason code that can be attached to each referral. These are different requirements. The case should say what a user must be able to inspect and whether the explanation must be available at scoring time.

Likewise, runtime means more than the time for one isolated prediction if the operation scores a large daily volume. Check the unit, volume, and system limit. For example, a model taking 20 milliseconds per claim would take $20 \times 30{,}000=600{,}000$ milliseconds to score 30,000 claims if scoring were strictly sequential. Since $600{,}000$ milliseconds equals 600 seconds, that is 10 minutes before accounting for data retrieval, batching, or other system work. Do not infer acceptable throughput from a per-claim number without checking the actual deployment setup.

Worked Example: Selecting a Rule-Based Triage Model

A hypothetical claims team requires a referral explanation that can be stated as a path of at most three yes-or-no tests, latency no greater than 20 milliseconds per claim, and validation MAE no greater than \$240. The GLM has MAE \$250, latency 5 milliseconds, and coefficient-based explanations but does not produce the required test path. A shallow tree has MAE \$235, latency 4 milliseconds, and paths of at most three tests. A random forest has MAE \$218, latency 45 milliseconds, and no approved short path explanation. Choose a model and quantify the relevant accuracy trade-off.

Method. Treat the explanation format, latency limit, and MAE limit as hard constraints. Compare error only among models that pass all three.

GLM. Its latency passes because $5 \leq 20$, but its MAE fails because $250 > 240$, and its coefficient explanation is not the required yes-or-no path. It is ineligible.

Shallow tree. Its MAE passes because $235 \leq 240$, its latency passes because $4 \leq 20$, and its maximum path length is exactly the permitted three tests. It is eligible.

Random forest. Its MAE passes because $218 \leq 240$, but latency fails because $45 > 20$, and the required explanation is unavailable. It is ineligible.

Recommendation and check. Select the shallow tree as the only eligible candidate. Its MAE is \$15 lower than the GLM’s, an improvement of $(250-235)/250=15/250=0.06$, or 6% relative to the GLM’s MAE. The forest’s MAE is \$17 lower than the tree’s, but it fails two mandatory requirements. A common mistake is to recommend the forest because it has the lowest MAE; the correction is to apply the explanation and runtime conditions before comparing eligible candidates. The recommendation depends on the stated three-test explanation rule and latency limit.

Common Decision Traps

Ranking before screening. A candidate that breaches a hard constraint is not a viable choice merely because it leads on prediction error.
Confusing preference with requirement. If a stakeholder says “ideally under 10 milliseconds,” clarify whether exceeding 10 milliseconds is prohibited or simply less desirable.
Using an unexplained composite score. A score without defined scales and weights can hide the very trade-off the recommendation should make explicit.
Comparing unlike evidence. Different targets, validation periods, or metrics can make a numerical ranking misleading. State the basis for comparison.
Equating a model label with interpretability. The operational test is whether the people who need an explanation can obtain and use the required information.
Ignoring ongoing work. Training and scoring are not the only operational costs; monitoring, review, and maintenance may affect whether a model remains usable.

Key takeaway: Select a model by first establishing comparable evidence, then enforcing hard operating constraints, and finally explaining the trade-offs among feasible candidates. A sound recommendation names the chosen model, the evidence supporting it, the constraints it meets, and the assumptions that could change the decision.

Check Your Understanding

For each hypothetical case, identify the feasible choice or state whether the evidence establishes dominance. Apply the stated requirements in order.

A claims team requires deviance at most 0.90, latency at most 10 milliseconds, an approved explanation, and maintenance no greater than 5 staff-days quarterly. A GLM has deviance 0.91, latency 6 milliseconds, approved explanation, and maintenance 2 days. Lasso has 0.88, 9 milliseconds, approved explanation, and 4 days. Boosting has 0.84, 30 milliseconds, no approved explanation, and 8 days. Which model is eligible? (Answer: Lasso only.)
Candidate A has deviance 0.86, interpretability rating 4, latency 8 milliseconds, and maintenance 5 days. Candidate B has 0.82, rating 2, latency 30 milliseconds, and maintenance 9 days. Candidate C has 0.88, rating 3, latency 7 milliseconds, and maintenance 4 days. With lower deviance, latency, and maintenance preferred, and higher interpretability preferred, does any candidate dominate another? (Answer: No. A is more accurate and interpretable than C, but C is faster and requires less maintenance; B has lower deviance but is worse on the other three dimensions than A.)
A service limit requires latency at most 15 milliseconds, MAE at most \$380, and a reason-code explanation. A GLM has MAE \$390, latency 4 milliseconds, and reason codes. A shallow tree has MAE \$370, latency 12 milliseconds, and reason codes. A forest has MAE \$340, latency 28 milliseconds, and no reason codes. Which candidate is eligible? (Answer: The shallow tree only.)
A stakeholder-approved score gives 60% weight to prediction points, 25% to interpretability points, and 15% to runtime points. Model X scores 70, 80, and 100 on those scales; Model Y scores 90, 40, and 60. Calculate both totals and select the higher-scoring model. (Answer: X scores $0.60(70)+0.25(80)+0.15(100)=42+20+15=77$; Y scores $0.60(90)+0.25(40)+0.15(60)=54+10+9=73$. Select X under the stated score.)
A hypothetical operation scores 24,000 claims sequentially. One candidate requires 15 milliseconds per claim and another requires 25 milliseconds per claim. A system limit allows at most 8 minutes for scoring. Ignoring other overhead as stipulated, does either candidate meet the limit? (Answer: The first requires $24{,}000(15)=360{,}000$ milliseconds $=360$ seconds $=6$ minutes and passes; the second requires $24{,}000(25)=600{,}000$ milliseconds $=600$ seconds $=10$ minutes and fails.)

What You'll Learn