Case Workshop: Communicating Auto Claim Severity Results

From Severity Metrics to a Decision

A claim-severity model predicts the cost of an individual claim. A report should do more than list a metric: it should connect the metric to the decision, explain what the comparison does and does not establish, and express the result in dollars where possible. In a hypothetical auto insurer, for example, a model used to set claim-handling reserves may be judged differently from one used to prioritize claims for review. The intended use determines which errors matter most.

This workshop uses hypothetical claim data. Assume throughout that the target is ultimate indemnity severity per claim, measured in dollars and adjusted to a common price level. The candidate predictions and actual outcomes are evaluated on the same holdout claims unless stated otherwise. These conditions matter: metrics are interpretable only when models predict the same target and are evaluated on comparable evidence.

Definition: For claim $i$, let $y_i$ be its observed severity and $\hat{y}_i$ the model's predicted severity. Define the signed prediction error as $e_i=\hat{y}_i-y_i$, so a positive error is an overprediction. For $n$ claims, the mean absolute error is $\operatorname{MAE}=\frac{1}{n}\sum_{i=1}^{n}|e_i|$, and the root mean squared error is $\operatorname{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}e_i^2}$. Both are in dollars when severity is measured in dollars; lower values indicate smaller errors under the metric being used.

MAE gives the average size of the prediction error without regard to direction. RMSE also has the units of dollars, but squaring makes it more sensitive to a few large errors. Neither metric alone tells whether a model systematically overpredicts or underpredicts: report signed errors or another suitable bias summary when direction matters. Nor does a lower overall metric prove that the model is suitable for every claim segment or operational use.

Worked Comparison on Common Holdout Claims

Worked Example: Choosing Between Two Severity Models

A hypothetical insurer compares two models on the same four holdout claims. The observed ultimate severities are $1,000, $2,000, $3,000, and $4,000. Model A predicts $1,200, $1,800, $3,300, and $3,700. Model B predicts $900, $1,900, $2,900, and $4,300. Calculate MAE and RMSE for each model and recommend one if the stated objective is to reduce typical dollar error while also considering large misses.

Method. For each model, subtract the actual severity from the prediction, take absolute values for MAE, and square the signed errors for RMSE. The common holdout set makes the comparison meaningful for these claims.

For Model A, the signed errors are $1{,}200-1{,}000=200$, $1{,}800-2{,}000=-200$, $3{,}300-3{,}000=300$, and $3{,}700-4{,}000=-300$. The absolute errors are $200, 200, 300, 300$, which sum to $1{,}000$. Thus $\operatorname{MAE}_A=1{,}000/4=\$250.00$. The squared errors are $40{,}000, 40{,}000, 90{,}000, 90{,}000$, which sum to $260{,}000$. Therefore $\operatorname{RMSE}_A=\sqrt{260{,}000/4}=\sqrt{65{,}000}=\$254.95$, rounded to the nearest cent.

For Model B, the signed errors are $900-1{,}000=-100$, $1{,}900-2{,}000=-100$, $2{,}900-3{,}000=-100$, and $4{,}300-4{,}000=300$. The absolute errors sum to $100+100+100+300=600$, giving $\operatorname{MAE}_B=600/4=\$150.00$. The squared errors sum to $10{,}000+10{,}000+10{,}000+90{,}000=120{,}000$. Hence $\operatorname{RMSE}_B=\sqrt{120{,}000/4}=\sqrt{30{,}000}=\$173.21$, rounded to the nearest cent.

Recommendation. Model B has both lower MAE and lower RMSE on these four claims, so it is the stronger candidate for the stated objectives on this holdout set. This is not evidence that it will perform better in every future period or subgroup; the comparison is limited to the evaluation data and target described.

Common mistake avoided: Do not average signed errors to calculate MAE. For Model A, signed errors sum to zero, but its absolute errors sum to $1,000. A zero signed average would conceal the size of the individual misses.

Why the Metrics Have These Interpretations

The metric formulas follow directly from the question each is designed to answer. To measure typical absolute dollar error across the evaluated claims, add each claim's error magnitude and divide by the number of claims. This gives $\operatorname{MAE}$. If a future set has $N$ claims and its average absolute error is $m$, then its total absolute error is $Nm$, because the average is the total divided by the number of claims. This is an arithmetic identity for a given set; using a holdout MAE to estimate a future total additionally assumes that the future claims have a comparable error distribution.

\operatorname{MAE}=\frac{\sum_{i=1}^{n}|e_i|}{n} \quad\Longrightarrow\quad \sum_{i=1}^{n}|e_i|=n\operatorname{MAE}

RMSE starts with squared errors so that larger misses receive more weight before averaging. Taking the square root returns the measure to the original dollar scale. For instance, an error twice as large contributes four times as much to the sum of squared errors. This explains why RMSE can rank two models differently from MAE: the ranking depends on whether the decision-maker values reducing average absolute misses or is especially concerned about unusually large misses.

Neither metric is a direct estimate of the error in the total predicted portfolio cost. Signed errors can cancel: an overprediction on one claim may offset an underprediction on another when predictions are summed. The sum of absolute errors cannot cancel, but it describes claim-by-claim misses, not necessarily the error in an aggregate reserve or payment total. State which quantity the report is discussing.

Translating Error Into Monetary Context

Worked Example: Scaling a Holdout MAE to a Portfolio Volume

In a hypothetical evaluation, Model C has a holdout MAE of $180 and Model D has a holdout MAE of $220. The team expects to score 800 claims in a future quarter. Estimate the difference in total absolute claim-level error if the holdout averages remain applicable, and state the qualification needed.

Method. Convert each per-claim MAE into an estimated sum of absolute errors by multiplying by the claim count. For Model C, $800\times \$180=\$144{,}000$. For Model D, $800\times \$220=\$176{,}000$. The difference is $\$176{,}000-\$144{,}000=\$32{,}000$. Equivalently, the MAE gap is $\$220-\$180=\$40$ per claim, and $800\times \$40=\$32{,}000$.

Interpretation. If the future claims have an error distribution comparable to the holdout claims, Model C corresponds to an estimated $32,000 lower sum of absolute claim-level errors over 800 claims. This is not a forecast of $32,000 in savings, a reduction in paid losses, or a $32,000 improvement in aggregate reserve accuracy. Those quantities require additional evidence about how predictions are used and how errors combine.

Common mistake avoided: Multiplying the MAE difference by claim volume does not prove a financial benefit. It scales a metric under a stated comparability assumption; it does not establish that operational costs or actual payments will fall by that amount.

A concise report should make the scale useful without making it sound more certain than it is. State the per-claim metric, the number of claims used for the illustrative translation, and the assumption that future claims resemble the evaluation claims. If claim mix, inflation, coverage, or claim-development practices may change, identify that as a reason the estimate may not carry forward. The intended use should also be explicit: a model for triaging claims for human review is not automatically appropriate for setting final payments.

Writing a Defensible Recommendation

Worked Example: Turning Results and Limitations Into a Report

A hypothetical insurer evaluates two candidate models on the same four claims, with observed severities of $500, $1,000, $1,500, and $2,000. Model E predicts $600, $900, $1,400, and $2,100. Model F predicts $400, $1,100, $1,200, and $2,200. The proposed use is to provide an initial severity estimate for adjusters, who retain responsibility for reviewing each claim. Calculate the metrics and draft a recommendation that avoids claiming more than the evidence supports.

Model E calculations. Its signed errors are $600-500=100$, $900-1{,}000=-100$, $1{,}400-1{,}500=-100$, and $2{,}100-2{,}000=100$. The absolute errors total $100+100+100+100=400$, so $\operatorname{MAE}_E=400/4=\$100.00$. The squared errors total $10{,}000+10{,}000+10{,}000+10{,}000=40{,}000$, so $\operatorname{RMSE}_E=\sqrt{40{,}000/4}=\$100.00$. Its signed errors sum to zero, so its mean signed error is $\$0.00$.

Model F calculations. Its signed errors are $400-500=-100$, $1{,}100-1{,}000=100$, $1{,}200-1{,}500=-300$, and $2{,}200-2{,}000=200$. The absolute errors total $100+100+300+200=700$, giving $\operatorname{MAE}_F=700/4=\$175.00$. The squared errors total $10{,}000+10{,}000+90{,}000+40{,}000=150{,}000$, giving $\operatorname{RMSE}_F=\sqrt{150{,}000/4}=\sqrt{37{,}500}=\$193.65$, rounded to the nearest cent. Its signed errors sum to $-100+100-300+200=-100$, so its mean signed error is $-100/4=-\$25.00$, indicating average underprediction on these four claims.

Recommendation. Model E is the preferred candidate for an initial estimate supporting adjuster review: on the shared evaluation set it has lower MAE and RMSE than Model F, and its signed errors average zero. The evidence is limited to four hypothetical holdout claims and does not establish performance on future claims, particular claim types, or unusually severe losses. Before operational use, evaluate it on a larger, representative set and monitor performance; adjusters should continue to review estimates rather than treating them as final payments.

Common mistake avoided: A zero mean signed error does not mean every prediction is correct. Model E misses each of the four claims by $100, even though its overpredictions and underpredictions cancel in the signed average.

What the Recommendation Must Qualify

A sound severity recommendation identifies the target, comparison basis, chosen metric, result, intended use, and most relevant limitations. Keep the conclusion proportionate to the evidence. A lower holdout MAE supports a statement about average absolute error on that evaluation set; it does not by itself establish suitability across time, subgroups, or claim types. Earlier material on reporting data and population limitations and on caveating small or sparse subgroups gives useful ways to frame these qualifications.

The comparison is especially vulnerable to misinterpretation when the models are assessed on different claims, different target definitions, or different price levels. A difference in metrics may then reflect the data or outcome definition rather than model quality. Verify that the claims, severity basis, and evaluation period are aligned before presenting a ranking. If they are not, state that the results are not directly comparable rather than choosing a winner from the displayed numbers.

Finally, match the recommendation to the decision. If large claim misses are consequential, show RMSE as well as MAE and discuss the high-severity cases. If the operational goal is a typical claim-level estimate, MAE may be more directly interpretable. Explain the metric choice in ordinary monetary terms and state what remains uncertain. This makes the report useful to decision-makers without turning a validation result into a guarantee.

Key takeaway: Recommend a severity model only in relation to a defined target, common evaluation evidence, and intended use. Translate metrics into dollars carefully, distinguish claim-level error from portfolio financial impact, and make limitations part of the recommendation rather than an afterthought. This approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each problem, use the stated severity definition and show how the result supports a suitably qualified conclusion.

A model predicts $800, $1,100, and $2,300 for three claims with actual severities of $700, $1,300, and $2,000. Calculate MAE and RMSE, rounding the latter to the nearest cent. (Answer: MAE = $200.00; RMSE = $216.02.)
A holdout MAE is $95 across claims with a comparable target definition. If 600 future claims are expected and holdout performance is assumed to remain applicable, what is the estimated sum of absolute claim-level errors? (Answer: $57,000, subject to the comparability assumption.)
On the same holdout claims, Model G has MAE $140 and RMSE $210; Model H has MAE $160 and RMSE $180. Which model has lower typical absolute error, and which has lower squared-error-sensitive error? (Answer: G has lower MAE; H has lower RMSE, so the preferred model depends on the decision's error priorities.)
Two severity models have MAEs of $130 and $115, but the first was evaluated on recent comprehensive claims and the second on older claims with a different severity basis. Can the displayed MAEs establish which model is better? (Answer: No; the evaluation evidence and target basis are not comparable, so a common evaluation is needed.)
For three claims, actual severities are $1,200, $2,000, and $2,800, while predictions are $1,000, $2,100, and $3,200. Calculate mean signed error using prediction minus actual, and MAE; interpret the direction. (Answer: Mean signed error = $100.00, indicating average overprediction; MAE = $233.33.)

What You'll Learn