Separate Ranking Gains from Better Claim Estimates

Two Different Questions About Claim Predictions

A claim model can order claims effectively from lower to higher expected cost yet predict the dollar amounts poorly. It can also produce plausible total costs while ordering claims imperfectly. These outcomes are not contradictory: ranking asks whether higher-risk claims tend to receive higher scores, while dollar estimation asks how close predicted amounts are to observed amounts.

The distinction matters when a hypothetical insurer uses a model for two different purposes. A claims team with limited review capacity may need to identify which claims are most likely to exceed a high-cost threshold. A pricing or reserving analysis may instead need credible estimates of claim dollars. The first decision depends mainly on ordering; the second depends on the numerical errors. A single metric cannot answer both questions.

Definition: Ranking performance measures how well a model orders observations by a target of interest, such as whether a claim will exceed a specified cost threshold. Dollar-estimation accuracy measures the distance between predicted and observed claim amounts on the target’s dollar scale. A model’s performance on one objective does not determine its performance on the other.

Before comparing results, specify the prediction target. For ranking, define what counts as a high-cost claim, such as an observed cost $Y$ of at least $8,000. For dollar estimation, state whether $Y$ is the observed claim amount, an ultimate amount, or another defined cost measure. The comparison is meaningful only when the candidate models are assessed on the same target and validation observations.

Evidence for Ranking and Dollar Estimation

For a binary high-cost target, area under the receiver operating characteristic curve (AUC) is a common ranking measure. It evaluates the model’s scores across all possible score cutoffs, rather than at just one cutoff. AUC does not measure whether a score is itself a dollar estimate or a calibrated probability.

Definition: For a set of high-cost claims and non-high-cost claims, empirical AUC is the fraction of all positive-negative pairs for which the high-cost claim receives the larger score. A tied pair contributes one-half. AUC of 1 means every such pair is ordered correctly; AUC of 0.5 corresponds to chance-level pair ordering in this pairwise sense.

Dollar predictions are commonly assessed using root mean squared error (RMSE). If $y_j$ is the observed cost and $\hat{y}_j$ is the prediction for validation claim $j$, for $n$ claims, then RMSE is measured in the same units as the cost. Squaring the errors gives larger errors more influence, so a few large misses can materially affect the result.

\operatorname{RMSE} = \sqrt{\frac{1}{n}\sum_{j=1}^{n}(y_j-\hat{y}_j)^2}

The justification for RMSE follows directly from its construction: each claim contributes its squared dollar error, the average summarizes squared error per claim, and the square root returns the result to dollars. In contrast, AUC depends only on the ordering of scores for positive-negative pairs. A strictly increasing transformation of all scores preserves their order and therefore preserves AUC, but it can change how close the scores are to dollar amounts.

State the decision.
Decide whether the task is to find high-cost claims for review, estimate claim dollars, or support both purposes.

Match the metric to the task.
Use a ranking measure such as AUC for high-cost ordering; use RMSE, MAE, or other dollar-scale evidence for amount prediction.

Read the results separately.
Do not infer dollar accuracy from a strong AUC, or good ranking from a small RMSE.

Make a purpose-specific conclusion.
State which model is better supported for the specified use and what remains unproven.

Worked Examples

Worked Example: Better Ranking but Worse Dollar Predictions

A hypothetical insurer validates two models on six claims. The high-cost label is defined as observed cost of at least $8,000. Model R produces ranking scores, and Model D produces dollar predictions. Compare their AUCs for the high-cost label and their dollar RMSEs. State which model has better evidence for each purpose.

The observed costs are $1,000, $2,000, $3,000, $8,000, $9,000, and $10,000. Thus, the final three claims are high-cost and the first three are non-high-cost. Model R’s scores are 100, 200, 300, 400, 500, and 600. They increase in the same order as the observed costs, so every one of the $3 \times 3=9$ positive-negative pairs is correctly ordered. Its AUC is $9/9=1.0000$.

Model D’s dollar predictions, in the same claim order, are $1,000, $2,000, $8,500, $7,000, $9,000, and $10,000. The predicted high-cost claims receive scores of $7,000, $9,000, and $10,000; the non-high-cost claims receive $1,000, $2,000, and $8,500. The $7,000 score is below the $8,500 score for a non-high-cost claim, producing one incorrectly ordered pair. The other eight pairs are correctly ordered, so Model D’s AUC is $8/9=0.8889$.

Now calculate the dollar RMSE for Model D. Its errors, defined as observed cost minus predicted cost, are $0, $0, −$5,500, $1,000, $0, and $0. Therefore:

\operatorname{RMSE}_D = \sqrt{\frac{0^2+0^2+(-5{,}500)^2+1{,}000^2+0^2+0^2}{6}} = \sqrt{\frac{31{,}250{,}000}{6}} = \$2{,}282.18

Model R’s scores are not dollar predictions. If they were provisionally treated as dollar predictions solely to compute a diagnostic, the errors would be $900, $1,800, $2,700, $7,600, $8,500, and $9,400. The sum of their squared errors is $229,710,000, giving:

\operatorname{RMSE}_R = \sqrt{\frac{229{,}710{,}000}{6}} = \$6{,}187.49

Thus, Model R ranks high-cost claims better, while Model D has lower dollar RMSE. The arithmetic is rounded to the nearest cent only in the final RMSE values. The comparison supports Model R for ordering high-cost claims and Model D for predicting dollars, subject to the validation evidence and use being appropriate. AUC alone would not establish Model R’s dollar accuracy.

Method and mistake avoided. Count positive-negative pair orderings for AUC, and calculate squared dollar errors for RMSE. A common mistake is to declare Model R the better dollar model because its AUC is higher. AUC uses ordering, not dollar distance.

Worked Example: AUC and a Fixed Review Capacity

A hypothetical claim team can review two claims. Five observed costs, in thousands of dollars, are 2, 4, 6, 12, and 20. Define high cost as at least $10,000. Model A assigns scores 0.1, 0.2, 0.3, 0.8, and 0.9; Model B assigns scores 0.1, 0.9, 0.2, 0.8, and 0.7. Higher scores indicate higher priority. Compare AUC and identify the two claims each model would send for review.

The high-cost claims have observed costs of $12,000 and $20,000; the other three are non-high-cost. For Model A, the two high-cost scores, 0.8 and 0.9, each exceed all three non-high-cost scores, 0.1, 0.2, and 0.3. There are $2 \times 3=6$ positive-negative pairs, all correctly ordered, so AUC is $6/6=1.0000$. Its two highest scores belong to the $12,000 and $20,000 claims, so both high-cost claims are selected.

For Model B, the high-cost scores are 0.8 and 0.7, while the non-high-cost scores are 0.1, 0.9, and 0.2. Score 0.8 exceeds two of the three non-high-cost scores, and score 0.7 also exceeds two. There are four correctly ordered pairs out of six, so AUC is $4/6=0.6667$. The two highest scores are 0.9 for the $4,000 claim and 0.8 for the $12,000 claim. Only one of the two selected claims is high-cost.

Model A is better supported for this ranking task, and it fills the two available review slots with both high-cost claims in this validation sample. Model B’s score of 0.9 should not be read as a 90% probability or a dollar amount: the problem supplies scores only for ranking. This example is checked against the stated review capacity of two and the stated high-cost threshold of $10,000.

Method and mistake avoided. AUC evaluates pairwise ordering across all score cutoffs; the two-claim review result evaluates the chosen capacity. Do not treat AUC as the fraction of claims correctly classified at a particular cutoff. AUC of 0.6667 does not mean that two-thirds of these five claims were classified correctly.

Worked Example: Aggregate Cost Agreement Does Not Prove Claim-Level Accuracy

A hypothetical portfolio has four observed claim costs, in thousands of dollars: 1, 3, 7, and 9. Compare Model C’s predictions of 0.5, 1.5, 3.5, and 4.5 with Model D’s predictions of 3, 1, 9, and 7. Calculate total predicted cost and RMSE for each. Also compare their ranking of claims at least $7,000.

The observed total is $1+3+7+9=20$ thousand dollars. Model C’s predicted total is $0.5+1.5+3.5+4.5=10$ thousand dollars. Its errors are 0.5, 1.5, 3.5, and 4.5 thousand dollars, so:

\operatorname{RMSE}_C = \sqrt{\frac{0.5^2+1.5^2+3.5^2+4.5^2}{4}} = \sqrt{\frac{35}{4}} = 2.9580\text{ thousand dollars}

Model D’s predicted total is $3+1+9+7=20$ thousand dollars, exactly matching the observed total. Its errors are −2, 2, −2, and 2 thousand dollars. Consequently:

\operatorname{RMSE}_D = \sqrt{\frac{(-2)^2+2^2+(-2)^2+2^2}{4}} = \sqrt{4} = 2\text{ thousand dollars}

For the high-cost ranking, the two claims at or above $7,000 are the third and fourth claims. Model C’s scores increase with the observed costs, so it ranks those two claims above both lower-cost claims. Model D gives the four claims scores of 3, 1, 9, and 7, so it also ranks the two high-cost claims above the lower-cost claims. Both therefore have perfect pairwise ordering for this particular high-cost definition, even though Model D reverses the order within each pair of claims.

The models do not have equal dollar evidence: Model D has lower RMSE, while its exact portfolio total alone would not establish that its individual predictions are accurate. Here the individual errors confirm that RMSE is 2 thousand dollars, rather than zero. The calculations use thousands throughout; RMSE is reported in thousands of dollars.

Method and mistake avoided. Compare aggregate totals, claim-level errors, and high-cost ordering as separate quantities. A common mistake is to treat a correct aggregate total as proof of accurate individual estimates. Overprediction on some claims can cancel underprediction on others.

Interpreting Conflicting Evidence

When ranking and dollar metrics disagree, first ask which intended decision each metric represents. A model with stronger AUC may be more useful for a review queue, but that does not establish it as the better cost estimator. A model with lower RMSE may predict dollars more closely on average, but it may still fail to place enough high-cost claims near the top of a limited-capacity review list.

The evaluation should state the target, the validation population, and the metric’s meaning. For ranking, specify the high-cost threshold and whether the operational decision uses a fixed capacity or a score cutoff. For dollar estimation, report a dollar-scale measure such as RMSE and explain that large errors receive greater weight. The mean error or aggregate prediction may add context, but neither replaces examination of individual prediction errors.

If both uses matter, report both kinds of evidence rather than selecting one metric as a universal winner. A decision statement might say that one model better orders claims above the specified threshold, while another has lower validation RMSE for dollar amounts. It should not imply that either model is superior for every purpose.

Key takeaway: Ranking evidence answers, “Which claims tend to be higher?” Dollar-accuracy evidence answers, “How close are the predicted amounts?” Define the target and intended action first, then use metrics that directly assess each question.

Check Your Understanding

For each problem, distinguish the ranking conclusion from any conclusion about dollar accuracy.

A validation set has two high-cost claims with scores 0.7 and 0.9, and two non-high-cost claims with scores 0.2 and 0.8. Calculate AUC. (Answer: 3 of 4 pairs are correctly ordered; AUC = 0.7500.)
Three observed claim costs are $1,000, $3,000, and $5,000. Predictions are $2,000, $3,000, and $4,000. Calculate RMSE to the nearest cent. (Answer: $\sqrt{2{,}000{,}000/3}=\$816.50$.)
Observed costs, in thousands, are 2, 8, and 10; ranking scores are 0.2, 0.9, and 0.7. Define high cost as at least $8,000. What is AUC, and which claim would be selected with capacity one? (Answer: AUC = 1.0000; the $8,000 claim is selected because its score, 0.9, is highest.)
A model’s observed costs are $4,000 and $6,000, and its predictions are $6,000 and $4,000. What are the observed and predicted totals, and the RMSE? Does matching total establish correct individual estimates? (Answer: Both totals are $10,000; RMSE = $2,000; no, the individual predictions are each wrong by $2,000.)
Two models have AUCs of 0.91 and 0.84, but the first has RMSE of $5,000 and the second has RMSE of $3,200. Which has stronger ranking evidence, and which has lower dollar error on this validation set? (Answer: The first has stronger ranking evidence; the second has lower dollar RMSE. Neither result alone establishes overall superiority for both uses.)

What You'll Learn