Evaluate Severity Predictions with RMSE

Why Evaluate Severity Predictions with RMSE?

A severity model predicts the cost of a claim, such as the amount paid for a covered loss. Evaluation asks how closely those predictions match observed claim severities on data not used to fit the model. A single summary can help compare predictions, but the choice of summary matters: errors of different sizes do not have equal consequences under every metric.

The root mean squared error, usually abbreviated RMSE, squares each prediction error, averages the squared errors, and takes the square root. Squaring gives large errors extra influence; taking the square root returns the result to the scale of the severity outcome. RMSE is therefore expressed in dollars when severity is measured in dollars. It is a summary of prediction errors on the evaluated cases, not a guarantee about future claims or a complete measure of business value.

Definition: For $n$ evaluated claims, let $y_i$ be the observed severity for claim $i$, and let $\hat{y}_i$ be the model's predicted severity for that claim. The unweighted root mean squared error is $$ \operatorname{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}. $$ Each evaluated claim receives equal weight. The result has the same units as $y_i$. Unless the evaluation specifies otherwise, this calculation is made on the original severity scale, using the same definition of severity for actuals and predictions.

For this calculation, the $n$ observed claim severities are fixed evaluation data; no probability distribution needs to be assumed. The metric itself does not say whether a prediction is acceptable. That depends on the purpose of the model, the size of the errors, and the cost of acting on them.

Calculate RMSE and Compare with a Baseline

A model's RMSE is more informative when it is compared with a simple reference prediction. One useful baseline for severity is a constant equal to the mean severity in the training data. For every evaluation claim, the baseline predicts that same training-set mean. It is essential not to calculate this baseline from the evaluation outcomes: doing so would use information the model would not have had when making predictions.

A fair comparison uses the same evaluation claims and the same actual severity definition for the candidate model and baseline. If the candidate's RMSE is lower, its predictions have lower average squared error on those cases. That is evidence of improvement under this metric; it does not, by itself, establish that the model is better for every operational decision.

Worked Example: Compare a Severity Model with a Training-Mean Baseline

A hypothetical insurer evaluates predictions for four settled claims. Severity and predictions are recorded in thousands of dollars, with no rounding before the final result. The training data have mean severity of $10,000, so the baseline predicts 10 for each claim. The observed evaluation severities are 4, 8, 10, and 18; the candidate model predicts 5, 7, 12, and 14. Calculate both RMSE values and decide which has the lower evaluation RMSE.

Method. Calculate each model's residual as observed severity minus predicted severity, square each residual, average the four squared residuals, and take the square root. This method avoids comparing predictions evaluated on different claims or fitting the baseline to the evaluation outcomes.

Candidate model. The residuals, in thousands of dollars, are $4-5=-1$, $8-7=1$, $10-12=-2$, and $18-14=4$. Their squared values are $1$, $1$, $4$, and $16$, with total $22$. Thus the mean squared error is $22/4=5.5$, in squared thousands of dollars, and $$ \operatorname{RMSE}_{\text{candidate}}=\sqrt{5.5}=2.345207\ldots\text{ thousand dollars}. $$

Baseline. The predictions are $10,10,10,10$. The residuals are $4-10=-6$, $8-10=-2$, $10-10=0$, and $18-10=8$. Their squares are $36$, $4$, $0$, and $64$, with total $104$. The mean squared error is $104/4=26$, so $$ \operatorname{RMSE}_{\text{baseline}}=\sqrt{26}=5.099020\ldots\text{ thousand dollars}. $$

Rounding only at the final step to the nearest cent, the candidate RMSE is $2,345.21 and the baseline RMSE is $5,099.02. The candidate has the lower RMSE on these four claims. The evaluation is internally checked: both calculations use four claims, the squared-error totals are 22 and 104, and the square roots are in thousands of dollars before conversion to dollars.

Why Squaring Changes the Result

The key feature of RMSE is not just that it averages errors; it squares them first. A residual twice as large contributes four times as much squared error. More generally, multiplying a residual by $c$ multiplies its squared contribution by $c^2$. The square root is applied after averaging, so it restores the units but does not undo the strong influence of unusually large residuals.

Justification: If the residuals are $e_i=y_i-\hat{y}_i$, then each squared residual $e_i^2$ is nonnegative and has squared outcome units. The arithmetic mean $\frac{1}{n}\sum e_i^2$ is therefore a typical squared-error summary, and its nonnegative square root has the same units as $e_i$. Also, because the square-root function is increasing for nonnegative inputs, two models evaluated on the same cases have the same ordering by RMSE as by mean squared error: $$ \operatorname{MSE}_A<\operatorname{MSE}_B \quad\Longleftrightarrow\quad \sqrt{\operatorname{MSE}_A}<\sqrt{\operatorname{MSE}_B}. $$

This explains both the appeal and the limitation of RMSE. It penalizes large misses, which may be important when a severe underprediction is costly. At the same time, one exceptionally large claim can dominate the summary and make the typical error for most claims hard to see. RMSE also does not indicate the direction of error: a positive residual $y_i-\hat{y}_i$ means underprediction, while a negative residual means overprediction, but squaring removes that sign.

Worked Example: See the Effect of One Large Error

Consider two hypothetical severity models evaluated on the same four claims, with amounts in thousands of dollars. The actual severities are 2, 3, 5, and 10. Model A predicts 2, 3, 5, and 20. Model B predicts 4, 1, 7, and 8. Calculate both RMSE values and compare them with their mean absolute errors (MAEs), where MAE is the average absolute residual. The MAE is included only to show how the choice of summary changes the emphasis.

Method. For each model, calculate residuals against the four actuals, then compute squared and absolute residuals. This direct calculation prevents a large residual from being overlooked when summarizing performance.

For Model A, the residuals $y_i-\hat{y}_i$ are $0,0,0,-10$. The squared residuals are $0,0,0,100$, so the mean squared error is $100/4=25$ and RMSE is $\sqrt{25}=5$ thousand dollars. Its absolute residuals total 10, giving MAE $10/4=2.5$ thousand dollars.

For Model B, the residuals are $2-4=-2$, $3-1=2$, $5-7=-2$, and $10-8=2$. Their squares are $4,4,4,4$, giving mean squared error $16/4=4$ and RMSE $\sqrt{4}=2$ thousand dollars. The absolute residuals total 8, so MAE is $8/4=2$ thousand dollars.

Model B has the lower RMSE, $2,000 versus $5,000, and the lower MAE, $2,000 versus $2,500. Model A's single $10,000 underprediction contributes all of its squared error. Rounded to the nearest cent, its RMSE is $5,000.00 and Model B's is $2,000.00. The calculation is checked by noting that Model B has four errors of magnitude $2,000, so its RMSE must equal $2,000. This example shows why a low error on most claims does not prevent one large miss from producing a high RMSE.

Keep the Outcome Scale and Decision in View

An RMSE is meaningful only relative to its scale and evaluation setup. An RMSE of $3,000 could be small for a portfolio whose typical claim is $100,000 and large for one whose typical claim is $4,000. It is not generally appropriate to compare raw RMSE values across different severity definitions, currencies, or portfolios without considering those differences. A ratio such as RMSE divided by mean severity can give context, but the ratio is not a replacement for understanding the loss amounts or the decision at hand.

The predicted and observed values must also refer to the same target. For example, if the target is paid severity, the actuals should not instead be incurred severity including reserves. If a model is fitted using a transformed outcome, its predictions must be returned to the original severity scale before calculating dollar-scale RMSE. A prediction that remains on a log scale cannot be subtracted from an actual amount in dollars.

RMSE does not encode asymmetric consequences. If underestimating a large claim is more costly than overestimating it by the same amount, ordinary RMSE gives the two errors identical squared penalties. Nor does RMSE measure ranking quality, explain why predictions are wrong, or establish whether a proposed business action is worthwhile. The right interpretation is narrow: it summarizes the magnitude of squared prediction errors, with larger misses receiving disproportionate weight.

Worked Example: Interpret RMSE Against the Severity Scale

A hypothetical model is tested on four claims, with severity measured in thousands of dollars. The actual values are 10, 20, 30, and 40; the model predicts 12, 18, 27, and 45. A baseline uses a training-set mean of 25, predicting 25 for every evaluation claim. Calculate both RMSE values, express the candidate RMSE as a percentage of the mean observed severity, and interpret the comparison without treating that percentage as a decision rule.

Method. Calculate each RMSE on the same four observed claims. Then calculate the evaluation mean severity solely as context for interpreting the size of the candidate RMSE; it is not used to fit the baseline.

The candidate residuals are $10-12=-2$, $20-18=2$, $30-27=3$, and $40-45=-5$. The squared residuals are $4,4,9,25$, which sum to 42. The mean squared error is $42/4=10.5$, so candidate RMSE is $\sqrt{10.5}=3.240370\ldots$ thousand dollars.

The baseline residuals are $10-25=-15$, $20-25=-5$, $30-25=5$, and $40-25=15$. The squared residuals are $225,25,25,225$, totalling 500. The mean squared error is $500/4=125$, and the baseline RMSE is $\sqrt{125}=11.180340\ldots$ thousand dollars.

The observed mean is $(10+20+30+40)/4=25$ thousand dollars. The candidate RMSE as a percentage of that mean is $3.240370\ldots/25\times100=12.96148\ldots\%$. Rounded only at the final step to the nearest cent for dollar RMSEs and two decimals for the percentage, the candidate RMSE is $3,240.37, the baseline RMSE is $11,180.34, and the ratio is 12.96%. The candidate improves on the baseline in this evaluation. Whether a $3,240.37 RMSE is acceptable still depends on the operational use and the cost of the errors; 12.96% is context, not a universal pass threshold. The totals and denominators are checked against the four claims in each calculation.

Common Exam Traps

Taking the square root too early. Calculate all squared residuals, average them, and then take one square root. Averaging residual magnitudes or taking separate roots does not give RMSE.
Using absolute errors instead of squared errors. That produces MAE, not RMSE. The two metrics can rank models differently, especially when one model has a large miss.
Using the evaluation-set mean as the baseline prediction. The baseline mean must come from training data if it represents a prediction made without access to evaluation outcomes.
Comparing different cases or different targets. A lower RMSE is a fair model comparison only when the outcomes and evaluated cases are aligned.
Interpreting the result as a typical signed error. RMSE is nonnegative and carries no direction. Inspect the residual signs when underprediction versus overprediction matters.

Key takeaway: Calculate RMSE as the square root of the mean squared residual on the original severity scale. Compare it with a training-based baseline on the same claims, then ask whether large errors and the outcome scale make the summary suitable for the decision.

Check Your Understanding

For each problem, use the stated evaluation cases and round monetary RMSE values to the nearest cent only at the end.

A hypothetical severity model has actual amounts of $6, $9, and $15, and predicts $5, $11, and $12. Calculate RMSE. (Answer: $2.16)
A training-set mean severity is $10,000. On four evaluation claims with actual severities of $4,000, $8,000, $12,000, and $16,000, a baseline predicts $10,000 each time. Calculate its RMSE. (Answer: $4,472.14)
On the same three hypothetical claims, Model A has residuals of $1,000, $-$1,000$, and $4,000; Model B has residuals of $-$2,000$, $0, and $2,000. Which model has lower RMSE, and what is that RMSE? (Answer: Model B, $1,632.99)
Actual severities are $3,000, $5,000, and $12,000. Model A predicts $3,000, $5,000, and $18,000; Model B predicts $5,000, $3,000, and $12,000. Calculate each RMSE and identify which model is preferred by RMSE. (Answer: Model A $3,464.10; Model B $1,632.99; Model B has lower RMSE)
A candidate model has RMSE $4,000 on a portfolio with mean observed severity $20,000. What is RMSE as a percentage of mean observed severity, and does that percentage alone establish acceptability? (Answer: 20.00%; no, acceptability depends on the decision and error consequences)

What You'll Learn