Scoring Severity Predictions on Validation Data

What Does a Severity Validation Score Measure?

A severity model predicts the size of a loss given that a claim has occurred. Validation therefore compares each observed positive claim amount with a prediction made without using that claim’s outcome in model development. The score summarizes a particular prediction target: for example, the average dollar error per claim, or the average error per policy. Those targets are not interchangeable.

Before calculating a score, identify the prediction unit, the observed quantity, and the population to which the validation result should apply. The earlier material on validation targets and prediction units emphasizes this alignment. In severity validation, a row may represent an individual claim or a group of claims summarized by a common observed amount and prediction. A weight can make a grouped row count as several claims, but it should not be added merely because a policy has greater exposure.

Definition: For validation claim $i$, let $y_i>0$ be its observed severity and $\hat{\mu}_i>0$ the predicted conditional mean severity. A claim-level score averages a loss $L(y_i,\hat{\mu}_i)$ over claims. With nonnegative claim weights $w_i$, its weighted average is $$ \overline{L}_w=\frac{\sum_i w_iL(y_i,\hat{\mu}_i)}{\sum_i w_i}, $$ provided $\sum_i w_i>0$. A weight of $w_i=1$ for every individual claim gives the ordinary claim average.

Dollar-Scale Scores and Gamma Deviance

Two direct summaries are mean absolute error (MAE) and root mean squared error (RMSE). For claim weights $w_i$, they are

\operatorname{MAE}_w=\frac{\sum_i w_i|y_i-\hat{\mu}_i|}{\sum_i w_i}, \qquad \operatorname{RMSE}_w= \sqrt{\frac{\sum_i w_i(y_i-\hat{\mu}_i)^2}{\sum_i w_i}}.

Both are reported in dollars. MAE treats a dollar of error equally regardless of claim size. RMSE gives relatively greater influence to large errors because it squares them. Neither requires a particular probability distribution, but each reflects a different penalty preference.

For a Gamma severity model, a distribution-based alternative is the Gamma unit deviance. Suppose claims are conditionally independent and $Y_i$ has a Gamma distribution with mean $\mu_i$ and variance $\phi\mu_i^2$, where $\phi>0$ is a common dispersion parameter. The model requires positive outcomes and positive fitted means. The unit deviance for an observation is

d_{\Gamma}(y_i,\hat{\mu}_i) =2\left(\frac{y_i}{\hat{\mu}_i}-1-\ln\frac{y_i}{\hat{\mu}_i}\right).

A weighted mean unit deviance is $\sum_i w_i d_{\Gamma}(y_i,\hat{\mu}_i)/\sum_i w_i$. The corresponding total deviance is $\sum_i w_i d_{\Gamma}(y_i,\hat{\mu}_i)$, and the scaled deviance is the total deviance divided by $\phi$, when $\phi$ is specified. Lower values indicate better performance under that scoring rule; the deviance is zero when observed and predicted severities agree.

Derivation: Under the stated Gamma model, write the shape as $1/\phi$ and the scale as $\phi\mu_i$. Apart from terms that do not depend on $\mu_i$, the log likelihood for one observed $y_i$ is $$ -\frac{1}{\phi}\left(\ln\mu_i+\frac{y_i}{\mu_i}\right). $$ The saturated model sets $\mu_i=y_i$, achieving the best fit for that observation. Twice the log-likelihood difference between the saturated fit and the fitted mean, multiplied by $\phi$, is $$ 2\left[\ln\hat{\mu}_i+\frac{y_i}{\hat{\mu}_i}-\ln y_i-1\right] =2\left(\frac{y_i}{\hat{\mu}_i}-1-\ln\frac{y_i}{\hat{\mu}_i}\right). $$ This is the Gamma unit deviance. It is nonnegative because $r-1-\ln r\geq 0$ for $r>0$, with equality only at $r=1$. The formula does not apply to zero severities under a Gamma model; zero-claim policies are not positive claim observations.

Worked Example: Comparing Dollar Errors With Gamma Deviance

A hypothetical validation sample contains three individual claims. Each claim has weight 1. The observed severities are $100, 200,$ and $400$ dollars, and the model predicts conditional means of $100, 100,$ and $200$ dollars. Calculate MAE, RMSE, and mean Gamma unit deviance. Treat the data as positive Gamma severities; the dispersion is not needed for unit deviance.

Method and calculations. The absolute errors are $0, 100,$ and $200$ dollars, so $$ \operatorname{MAE}=\frac{0+100+200}{3}=100.00. $$ The squared errors are $0, 10{,}000,$ and $40{,}000$ dollars squared. Thus $$ \operatorname{RMSE} =\sqrt{\frac{0+10{,}000+40{,}000}{3}} =\sqrt{16{,}666.6667} =129.10\text{ dollars}. $$

For the first claim, the observed-to-predicted ratio is $100/100=1$, giving $d_{\Gamma}=2(1-1-\ln 1)=0$. For the second it is $200/100=2$, giving $$ d_{\Gamma}=2(2-1-\ln 2)=2(1-0.693147)=0.613706. $$ For the third, $400/200=2$, so its deviance is also $0.613706$. The mean is $(0+0.613706+0.613706)/3=0.409137$, or 0.4091 to four decimals.

Check: The two claims underpredicted by the same factor have equal Gamma unit deviance, as the formula depends on the ratio $y_i/\hat{\mu}_i$. Common mistake avoided: dividing by the number of claims after taking the square root in RMSE. The square root is taken after averaging squared errors.

Claim Weights and the Target Population

A weighted score is meaningful only when the weights express the intended contribution of each row. If a validation table has one row per cell and a cell represents $w_i$ claims with the same recorded severity and prediction, using $w_i$ reproduces a claim-level average. A design or sampling weight can instead represent how much of a target population a sampled claim stands for. In either case, state what the weights mean.

If each row is already an individual claim, count each claim once for an ordinary claim-level score. Policy exposure is not automatically a severity weight: severity is conditional on a claim, whereas exposure measures time or risk participation. Applying exposure again without a defined target changes the estimand.

Worked Example: Weighting Grouped Claim Records

A hypothetical validation extract has three rows, each representing a group of claims with the same observed severity and prediction. Row 1 represents 1 claim with observed severity $100$ and prediction $100$; row 2 represents 3 claims with observed severity $300$ and prediction $200$; row 3 represents 2 claims with observed severity $200$ and prediction $300$. Find the claim-weighted MAE and RMSE, then compare them with the unweighted row scores.

Method. Use claim counts $w=(1,3,2)$. The absolute errors are $0,100,100$. There are $1+3+2=6$ represented claims, so $$ \operatorname{MAE}_w =\frac{1(0)+3(100)+2(100)}{6} =\frac{500}{6} =83.33\text{ dollars}. $$

The squared errors are $0,10{,}000,10{,}000$, giving $$ \operatorname{RMSE}_w =\sqrt{\frac{1(0)+3(10{,}000)+2(10{,}000)}{6}} =\sqrt{8{,}333.3333} =91.29\text{ dollars}. $$ If the three rows are incorrectly given equal weight, the row-level MAE is $(0+100+100)/3=66.67$ dollars and the row-level RMSE is $\sqrt{(0+10{,}000+10{,}000)/3}=81.65$ dollars.

Check: The weighted result gives the row representing three claims three times the influence of a single claim. Common mistake avoided: treating a grouped row as one claim merely because it occupies one row in the extract. If the rows instead represented individual claims, the weights would not be appropriate.

Distribution Choice and Lognormal Severity

A severity score should reflect the predictive distribution when the model supplies one. For a lognormal model, suppose $\ln Y_i$ is normally distributed with location $\eta_i$ and variance $\sigma^2$. The fitted location is not the arithmetic mean severity: $$ \operatorname{Median}(Y_i)=e^{\eta_i}, \qquad E[Y_i]=e^{\eta_i+\sigma^2/2}. $$ The distinction matters if the target is expected claim severity in dollars. A squared log residual, $(\ln y_i-\eta_i)^2$, scores the location on the log scale; dividing by $\sigma^2$ gives a standardized squared log residual. It is not a dollar-scale error measure and should not be described as one.

A full lognormal predictive log score uses the density on the original dollar scale. With the same $\sigma^2$ for each observation, the negative log density for an observed positive severity is $$ \frac{1}{2}\ln(2\pi\sigma^2) +\frac{(\ln y_i-\eta_i)^2}{2\sigma^2} +\ln y_i. $$ The final $\ln y_i$ term is the change-of-variable term for scoring the density of $Y_i$, rather than the density of $\ln Y_i$. A Gamma deviance and a lognormal log score are based on different likelihoods; do not compare their raw values as if they were the same measure. When comparing candidate distributions, use a consistently defined predictive score on the same outcome scale and validation observations.

Worked Example: Scoring a Lognormal Location Prediction

A hypothetical lognormal severity model has $\eta=\ln(100)$ and $\sigma^2=0.25$ for a validation claim whose observed severity is $200$ dollars. Calculate its predicted median, predicted arithmetic mean, and standardized squared log residual. Report the squared log residual to four decimals.

Method. The median prediction is $e^\eta=e^{\ln(100)}=100$ dollars. The arithmetic mean prediction is $$ e^{\eta+\sigma^2/2} =e^{\ln(100)+0.25/2} =100e^{0.125} =113.3148\text{ dollars}. $$ The log residual is $\ln(200)-\ln(100)=\ln 2=0.693147$. Therefore its standardized squared value is $$ \frac{(\ln 2)^2}{0.25} =\frac{0.480453}{0.25} =1.9218. $$

Check: The arithmetic mean exceeds the median because the lognormal distribution is right-skewed. Common mistake avoided: reporting $100$ dollars as the mean merely because $\eta=\ln(100)$; it is the median for this model, not its arithmetic mean.

Claim-Level and Policy-Level Summaries

A claim-level average gives every claim equal influence by default. A policy-level average can instead give every policy equal influence, even when policies have different claim counts. Neither is universally preferable; the validation question determines the correct unit. For a policy with several claims, first define the policy summary being scored—for example, average severity among its claims. Then calculate a score for each policy and average those policy scores if the target is an average policy.

Changing from claim weighting to equal policy weighting can reverse a model comparison. Policies with many claims contribute more to claim-level results, while a policy-level average prevents those policies from dominating solely through claim count. Also keep the severity target distinct from total losses: a policy’s total claim amount depends on both frequency and severity and is not evaluated by a severity score alone.

Worked Example: A Model Ranking Reverses by Scoring Unit

Consider two hypothetical policies. Policy A has one claim with observed severity $1{,}000$ dollars. Policy B has nine claims, each with observed severity $100$ dollars. Model 1 predicts $500$ dollars for A and $100$ dollars for each B claim. Model 2 predicts $1{,}000$ dollars for A and $160$ dollars for each B claim. Compare MAE by claim and by policy, where each policy’s score is its average absolute claim error.

Claim-level method. Model 1 has one error of $500$ dollars and nine errors of $0$, so $$ \operatorname{MAE}_{\text{claim},1} =\frac{500+9(0)}{10}=50.00\text{ dollars}. $$ Model 2 has one error of $0$ and nine errors of $60$, so $$ \operatorname{MAE}_{\text{claim},2} =\frac{0+9(60)}{10}=54.00\text{ dollars}. $$ The claim-level score favors Model 1.

Equal-policy method. Model 1’s policy scores are $500$ dollars for A and $0$ dollars for B, giving $(500+0)/2=250.00$ dollars per policy. Model 2’s policy scores are $0$ dollars for A and $60$ dollars for B, giving $(0+60)/2=30.00$ dollars per policy. The equal-policy score favors Model 2.

Check: The arithmetic uses ten claims but only two policies, and each denominator answers a different question. Common mistake avoided: presenting one of these results as the uniquely correct average without identifying the target unit.

Practical Checks Before Reporting a Score

Confirm the outcome and unit.
Use positive claim severity for a severity model. Do not silently add zero-claim policies or substitute policy total losses.

Choose the loss measure.
Use dollar-scale MAE or RMSE when those penalties answer the question; use a distribution-based score when the predictive distribution and its assumptions are appropriate.

Define weights explicitly.
Explain whether weights count represented claims, adjust for sampling, or define another target population. Do not apply them by habit.

Report the aggregation.
State whether the result is claim-weighted, policy-weighted, or another specified summary, and give the units or score scale.

A low validation score does not establish that a model is well calibrated in every segment or stable over time; those are distinct checks. It does show performance under the chosen loss, target population, and validation sample. This distinction keeps a severity score interpretable and aligned with its intended use.

Check Your Understanding

For each problem, identify the scoring unit and apply the specified measure.

A Gamma model predicts $100$ dollars for a claim observed at $150$ dollars. Calculate the Gamma unit deviance to four decimals. (Answer: $0.1891$)
Two grouped rows represent 2 and 1 claims. Their observed severities are $80$ and $160$ dollars, and predictions are $100$ and $120$ dollars. Find claim-weighted MAE. (Answer: $\$26.67$)
Policy X has two claims with absolute errors of $20$ dollars each; Policy Y has one claim with an absolute error of $100$ dollars. Find claim-level MAE and equal-policy average MAE, where each policy score is its mean claim error. (Answer: claim-level $\$46.67$; equal-policy $\$60.00$)
A lognormal model has $\eta=\ln(50)$, $\sigma^2=0.16$, and observed severity $100$ dollars. Find its arithmetic mean prediction and standardized squared log residual, rounding dollars to cents and the residual to four decimals. (Answer: mean $\$54.16$; residual $3.0028$)
Two grouped rows represent 3 and 1 claims. Their observed severities are $100$ and $200$ dollars, and predictions are $120$ and $150$ dollars. Find claim-weighted RMSE, rounding to cents. (Answer: $\$30.41$)

What You'll Learn