Choosing Metrics for Frequency and Severity Predictions

Why Metric Choice Depends on the Prediction

A claim-frequency prediction describes an expected number of claims over a defined exposure period. A claim-severity prediction describes the expected amount per claim, usually conditional on a claim occurring. These predictions have different outcome scales and different data structures: frequency data include zero-claim observations, while a severity data set generally contains only positive claim amounts. A metric that works well for one target may be unsuitable for the other.

The choice also depends on the decision. A team estimating expected claim counts may care about errors measured in claims per policy-period. A team forecasting dollar costs may care about errors in dollars, or about relative errors across small and large claims. There is no universally best metric: the metric should be selected to reflect both the prediction target and the consequences of error.

Definition: For observed outcomes $y_i$ and predictions $\hat{y}_i$, an evaluation metric summarizes prediction errors over a specified set of observations. For frequency, $y_i$ is a claim count and $\hat{y}_i$ is a predicted expected count for the stated exposure. For severity, $y_i$ is a positive claim amount and $\hat{y}_i$ is a predicted claim amount. Always state the unit of observation and the target being scored.

If policies have different exposure periods, raw counts are not directly comparable without accounting for exposure: a count prediction should correspond to the exposure for that observation. Likewise, a severity metric should be calculated on the observations for which severity is defined. Scoring claims only cannot, by itself, evaluate how well a model predicts which policies will have no claims.

Metrics for Frequency Predictions

For observed counts $y_1,\ldots,y_n$ and predicted counts $\hat{y}_1,\ldots,\hat{y}_n$, mean absolute error (MAE) and root mean squared error (RMSE) are both defined and interpretable when observed counts are zero. MAE reports the average absolute error in claims per observation. RMSE is also in claims per observation, but gives greater influence to large errors because it squares them before averaging.

\operatorname{MAE}=\frac{1}{n}\sum_{i=1}^{n}|y_i-\hat{y}_i|, \qquad \operatorname{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}.

A count-oriented alternative is Poisson deviance. For an observed count $y_i$ and a positive predicted mean $\hat{\mu}_i$, its contribution is $2[y_i\log(y_i/\hat{\mu}_i)-(y_i-\hat{\mu}_i)]$. The convention is that the term $y_i\log(y_i/\hat{\mu}_i)$ equals zero when $y_i=0$. The mean Poisson deviance is the average contribution over observations. It is a likelihood-based measure for a Poisson count model, whose mean and variance are both $\hat{\mu}_i$; if that distributional assumption is not suitable, the deviance may still be calculated as a score, but its Poisson likelihood interpretation does not apply. It is not measured in claims or dollars, so it should not be read as an average claim-count error.

D_{\mathrm{Pois}}=\frac{2}{n}\sum_{i=1}^{n} \left[y_i\log\left(\frac{y_i}{\hat{\mu}_i}\right)-(y_i-\hat{\mu}_i)\right].

The zero-count case is especially useful to check. When $y_i=0$, the contribution simplifies to $2\hat{\mu}_i$: the score remains finite as long as the predicted mean is finite. A prediction of exactly zero, however, is not permitted in the displayed formula for a positive observed count, because it would require division by zero and an undefined logarithm.

Worked Example: Comparing Frequency Predictions With Zeros

For four hypothetical policy-periods with equal exposure, the observed claim counts are $0,1,3,4$. Model A predicts $0.5,0.5,2.5,3.5$ claims, while Model B predicts $0.1,0.1,2.5,5.5$. Calculate MAE and RMSE for each model and compare them.

Method. Calculate each absolute error for MAE and each squared error for RMSE, then average across the four policy-periods. The predictions are expected counts, so fractional values are valid.

For Model A, the absolute errors are $0.5,0.5,0.5,0.5$. Their sum is $2.0$, giving $\operatorname{MAE}=2.0/4=0.5$ claims per policy-period. The squared errors are $0.25,0.25,0.25,0.25$, with sum $1.00$. Thus $\operatorname{RMSE}=\sqrt{1.00/4}=\sqrt{0.25}=0.5$ claims per policy-period.

For Model B, the absolute errors are $0.1,0.9,0.5,1.5$. Their sum is $3.0$, giving $\operatorname{MAE}=3.0/4=0.75$ claims per policy-period. The squared errors are $0.01,0.81,0.25,2.25$, with sum $3.32$. Therefore $\operatorname{RMSE}=\sqrt{3.32/4}=\sqrt{0.83}\approx0.9110$ claims per policy-period, rounded to four decimals.

Both metrics favor Model A on these observations. The zero-count policies have been retained in both calculations; MAE and RMSE do not require division by the observed count. The comparison is only meaningful for the stated equal-exposure policy-periods and this particular evaluation sample.

Metrics for Severity Predictions

Claim amounts are commonly right-skewed: many claims are relatively small, while a few are much larger. MAE and RMSE answer different questions in this setting. MAE gives a dollar-valued summary that is less affected by an extreme error than RMSE. RMSE also has dollar units, but its squared-error calculation can make a small number of large claims dominate the result. If large dollar errors are especially consequential, that emphasis may be appropriate; if the aim is typical absolute error, MAE may be easier to interpret.

A log-scale measure can be useful when relative differences matter more than dollar differences. For nonnegative outcomes, root mean squared logarithmic error (RMSLE) compares the logarithms of one plus the observed and predicted amounts:

\operatorname{RMSLE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n} \left[\log(1+\hat{y}_i)-\log(1+y_i)\right]^2}.

Here $\log$ denotes the natural logarithm, and both outcomes and predictions must be nonnegative. The added one allows zero values: $\log(1+0)=0$. This addition is tied to the chosen unit, however. Expressing amounts in dollars rather than thousands of dollars changes the numerical values inside the logarithms and can change RMSLE. State the unit used and keep it consistent when comparing models.

Why the RMSLE formula reflects shifted ratios: Subtracting the two logarithms gives $\log(1+\hat{y})-\log(1+y)=\log((1+\hat{y})/(1+y))$. Thus each squared term measures the squared log ratio of the shifted prediction to the shifted observation. A ratio of $2$ and a ratio of $1/2$ have equal squared log errors because $\log(1/2)=-\log(2)$. Averaging these squared differences and taking the square root gives the RMSLE formula. This is a relative comparison of shifted amounts, not an ordinary percentage error.

Worked Example: Comparing Dollar and Log-Scale Severity Errors

A hypothetical severity model predicts the following amounts for three claims. Observed severities in dollars are $9,99,999$, and predicted severities are $19,49,499$. Calculate MAE, RMSE, and RMSLE using dollars for the log-scale calculation.

Method. First calculate prediction minus observation. For MAE and RMSE, retain the dollar differences. For RMSLE, use $\log(1+\text{prediction})-\log(1+\text{observation})$. This distinction avoids confusing ordinary amount ratios with the shifted ratios used by RMSLE.

The signed dollar errors are $19-9=10$, $49-99=-50$, and $499-999=-500$. The absolute errors are $10,50,500$, so their sum is $560$ and \[ \operatorname{MAE}=\frac{560}{3}=\$186.67, \] rounded to the nearest cent.

The squared dollar errors are $10^2=100$, $(-50)^2=2{,}500$, and $(-500)^2=250{,}000$. Their sum is $252{,}600$, so the mean squared error is $252{,}600/3=84{,}200$ dollars squared. Therefore \[ \operatorname{RMSE}=\sqrt{84{,}200}\approx \$290.17, \] rounded to the nearest cent.

For RMSLE, the prediction-minus-observation log differences are \[ \log(20)-\log(10)=\log 2,\qquad \log(50)-\log(100)=-\log 2,\qquad \log(500)-\log(1{,}000)=-\log 2. \] The shifted ratios are $2,1/2,1/2$, respectively. Each squared log difference is therefore $(\log 2)^2$, and \[ \operatorname{RMSLE} =\sqrt{\frac{3(\log 2)^2}{3}} =\log 2 \approx 0.6931, \] rounded to four decimals. The dollar errors differ substantially, while the shifted ratios have the same log-error magnitude. The calculation illustrates why RMSLE and dollar-based metrics describe different aspects of predictive accuracy.

Choose the Metric for the Intended Decision

Frequency and severity predictions are sometimes combined to estimate expected claim cost. For a policy-period, if $\hat{f}$ is the predicted expected claim count and $\hat{s}$ is the predicted mean severity per claim, the predicted expected aggregate cost is $\hat{f}\hat{s}$. This product is a distinct prediction target. A good frequency metric alone does not guarantee good aggregate-cost predictions, just as a good severity metric alone does not.

Worked Example: Frequency Accuracy Versus Expected Cost Accuracy

Consider three hypothetical policy-periods with equal exposure. Their observed claim counts are $0,1,2$. The single claim in the second period costs $\$120$; the two claims in the third period cost $\$300$ and $\$500$. Thus observed aggregate costs are $\$0,\$120,\$800$. Model A predicts frequencies $0.2,0.8,1.5$ and conditional mean severities $\$200,\$150,\$400$. Model B predicts frequencies $0.05,1.1,1.8$ and conditional mean severities $\$200,\$500,\$600$. Compare frequency MAE and aggregate-cost MAE.

Frequency comparison. Model A's absolute frequency errors are $|0.2-0|=0.2$, $|0.8-1|=0.2$, and $|1.5-2|=0.5$. Its frequency MAE is $(0.2+0.2+0.5)/3=0.30$ claims per policy-period.

Model B's absolute frequency errors are $|0.05-0|=0.05$, $|1.1-1|=0.1$, and $|1.8-2|=0.2$. Its frequency MAE is $(0.05+0.1+0.2)/3=0.1167$, rounded to four decimals. Model B is better by frequency MAE.

Aggregate-cost comparison. Multiply each predicted frequency by its predicted conditional mean severity. Model A predicts costs $0.2(\$200)=\$40$, $0.8(\$150)=\$120$, and $1.5(\$400)=\$600$. Compared with observed costs $\$0,\$120,\$800$, the absolute cost errors are $\$40,\$0,\$200$. Its cost MAE is $\$240/3=\$80$.

Model B predicts costs $0.05(\$200)=\$10$, $1.1(\$500)=\$550$, and $1.8(\$600)=\$1{,}080$. The absolute cost errors are $\$10,\$430,\$280$, which sum to $\$720$. Its cost MAE is $\$720/3=\$240$. Model A is better by aggregate-cost MAE, despite Model B's better frequency MAE.

The comparison shows why the scored target should match the decision. If the task is to forecast claim counts, frequency errors are directly relevant. If the task is to estimate expected costs, assess predicted costs against observed aggregate costs. The severity prediction for a period with no observed claims is still a conditional mean severity prediction; it is not an observed claim amount to include in a claim-level severity error calculation.

Common Mistakes When Selecting a Metric

A frequent error is to apply mean absolute percentage error (MAPE) to frequency data that include zero counts. For an observation with $y_i=0$ and $\hat{y}_i=0.2$, the percentage error term $|y_i-\hat{y}_i|/|y_i|$ requires division by zero. Calling the error $0\%$ is incorrect, and silently dropping the observation changes the evaluation set and can make the summary misleading.

For example, the wrong calculation might mark the zero-count policy as a zero-percent error or remove it, then average percentage errors only for positive counts. Neither approach evaluates the model on all policy-periods as intended. The correction is to choose a metric that is defined at zero, such as MAE or RMSE for count errors, or Poisson deviance when its count-model interpretation is appropriate.

Do not confuse targets. Claim frequency, claim severity, and aggregate cost are different outcomes and need not rank models in the same order.
Do not assume a log metric is unit-free. RMSLE with a one-unit shift can change when dollars are converted to thousands of dollars.
Do not overlook large errors. RMSE gives extreme dollar or count errors more influence than MAE; decide whether that emphasis matches the use.
Do not score only positive claims when the decision concerns all policies. A severity-only score does not assess the prediction of whether a claim occurs.

Key takeaway: Name the prediction target, observation unit, and outcome scale before selecting a metric. For counts with zeros, MAE, RMSE, and (with appropriate interpretation) Poisson deviance are available. For positive, skewed severities, compare dollar-based metrics with a log-scale metric only when its shifted, unit-dependent interpretation fits the decision.

Check Your Understanding

For each problem, choose or calculate the metric requested and check that the outcome and prediction units are consistent.

Observed claim counts are $0,1,3$, and predicted counts are $0.2,0.5,2$. Calculate frequency MAE. (Answer: $0.5667$ claims per observation.)
For two policy-periods, observed counts are $0,2$ and predicted counts are $1,1$. Calculate MAE and RMSE, then state whether they agree. (Answer: MAE $=1$ claim per observation; RMSE $=1$ claim per observation; they agree.)
For two hypothetical claims, observed amounts are $\$50,\$150$, and predicted amounts are $\$100,\$100$. Calculate MAE and RMSE in dollars. (Answer: MAE $=\$50$; RMSE $=\$50$.)
Observed nonnegative severities are $0,9$, and predictions are $1,4$. Calculate RMSLE using natural logarithms and a consistent unit. (Answer: $\log 2\approx0.6931$.)
For two count observations, the observed counts are $0,2$, and the predicted Poisson means are $0.25,1$. Calculate mean Poisson deviance. (Answer: $\{0.5+(4\log 2-2)\}/2\approx0.6363$.)

What You'll Learn