Diagnosing Conditional Severity Fits

Why Diagnose Conditional Severity Fits?

A severity model assigns an expected claim amount to each claim, conditional on the modeled characteristics. A model can have plausible coefficients and still fit poorly in important parts of the claim population. For example, it may systematically underpredict claims with large fitted means, or its fitted distribution may produce too few claims in the upper tail. Aggregate fit statistics can conceal these patterns.

Diagnosis compares observed responses with predictions in ways that preserve the model’s target and sampling assumptions. The first question is not simply whether observed claims are larger than predicted claims. It is whether discrepancies show a systematic pattern across fitted means, claim sizes, or coverage-limit groups, rather than ordinary variation around a reasonable conditional model.

Definition: For claim $i$, let $Y_i$ be the observed severity response and $\mu_i$ the model’s fitted conditional mean, $E[Y_i\mid \mathbf{x}_i]=\mu_i$. An observed-to-predicted ratio for a group $G$ is $\sum_{i\in G}Y_i/\sum_{i\in G}\mu_i$. A ratio above 1 means the group’s total observed severity exceeds its total predicted severity; a ratio below 1 means it falls short.

This comparison is meaningful only when observed and predicted amounts refer to the same response, claim population, and units. If the modeled response is paid severity, predictions for ground-up severity are not an appropriate denominator. Similarly, observations affected by selection or a policy limit need to be handled consistently with the model’s target and data assumptions. The diagnostic itself does not repair a mismatch in the data definition.

Group Calibration and Residual Patterns

A useful first diagnostic is to divide claims into groups using their fitted means, such as low, middle, and high ranges. Compare the total observed severity with the total predicted severity in each group. Grouping by fitted mean can reveal conditional calibration patterns that disappear in an overall total. Groups should generally be defined from predictors or fitted values, not chosen after inspecting which claims had the largest residuals.

For an individual claim, the raw residual is $Y_i-\mu_i$. A positive residual indicates underprediction for that claim; a negative residual indicates overprediction. Raw residuals are not directly comparable across fitted means when the response variance changes with the mean. For a Gamma severity model with conditional variance $\operatorname{Var}(Y_i\mid\mathbf{x}_i)=\phi\mu_i^2$, the Pearson residual is

r_i^{P}=\frac{Y_i-\mu_i}{\sqrt{\phi}\,\mu_i}.

Here $\phi>0$ is the dispersion parameter. This standardizes the discrepancy by the model’s conditional standard deviation. It does not make every residual exactly normally distributed, nor does it make a few large residuals proof of model failure. It provides a scale-aware way to look for patterns and unusually large discrepancies.

For a Gamma model, one can also inspect signed deviance residuals:

r_i^{D} =\operatorname{sign}(Y_i-\mu_i) \sqrt{\frac{2}{\phi}\left(\frac{Y_i}{\mu_i}-1-\ln\left(\frac{Y_i}{\mu_i}\right)\right)}.

These use the Gamma model’s deviance contribution and retain the direction of the error. Pearson and deviance residuals are model-dependent tools: use the variance function and deviance appropriate to the fitted family. A residual formula should not be transferred unchanged to a different severity distribution.

Why the Group Ratio Is a Weighted Residual Summary

The observed-to-predicted ratio has a direct connection to relative claim-level errors. Define the relative residual $q_i=(Y_i-\mu_i)/\mu_i$, which is meaningful for positive fitted means. Then:

\frac{\sum_{i\in G}Y_i}{\sum_{i\in G}\mu_i}-1 = \frac{\sum_{i\in G}(Y_i-\mu_i)}{\sum_{i\in G}\mu_i} = \frac{\sum_{i\in G}\mu_i q_i}{\sum_{i\in G}\mu_i}.

Thus, the group’s observed-to-predicted ratio minus 1 is a fitted-mean-weighted average of relative residuals. Claims with larger fitted means contribute more to the group total. This explains why the ratio summarizes aggregate calibration but is not generally equal to the unweighted average of claim-level ratios.

Worked Example: Finding Miscalibration Across Fitted-Mean Ranges

A hypothetical insurer reviews 15 fully observed claims from a single severity target. Claims are grouped by fitted mean. The low-mean group has predicted total severity of $500 and observed total severity of $450. The middle group has predicted total severity of $1,000 and observed total severity of $1,100. The high-mean group has predicted total severity of $2,000 and observed total severity of $2,600. Calculate each group’s observed-to-predicted ratio and the overall ratio. Identify the pattern.

Method. For each group, divide observed total severity by predicted total severity.

Low-mean group: $450/500=0.9000$. Its observed total is 10% below its predicted total.

Middle group: $1{,}100/1{,}000=1.1000$. Its observed total is 10% above its predicted total.

High-mean group: $2{,}600/2{,}000=1.3000$. Its observed total is 30% above its predicted total.

The overall observed total is $450+1{,}100+2{,}600=4{,}150$. The overall predicted total is $500+1{,}000+2{,}000=3{,}500$. Therefore, the overall ratio is $4{,}150/3{,}500=1.1857$, to four decimal places.

Interpretation. The increasing ratios suggest that the model underpredicts more strongly in the higher fitted-mean range. The overall ratio of 1.1857 does not show where the discrepancy occurs; the grouped ratios do. The pattern is a diagnostic signal to investigate, not by itself a proof that a particular predictor or model form is wrong.

Common mistake avoided. Averaging the three group ratios equally gives $(0.9+1.1+1.3)/3=1.1000$, which is not the overall observed-to-predicted ratio. The correct overall ratio uses the totals, $4{,}150/3{,}500=1.1857$, because groups have different predicted totals.

Diagnosing Claim-Size and Tail Patterns

A severity model specifies more than a mean when its distributional assumptions are used. A claim-size diagnostic can therefore compare observed tail counts with the counts implied by fitted conditional distributions. For each claim, calculate the model-implied probability of exceeding a relevant threshold, then sum those probabilities. Under a correctly specified model, that sum is the expected number of exceedances. A persistent excess of large claims may indicate that the fitted distribution understates tail probability, but a modest discrepancy can occur by chance.

Another useful display plots residuals against fitted means or against relevant predictors. Look for curvature, a changing spread, or runs of residuals with the same sign. For Gamma models, increasing raw residual spread with fitted mean can be expected because the conditional standard deviation is $\sqrt{\phi}\mu_i$; assess standardized residuals as well. When claims are sorted or grouped by their observed sizes, remember that the response itself is being used to form the groups. Such a display can be useful for finding extreme claims, but an apparent pattern from that grouping alone is not an independent calibration test.

Worked Example: Checking the Fitted Upper Tail

Suppose a hypothetical portfolio’s individual severities are modeled as independent conditional Gamma variables with mean $\mu_i$ and variance $\phi\mu_i^2$, where $\phi=0.25$. For a Gamma distribution, this corresponds to shape $1/\phi=4$ and scale $\phi\mu_i$. In a review of 30 claims, each claim is tested against the threshold $2\mu_i$, using its own fitted mean. Five observed claims exceed their respective thresholds. Calculate the model’s expected number of exceedances and compare it with the observed count.

Method. If $Y_i$ has the stated Gamma distribution, $Y_i/\mu_i$ has shape 4 and scale 0.25, so the exceedance probability at $2\mu_i$ is the same for every claim. At this threshold, the corresponding unit-scale Gamma variable has threshold $2/0.25=8$. The integer-shape Gamma survival formula gives

P(Y_i\ge 2\mu_i) =e^{-8}\left(1+8+\frac{8^2}{2!}+\frac{8^3}{3!}\right) =e^{-8}(126.3333) \approx 0.0424.

The expected number of exceedances is $30(0.0424)=1.272$, approximately 1.27 claims. The observed count is 5, which is $5/1.272\approx3.93$ times that expectation. The difference merits investigation of the upper tail, the fitted dispersion, and the relevant model assumptions. It is not, by itself, a formal significance test.

Check. The exceedance probability is below 0.5, as expected for a threshold twice the mean; multiplying it by 30 gives an expected count between 1 and 2. The observed count of 5 is therefore well above the model-based expectation.

Common mistake avoided. Comparing all claims with one dollar threshold would be incorrect if their fitted means differ. Here each claim is compared with its own $2\mu_i$, so the probability calculation is conditional on its fitted mean.

Coverage-Limit Groups and Data Comparability

A residual or observed-to-predicted summary can also be compared across coverage-limit groups. This can reveal that fit quality differs among groups with different limits, but it does not establish that the limit caused the difference. Limit groups may differ in policyholder mix, claim selection, or other modeled and unmodeled characteristics. Inspect group sizes and the distribution of fitted means as well as the ratios.

The comparison requires care about what was observed. In the example below, every response is assumed to be a complete observation of the same ground-up severity target; the policy limit identifies a group but does not cap the recorded response. If recorded amounts are capped, selected, or otherwise altered by coverage terms, comparing them as though they were complete observations of that target is not valid. The data treatment must match the modeled response.

Worked Example: Comparing Fit Across Coverage-Limit Groups

A hypothetical analysis models complete ground-up severity for claims on policies with two different coverage limits. Assume the observed responses are not capped or selected by those limits, and both groups use the same severity target. For the $25,000-limit group, the predicted total is $900 and the observed total is $855. For the $100,000-limit group, the predicted total is $1,800 and the observed total is $2,250. Calculate the ratios and state what the difference supports.

Method. Calculate observed total divided by predicted total separately for each limit group.

For the $25,000-limit group, $855/900=0.9500$. Observed severity is 5% below predicted severity.

For the $100,000-limit group, $2{,}250/1{,}800=1.2500$. Observed severity is 25% above predicted severity.

Across both groups, the observed total is $855+2{,}250=3{,}105$, and the predicted total is $900+1{,}800=2{,}700$. The combined ratio is $3{,}105/2{,}700=1.1500$.

Interpretation. The combined ratio hides the difference between the groups: the smaller-limit group is slightly below prediction, while the larger-limit group is notably above it. This supports investigating differences in calibration or population composition across limit groups. It does not show that changing the limit would cause severity to change.

Common mistake avoided. It would be wrong to interpret this comparison as a clean test of the effect of limits if the observed responses were capped at those limits. The stated assumption that all responses are complete ground-up severities is essential to this particular comparison.

A Practical Diagnostic Sequence

A disciplined sequence helps keep model diagnostics interpretable:

Confirm the target.
Check that the observed severity and fitted mean describe the same payment or loss measure, units, and claim population.

Check aggregate calibration.
Calculate total observed severity divided by total predicted severity, then report the corresponding totals.

Break down the comparison.
Examine observed-to-predicted ratios by fitted-mean range and relevant predictors, including coverage-limit groups where the target supports that comparison.

Inspect residual and tail behavior.
Use residuals appropriate to the fitted variance structure and compare observed tail counts with the fitted distribution’s expected counts.

Interpret cautiously.
Consider group sizes, sampling variation, dependence, and differences in claim composition before changing the model.

A common exam trap is treating any ratio above 1 as evidence that a model must be rejected. A ratio is a descriptive summary; its interpretation depends on the size and stability of the group, how the group was formed, and whether the response data satisfy the modeling assumptions. Another trap is to blame a coverage limit for a residual pattern without checking whether the limit changed what entered the data or what amount was recorded.

Key takeaway: Diagnose conditional severity fit with totals and residual patterns across meaningful groups, not a single overall statistic. Ratios identify where observed severity differs from prediction; they do not, on their own, identify the cause.

Check Your Understanding

For each problem, use the stated response definition and assumptions.

A fitted-mean group has predicted total severity of $2,400 and observed total severity of $2,640. Calculate its observed-to-predicted ratio and state the direction of calibration difference. (Answer: Ratio $=1.1000$; observed severity is 10% above predicted.)
Under a Gamma severity model with $\phi=0.5$, the shape is 2 and the scale is $0.5\mu_i$. For 40 independent claims, each is checked against $2\mu_i$. Use $P(Y_i\ge2\mu_i)=e^{-4}(1+4)$ to find the expected exceedance count. (Answer: Probability $\approx0.0916$; expected count $\approx3.66$.)
Two fitted-mean groups have observed/predicted totals of $0.90$ and $1.20$, with predicted totals of $500 and $1,500, respectively. Calculate the overall ratio; do not average the two ratios equally. (Answer: Overall ratio $=(450+1{,}800)/(500+1{,}500)=1.1250$.)
A limit-group comparison gives observed total $1,020 and predicted total $1,200 for one group, and observed total $2,100 and predicted total $1,500 for another. Calculate both ratios and describe the pattern without claiming a causal limit effect. (Answer: Ratios are $0.8500$ and $1.4000$; the first group is below prediction and the second above prediction.)
A model predicts complete ground-up severity, but the data contain only payments capped at each policy’s limit. Is a direct observed-to-predicted comparison of those payments a valid calibration check for the complete ground-up target? (Answer: No; the recorded payment and modeled target differ, so the comparison is not valid without handling the limit’s effect consistently.)

What You'll Learn