Diagnostics for Frequency and Severity Fits

Why Diagnose a Fitted Model?

A fitted frequency or severity model reduces a set of observations to a mathematical description. Before using that description in further calculations, an actuary should check whether it captures important features of the data. A model can match the average claim count while missing unusually high counts; a severity model can fit the center of the loss distribution while understating its upper tail. Diagnostics make these discrepancies visible.

A diagnostic is evidence about model adequacy, not an automatic verdict. Results depend on the amount and quality of data, how the model was fitted, and how observations were grouped. In particular, a small expected count can make a grouped count comparison unstable, while a few extreme losses can have a large effect on tail comparisons.

Definition: A frequency diagnostic compares observed claim counts with counts or count summaries implied by a specified frequency model. A severity diagnostic compares observed loss values or their empirical distribution with probabilities, quantiles, or tail quantities implied by a specified severity model. A discrepancy is a reason to investigate the model and data; by itself, it does not identify the cause.

Observed and Expected Frequency Counts

Suppose $N$ comparable exposure units produce independent claim counts with the same fitted frequency distribution. Divide the possible counts into $m$ disjoint categories. If the fitted model assigns probability $p_j$ to category $j$, the expected number of observations in that category is $E_j=Np_j$. Let $O_j$ be the observed number. A basic diagnostic is the difference $O_j-E_j$; its standardized form helps account for the scale of the expected count.

E_j=Np_j,\qquad r_j=\frac{O_j-E_j}{\sqrt{E_j}},\qquad X^2=\sum_{j=1}^{m}\frac{(O_j-E_j)^2}{E_j}.

The quantity $r_j$ is a Pearson residual for category $j$, and $X^2$ is the grouped Pearson statistic. A large positive residual means the category occurred more often than expected; a large negative one means it occurred less often. The contributions to $X^2$ are nonnegative, so the total measures discrepancy without allowing positive and negative differences to cancel.

The formula has a direct justification. Under a multinomial count model with fixed total $N$, the count in category $j$ has mean $Np_j=E_j$ and variance $Np_j(1-p_j)$. Its standard deviation is therefore approximately $\sqrt{E_j}$ when $p_j$ is small. Dividing the difference by $\sqrt{E_j}$ expresses it in approximate standard-deviation units. Summing squared scaled differences gives the Pearson form. This is a useful diagnostic, but it does not make every small-sample comparison a reliable formal test.

When the fitted model has parameters estimated from the same observations, estimating those parameters uses information in the data and generally reduces the effective degrees of freedom. Under the usual regularity conditions, a formal grouped goodness-of-fit test uses approximately $m-1-k$ degrees of freedom, where $k$ parameters were estimated. Sparse expected counts can invalidate the usual approximation. Combine sensible adjacent categories when necessary, and state which categories were combined; do not interpret a statistic from tiny expected counts as if it were automatically reliable.

Worked Example: Grouping Sparse Poisson Count Categories

In a hypothetical portfolio, 20 equal exposure units are observed. A fitted Poisson model has mean $\lambda=0.85$ claims per unit. The observed counts are 8 units with zero claims and 12 with at least one claim. Calculate the expected counts and grouped Pearson statistic. The fitted rate is treated as specified for this diagnostic.

Method. Calculate Poisson category probabilities, multiply by 20, and compare the two observed and expected counts. Grouping all positive counts avoids relying on a separate category with a small expected count.

For a Poisson random variable, $P(N=0)=e^{-\lambda}$. Thus, $P(N=0)=e^{-0.85}\approx0.4274149319$, and $P(N\geq1)=1-e^{-0.85}\approx0.5725850681$.

The expected zero-count total is $E_0=20(0.4274149319)=8.548298638$. The expected positive-count total is $E_{1+}=20(0.5725850681)=11.451701362$. These add to $8.548298638+11.451701362=20$, as required.

The two Pearson contributions, using unrounded expected counts, are $(8-8.548298638)^2/8.548298638\approx0.03517$ and $(12-11.451701362)^2/11.451701362\approx0.02625$. Therefore $X^2\approx0.03517+0.02625=0.06142$, or 0.0614 to four decimal places.

As a check on the grouping choice, the fitted expected number with exactly two or more claims is $20[1-e^{-0.85}(1+0.85)]\approx4.18565$. That is a relatively small expected count, so keeping it as a separate cell would be questionable for a conventional chi-square approximation. The small grouped statistic indicates close agreement for this particular split; it does not prove that the Poisson model is correct.

Common mistake avoided: Do not calculate a two-or-more probability as $1-e^{-\lambda}$, which is the probability of at least one claim. The probability of two or more is $1-e^{-\lambda}(1+\lambda)$.

Empirical Severity and Distribution Comparisons

For observed losses $x_1,\ldots,x_n$, the empirical distribution function at a threshold $x$ is the fraction of observations no greater than $x$. Its complement, using a strict exceedance convention, is the fraction greater than $x$. Comparing these empirical proportions with fitted model probabilities is a direct way to check whether a model describes the body and tail of the observed losses.

\widehat F_X(x)=\frac{1}{n}\sum_{i=1}^{n} I(x_i\leq x), \qquad \widehat S_X(x)=\frac{1}{n}\sum_{i=1}^{n} I(x_i>x).

Here $I(\cdot)$ equals 1 when its condition is true and 0 otherwise. Compare $\widehat F_X(x)$ with the fitted $F_X(x)$, or compare $\widehat S_X(x)$ with the fitted $S_X(x)$, at several thresholds. A distribution-comparison plot can show systematic differences across the range. A tail-focused comparison gives greater attention to exceedances at high thresholds, where a model used for large-loss assessment must be credible.

Thresholds should be chosen and interpreted carefully. Several empirical tail proportions calculated from the same losses are dependent: the losses exceeding a larger threshold are a subset of those exceeding a smaller one. Therefore, do not treat a list of nested exceedance comparisons as independent tests. Also record the number of observations behind a tail proportion. A small number of extreme observations makes the empirical tail estimate variable and sensitive to individual claims.

Worked Example: Finding a Severity Model’s Tail Shortfall

Ten hypothetical claim amounts, in thousands of dollars, are $0.2, 0.4, 0.6, 0.8, 1.1, 1.3, 1.8, 2.2, 3.5,$ and $4.0$. Compare them at thresholds of $1,000, $2,000, and $3,000 with a fitted exponential severity model having mean $1,000. Treat this fitted parameter as specified for the comparison.

Method. Count observed strict exceedances at each threshold, then compare their proportions with the model survival probabilities. For an exponential distribution with mean $1,000, $S_X(x)=e^{-x/1000}$.

Six losses exceed $1,000, so the empirical exceedance proportion is $6/10=0.6000$. The fitted probability is $e^{-1}\approx0.3679$, giving an expected exceedance count of $10(0.3679)=3.6788$. The observed count is higher.

Three losses exceed $2,000, so the empirical proportion is $3/10=0.3000$. The model probability is $e^{-2}\approx0.1353$, and the expected count is $10(0.1353)=1.3534$.

Two losses exceed $3,000, so the empirical proportion is $2/10=0.2000$. The model probability is $e^{-3}\approx0.0498$, and the expected count is $10(0.0498)=0.4979$. Thus, at this threshold, the empirical proportion is about $0.2000/0.0498\approx4.02$ times the fitted tail probability.

The increasing contrast at higher thresholds is evidence that this fitted exponential model may understate the observed tail. However, the highest-threshold comparison is based on only two claims, so it is not conclusive. The thresholds overlap in the observations they count, and the comparison should be read as a diagnostic pattern rather than as three independent tests.

Common mistake avoided: A close fit at lower thresholds does not establish a good upper-tail fit. Check exceedances at relevant high thresholds instead of relying only on an overall impression or a central summary.

Checking for Frequency Overdispersion

For independent, identically distributed Poisson counts with mean $\lambda$, the variance is also $\lambda$. Consequently, a sample variance substantially above the sample mean can be evidence of overdispersion relative to a homogeneous Poisson model. A useful descriptive index is the sample variance divided by the sample mean. An index near 1 is consistent with the Poisson mean-variance relationship; a value well above 1 calls for investigation.

This comparison requires care. Counts should be on comparable exposure bases; pooling unequal exposures without adjustment can create apparent extra variation. Overdispersion can arise from several features, such as different underlying rates across exposure units, dependence, or an inadequate frequency model. The index identifies a mismatch with the simple Poisson relationship; it does not determine which explanation applies.

Worked Example: Interpreting Excess Variation in Annual Counts

A hypothetical group of ten equal-exposure units has annual claim counts $0,0,1,1,1,2,2,3,5,5$. Assess the data’s mean-variance relationship under a Poisson benchmark with mean estimated by the sample mean. Also compare the observed number of counts at least 4 with the fitted Poisson expectation.

Method. Calculate the sample mean and unbiased sample variance, then compare their ratio with 1. For the tail check, calculate the fitted Poisson probability of four or more claims.

The counts sum to $0+0+1+1+1+2+2+3+5+5=20$, so the sample mean is $20/10=2$. Their squared deviations from 2 sum to $4+4+1+1+1+0+0+1+9+9=30$. The sample variance is $30/(10-1)=3.3333$, and the sample variance-to-mean index is $3.3333/2=1.6667$.

The sample variance exceeds the Poisson variance benchmark of 2. This is evidence of overdispersion in these observations, not proof that the counts follow a particular alternative model.

Two of the ten observed counts are at least 4, so the empirical proportion is $2/10=0.2000$. Under Poisson mean 2, $P(N\geq4)=1-e^{-2}(1+2+2^2/2!+2^3/3!)$. The sum inside the parentheses is $1+2+2+8/6=6.3333333$. Since $e^{-2}\approx0.1353353$, the fitted tail probability is approximately $1-(0.1353353)(6.3333333)=0.1428764$. The expected number among ten units is $10(0.1428764)=1.4288$.

The observed count of two is above the fitted expectation of about 1.4288, but the difference is small in absolute terms and there are only ten observations. Taken together, the variance index and tail comparison suggest checking the Poisson assumption more closely; neither establishes a definitive failure on its own.

Common mistake avoided: The Poisson variance equals its mean as a population property. When the mean and variance are estimated from a small sample, their observed values need not be equal even when a Poisson model is plausible.

How to Use Diagnostics Responsibly

A disciplined diagnostic review connects each comparison to the model assumptions and the data available. For frequency, examine observed-versus-expected categories and variation relative to the fitted mean. For severity, examine empirical distribution comparisons at several points and pay particular attention to thresholds relevant to the intended use of the model. In both cases, retain the units, exposure basis, grouping choices, and number of observations behind each comparison.

Specify the comparison.
State the fitted model, its parameters, and the exposure or severity data being compared. Make clear whether parameters were estimated from these same observations.

Calculate expected quantities.
Use the model probabilities for the stated count categories or severity thresholds, and check that expected category counts sum to the total number of observations.

Inspect discrepancies and data support.
Report observed differences alongside their scale, and note sparse categories, few tail observations, or unequal exposures that affect interpretation.

Describe the evidence, not a verdict.
Say what feature appears poorly represented and what should be checked next. Do not infer a cause from a diagnostic alone.

A frequent exam trap is to call any observed count above its expectation evidence of overdispersion. One category can exceed expectation while another falls below it under an adequate model. Overdispersion concerns excess variability relative to the specified frequency model and should be assessed from the pattern of counts or a suitable dispersion diagnostic, not from one positive difference alone.

Key takeaway: Expected counts, empirical distribution comparisons, tail exceedances, and variance-to-mean checks reveal different aspects of fit. Interpret each in the context of its assumptions, sample size, exposure basis, and tail data; a diagnostic flags a question for investigation rather than proving a model correct or incorrect.

Check Your Understanding

For each problem, use the stated model and data to compute the requested diagnostic quantity.

Thirty equal exposure units have a fitted Poisson mean of 0.1 claim per unit. What is the expected number of units with zero claims? (Answer: $30e^{-0.1}\approx27.145$ units.)
A fitted exponential severity model has mean $2,000. For 50 claims, what is the expected number exceeding $4,000? (Answer: $50e^{-2}\approx6.7668$ claims.)
Of 12 observed losses, 5 exceed $1,000. A fitted model assigns probability 0.25 to exceeding $1,000. Give the empirical exceedance proportion and fitted expected exceedance count. (Answer: $5/12\approx0.4167$ and $12(0.25)=3$ claims.)
Twelve equal-exposure annual counts have sample mean 3 and sample variance 6. What is the variance-to-mean index, and what does it suggest relative to a Poisson model? (Answer: $6/3=2$; it is evidence of overdispersion, not proof of its cause.)
A fitted count model gives expected counts 3 and 2 in two adjacent categories, while the observed counts are 1 and 4. What is their combined observed count and combined expected count, and why might combining them help? (Answer: 5 observed and 5 expected; combining avoids interpreting two sparse expected-count cells separately.)

What You'll Learn