Recognizing Overdispersion In Claim Counts

Why Overdispersion Matters

A Poisson frequency model is useful for pricing because it connects each policy’s expected claim count to its exposure and rating characteristics. But a good fit to average counts does not guarantee that the model describes how variable counts are around those averages. If counts vary more than the fitted Poisson model permits, the model may understate uncertainty and its fit should be examined before relying on its conclusions.

The central comparison is conditional: for policies with the same modeled characteristics and exposure, does observed claim-count variability exceed what their fitted means imply? This distinction matters because policies can have different fitted means even when a Poisson model is correct for each policy.

In a frequency model with a log link and an exposure offset, a typical fitted mean has the form \(\mu_i=e_i\exp(x_i^\mathsf{T}\widehat\beta)\), where \(e_i\) is exposure and \(x_i\) represents the policy’s modeled rating characteristics. The offset scales the expected count for exposure; it does not change the Poisson model’s conditional equality of mean and variance. The question here is whether that equality is adequate after conditioning on the fitted means.

Why the Poisson Mean Equals Its Variance

The equidispersion condition is a property of the Poisson distribution, not an empirical rule that every small group must satisfy exactly. For \(Y\sim\operatorname{Poisson}(\mu)\), the probability of count \(y\) is \(P(Y=y)=e^{-\mu}\mu^y/y!\), for \(y=0,1,2,\ldots\). The first two factorial moments are \(E[Y]=\mu\) and \(E[Y(Y-1)]=\mu^2\). Since \(Y^2=Y(Y-1)+Y\), the variance follows:

This gives the model’s benchmark: a policy with fitted mean \(0.6\) has conditional variance \(0.6\), while a policy with fitted mean \(2.4\) has conditional variance \(2.4\). A count is discrete and will not equal its expected value in general. Overdispersion is a claim about variability across observations relative to the fitted benchmark, not about whether an individual policy has a count above or below its mean.

Assessing Excess Variation With Pearson Residuals

For policy \(i\), the Pearson residual is the observed count minus its fitted mean, scaled by the square root of the fitted mean. Squaring and adding these residuals gives the Pearson statistic \(X_P^2\). Dividing by the residual degrees of freedom gives a familiar summary of excess variation.

Here \(n\) is the number of policy observations and \(p\) is the number of fitted model parameters, including an intercept if one is fitted. The quantity \(n-p\) is the residual degrees of freedom in the usual large-sample approximation. A value of \(\widehat\phi\) near 1 is broadly consistent with Poisson-scale variation. A value materially above 1 is a warning that the counts may be overdispersed relative to the fitted means. A value below 1 indicates less residual variation than the Poisson benchmark, not overdispersion.

This ratio is a diagnostic, not a universal pass-or-fail rule. Its interpretation depends on the amount of data, whether the mean structure is reasonable, and whether the model’s assumptions are appropriate. Small samples can produce substantial variation in the ratio. Also, fitted parameters use information from the same counts being checked, which is why the residual degrees of freedom account approximately for the parameters estimated.

A large Pearson ratio signals a mismatch between observed residual variation and the Poisson benchmark, but does not by itself establish why the mismatch occurred. Potential explanations include an inadequate mean specification, heterogeneous risk not represented by the rating variables, dependence between observations, or problems in the data. Determining which explanation is relevant requires further investigation; the dispersion calculation alone does not identify a cause.

Unequal Means Require a Conditional Comparison

When policies have different exposures or rating characteristics, their fitted Poisson means differ. A policy with a larger fitted mean is expected to have a larger variance under the model. Consequently, the raw variance of all counts pooled together is not, by itself, a valid test of conditional Poisson equidispersion.

The reason can be shown with the law of total variance. Let \(\mu\) denote a policy’s fitted mean, which varies across the portfolio, and suppose counts are Poisson conditional on that mean. Then:

The first term, \(E[\mu]\), is the average conditional Poisson variance. The second term is the extra variation that arises simply because policy means differ. Therefore, the portfolio-wide variance can exceed the portfolio-wide average fitted mean even if the conditional Poisson model is exactly right. Pearson residuals address this by scaling each policy’s deviation using its own fitted mean.

Interpreting the Evidence Carefully

A dispersion estimate well above 1 is a useful signal, but its meaning should be stated precisely: the residual count variation is larger than the fitted Poisson model’s benchmark, after an approximate adjustment for estimated parameters. It is not a direct estimate of a particular physical source of variation, and it does not prove that the fitted means are wrong. A model can describe average frequency reasonably while still failing to describe the spread of counts.

A few practical checks help keep the interpretation sound:

• Use the fitted mean for each policy in the Pearson denominator; do not treat unequal means as if they were equal.
• Use residual degrees of freedom that account for estimated parameters, rather than dividing automatically by the number of policies.
• Interpret the ratio in light of sample size. A value modestly above 1 in a small sample is not conclusive, and no single cutoff applies in every setting.
• Separate the finding from its explanation. A large ratio identifies a lack of fit in variability; further analysis is needed to determine the reason.
• Do not assume that a raw portfolio variance larger than the average fitted mean establishes overdispersion. Variation among the fitted means can create that result even under conditional Poisson assumptions.

Recognizing overdispersion is therefore an intermediate but important model-checking step: first establish what the Poisson model predicts conditionally, then compare observed residual variation with that benchmark, and only then investigate what the evidence may imply. This approach is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.