Overdispersion In Claim Counts

Why Claim-Count Variability Matters

A Poisson count model is a useful starting point for insurance frequency: when a policy has a fixed expected claim count, the model equates that count's variance with its mean. Real portfolios, however, may contain risks that differ in ways not represented in the model. The observed counts can then vary more than the Poisson benchmark predicts.

That pattern is called overdispersion. It matters because a model that captures the average number of claims but understates their variability can give an incomplete description of the data. The first task is to compare count variability with the appropriate mean; the next is to ask whether the excess reflects omitted differences among risks or some other feature of the data.

Definition: A count variable is overdispersed relative to a Poisson model when its variance exceeds its mean under the comparison being made. For a Poisson random variable \(N\sim\operatorname{Poisson}(\lambda)\), with fixed \(\lambda>0\), \(E[N]=\lambda\) and \(\operatorname{Var}(N)=\lambda\). Thus the Poisson benchmark is equality of variance and mean, not a requirement that a finite observed sample have exactly equal sample moments.

The Mean-Variance Comparison

For supplied theoretical moments, compare the variance directly with the mean. For observed counts, calculate the sample mean and sample variance using the stated data and denominator convention. A useful descriptive summary is the variance-to-mean ratio: divide the variance by the mean, provided the mean is positive. A ratio above 1 indicates a variance larger than the mean in that comparison; a ratio near 1 is consistent with the Poisson moment relationship.

The ratio is a diagnostic, not a stand-alone proof that a specific alternative distribution is correct. With finite samples, sample moments fluctuate, and an estimated ratio above 1 does not by itself identify the cause. The data also need to be compared on a suitable basis: if expected counts differ by exposure or by known risk characteristics, those differences should not be mistaken for unexplained variability.

Worked Example: Comparing Sample Count Moments

A hypothetical book of seven comparable policies has observed annual claim counts \(0,0,1,1,2,2,4\). Treat these as a descriptive sample and use the usual sample variance, with denominator \(n-1\). Does its observed variability exceed the Poisson moment benchmark?

Method and assumptions. This is a sample-moment diagnostic, not a formal test. The Poisson benchmark relates a count's theoretical variance and mean when its expected count is fixed. We compare the calculated sample variance with the calculated sample mean, recognizing that sample values need not match the theoretical moments exactly.

There are \(n=7\) observations. Their sum is \(0+0+1+1+2+2+4=10\), so the sample mean is \(\bar{x}=10/7=1.4286\). The sum of squared counts is \(0^2+0^2+1^2+1^2+2^2+2^2+4^2=26\).

The sample variance is \(s^2=[\sum x_i^2-(\sum x_i)^2/n]/(n-1)\). Substitution gives \(s^2=[26-100/7]/6=(82/7)/6=41/21=1.9524\).

The variance exceeds the mean by \(41/21-10/7=41/21-30/21=11/21=0.5238\). The variance-to-mean ratio is \((41/21)/(10/7)=41/30=1.3667\), to four decimals. The observed sample is therefore more variable than the Poisson moment benchmark by this descriptive comparison.

Check. The mean is about \(1.43\), while the sample variance is about \(1.95\); their difference is about \(0.52\), matching \(11/21\). The common mistake avoided here is dividing the sum of squared deviations by \(n\) while calling the result the usual sample variance. That would produce a different statistic; the problem specifies denominator \(n-1=6\).

Why Risk Heterogeneity Can Create Overdispersion

A key explanation is that policies may have different underlying expected claim counts. Suppose a policy's expected count, denoted by the random variable \(\Lambda\), varies across risks. Conditional on its own value \(\Lambda=\lambda\), assume the count \(N\) follows a Poisson distribution with mean and variance \(\lambda\). The unconditional count then includes both ordinary Poisson variation within a risk level and variation among the risk levels.

The law of total variance gives the reason precisely. It states that the variance of a random variable equals the expected conditional variance plus the variance of its conditional expectation. Under the stated conditional Poisson assumption, the expected conditional variance is \(E[\Lambda]\), and the variance of the conditional expectation is \(\operatorname{Var}(\Lambda)\). Therefore:

\begin{aligned} E[N] &= E\{E[N\mid\Lambda]\}=E[\Lambda],\\ \operatorname{Var}(N) &=E\{\operatorname{Var}(N\mid\Lambda)\} +\operatorname{Var}\{E[N\mid\Lambda]\}\\ &=E[\Lambda]+\operatorname{Var}(\Lambda). \end{aligned}

Because a variance cannot be negative, \(\operatorname{Var}(N)\ge E[N]\). If expected counts genuinely vary across risks, so that \(\operatorname{Var}(\Lambda)>0\), the marginal variance is strictly greater than the marginal mean. This shows how a collection of risks that are individually Poisson can display overdispersion when combined without accounting for their differing expected counts.

Important condition: The heterogeneity result assumes that counts are Poisson conditional on each risk's expected count and that \(\Lambda\) varies across the population. If \(\Lambda\) is fixed for every risk, then \(\operatorname{Var}(\Lambda)=0\), and this particular source of excess variance disappears. A variance above the mean does not, by itself, prove that unmodeled risk heterogeneity is the cause.

Worked Example: Two Unmodeled Risk Levels

Consider a hypothetical portfolio in which half the policies have conditional claim counts \(N\mid\Lambda=1\sim\operatorname{Poisson}(1)\), and half have \(N\mid\Lambda=3\sim\operatorname{Poisson}(3)\). The policy type is not included in a single pooled count model. Find the pooled mean, variance, and variance-to-mean ratio.

Method and assumptions. Use the law of total expectation and the law of total variance. Conditional on the risk level, the claim count is Poisson, so its conditional mean and variance both equal that level. The two levels have probability \(1/2\) each.

The expected count across the portfolio is \(E[N]=E[\Lambda]=(1/2)(1)+(1/2)(3)=2\). The mean risk level is therefore 2 claims per policy period.

The variance of the risk level is \(\operatorname{Var}(\Lambda)=E[\Lambda^2]-(E[\Lambda])^2\). Here \(E[\Lambda^2]=(1/2)(1^2)+(1/2)(3^2)=5\), so \(\operatorname{Var}(\Lambda)=5-2^2=1\).

The pooled count variance is \(\operatorname{Var}(N)=E[\Lambda]+\operatorname{Var}(\Lambda)=2+1=3\). Thus the variance-to-mean ratio is \(3/2=1.5\). The pooled count is overdispersed relative to a single Poisson model with mean 2.

Check. The additional variance of 1 is exactly the variance between the two risk-level means. The result also follows directly: the average conditional variance is \([(1)+(3)]/2=2\), and the variance of the conditional means is 1, giving \(2+1=3\). Treating all policies as though each had the same expected count would omit that between-risk component.

Separate Known Exposure Differences From Excess Variation

The comparison must respect the model's expected count for each observation. For example, a longer policy exposure ordinarily corresponds to a larger expected claim count than a shorter exposure, even when both share the same underlying claim rate. Pooling counts with different expected values and comparing their pooled variance only with their pooled mean can make this predictable variation look like unexplained overdispersion.

The same variance decomposition explains the issue. If policies have known, fixed expected counts \(\mu_i\), and each count is conditionally Poisson with mean \(\mu_i\), the variance across a mixture of policies includes variation in those means. A model that recognizes the different \(\mu_i\) compares each count with its own expectation. A model that assigns one common mean to all policies fails to represent that known difference. Thus a pooled variance greater than a pooled mean is not sufficient to diagnose excess variation after accounting for exposure and other modeled risk differences.

Worked Example: Exposure Variation Is Not Necessarily Unmodeled Excess

A hypothetical insurer observes two equally sized groups of policies. Each policy in group A has \(0.5\) policy-years of exposure, and each in group B has \(1.5\) policy-years. Assume a fixed rate of 1 expected claim per policy-year and, conditional on exposure, a Poisson count. Compare the pooled count moments with the moments conditional on exposure.

Method and assumptions. Exposure is known and fixed for each policy. The expected claim count is the rate multiplied by exposure, so group A has \(N\sim\operatorname{Poisson}(0.5)\), while group B has \(N\sim\operatorname{Poisson}(1.5)\). Use equal group weights to calculate the pooled moments.

The pooled mean is \((1/2)(0.5)+(1/2)(1.5)=1\). The average of the conditional variances is also \((1/2)(0.5)+(1/2)(1.5)=1\), because each conditional distribution is Poisson.

The variance between the conditional means is \((1/2)(0.5-1)^2+(1/2)(1.5-1)^2=(1/2)(0.25)+(1/2)(0.25)=0.25\). Therefore, the variance across the pooled policy population is \(1+0.25=1.25\). Its ratio to the pooled mean is \(1.25/1=1.25\).

Interpretation and check. The pooled moments show variance greater than mean, but the group-specific counts satisfy the Poisson mean-variance relationship: group A has both moments \(0.5\), and group B has both moments \(1.5\). The extra pooled variance of \(0.25\) comes from the known exposure difference. The common mistake avoided is to call this unexplained overdispersion without first accounting for exposure.

How to Use the Diagnostic Carefully

For exam-style calculations, make the comparison explicit and keep the interpretation tied to the assumptions. First identify whether the problem supplies theoretical moments or observed data. For observations, use the stated sample-variance convention. Then compare variance with mean and, if useful, report the ratio. Finally, check whether exposures or known risk categories imply different expected counts before attributing the difference to omitted heterogeneity.

Overdispersion can have several possible explanations. Unmodeled variation in expected claim counts is one important mechanism, as the derivation shows. A pattern in a limited sample can also arise from sampling variation, data problems, or a mean structure that does not adequately represent known differences. The diagnostic identifies a mismatch with the simple Poisson moment relationship; additional context is needed to explain that mismatch.

Common exam trap: Concluding that any pooled variance greater than the pooled mean proves an inadequate conditional Poisson assumption. The comparison is informative only after clarifying which mean and variance are being compared and whether expected counts vary across observations. Known exposure differences can increase pooled variance even when counts are Poisson conditional on exposure.

Identify the comparison.
Determine whether the quantities are theoretical moments or sample moments, and note the sample-variance denominator if applicable.

Compare variance with mean.
Variance greater than mean indicates overdispersion relative to the stated Poisson benchmark; a ratio above 1 summarizes the size of the difference.

Check expected-count differences.
Consider whether exposure or known risk groups produce different means before interpreting the excess as unexplained variability.

State the conclusion with limits.
Describe the observed moment pattern and a plausible mechanism, but do not claim that the diagnostic alone establishes the cause.

Key takeaway: Under a fixed-mean Poisson model, variance equals mean. When conditional Poisson claim counts have risk-specific expected counts, the variation among those means adds to the overall count variance. Diagnose the moment pattern carefully, while separating known exposure variation from genuinely unexplained excess.

Check Your Understanding

For each problem, state the relevant comparison and interpret the result in context.

A hypothetical sample has counts \(0,1,1,2,3\). Using the sample variance with denominator \(n-1\), calculate the mean, variance, and variance-to-mean ratio. Is the sample variance greater than the mean? (Answer: Mean \(=1.4\), sample variance \(=1.3\), ratio \(=0.9286\); no.)
A count model's supplied mean is \(2.4\) and its supplied variance is \(3.6\). Calculate the variance-to-mean ratio and classify the moment relationship. (Answer: Ratio \(=1.5\); the variance exceeds the mean, so the supplied moments are overdispersed relative to Poisson.)
Conditional on a risk-specific mean \(\Lambda\), suppose counts are Poisson. If half the hypothetical risks have \(\Lambda=0.4\) and half have \(\Lambda=1.6\), calculate the marginal mean and variance. (Answer: Mean \(=1\); \(\operatorname{Var}(\Lambda)=0.36\), so marginal variance \(=1.36\).)
A hypothetical portfolio has equal numbers of policies with fixed expected counts \(0.25\) and \(1.25\). Each count is Poisson conditional on its expected count. Find the pooled mean and variance, and say whether pooled overdispersion alone establishes excess conditional variability. (Answer: Pooled mean \(=0.75\); variance \(=0.75+0.25=1.00\), so the pooled variance exceeds the mean, but this is explained by the known difference in expected counts.)
For conditionally Poisson counts, suppose the average risk-specific mean is \(1.8\) and the variance of the risk-specific means is \(0.7\). Find the marginal variance and variance-to-mean ratio. (Answer: Marginal variance \(=1.8+0.7=2.5\); ratio \(=2.5/1.8=1.3889\).)

What You'll Learn