Poisson Variance Assumptions and Credibility Diagnostics

Why Check the Poisson Variance Assumption?

A standard frequency credibility calculation often starts with a Poisson model for claim counts. Under that model, the mean count and the variance of the count are equal. This is useful: the expected frequency determines not only the central estimate but also its sampling variability. If actual counts vary more than the model allows, a credibility calculation based on Poisson variation can overstate how much confidence the experience deserves.

A variance diagnostic is a check on the model, not a substitute for understanding the data. Excess variation can reflect differences among risks, dependence among claims, changes over time, or random fluctuation in a limited sample. The diagnostic can flag a mismatch; by itself it cannot identify the cause.

Definition: For independent claim counts \(N_1,\ldots,N_n\) observed over equal exposures under a common Poisson frequency \(\lambda\), the Poisson variance assumption is \(N_i\sim\operatorname{Poisson}(\lambda)\), so \(E[N_i]=\operatorname{Var}(N_i)=\lambda\). The dispersion index is the observed sample variance divided by the sample mean. A value near 1 is consistent with the model’s mean-variance relationship; a value materially above 1 indicates more variation than that relationship predicts.

The model requires more than a mean-variance equality. It assumes counts are generated by a Poisson process at a common frequency, with independent increments over nonoverlapping exposure periods. For separate risks, it also requires that the observations be independent and comparable after accounting for exposure. Unequal exposure must be allowed for explicitly; raw counts from different exposure amounts should not be compared as if their expected values were equal.

Equal-Exposure Diagnostics

For \(n\) independent observations with equal exposure, estimate the common expected count by the sample mean. The usual sample variance uses \(n-1\) in its denominator. Their ratio provides a direct, descriptive check of the Poisson mean-variance relationship.

\bar{N}=\frac{1}{n}\sum_{i=1}^{n}N_i,\qquad s_N^2=\frac{1}{n-1}\sum_{i=1}^{n}(N_i-\bar{N})^2,\qquad D=\frac{s_N^2}{\bar{N}}.

Here \(D\) is the dispersion index, provided \(\bar{N}>0\). If \(D\) is near 1, the observed variance is close to the Poisson variance estimated by the mean. If \(D>1\), the counts are overdispersed relative to that benchmark; if \(D<1\), they are underdispersed. “Near” is important: with few observations, the index can move substantially just from random variation.

Worked Example: Comparing Sample Variance With the Mean

Hypothetical data and model. Eight independent, equal-exposure groups have claim counts \(2,3,1,4,2,3,5,0\). Use a common Poisson frequency estimated from these observations. There is no interest-rate or mortality assumption in this claim-count problem; the fitted Poisson parameter is the mean count per group.

Step 1: Calculate the sample mean. The total count is \(2+3+1+4+2+3+5+0=20\), so \(\bar{N}=20/8=2.5\).

Step 2: Calculate the sample variance. The squared deviations from 2.5 are \(0.25,0.25,2.25,2.25,0.25,0.25,6.25,6.25\). Their sum is 18, so \(s_N^2=18/(8-1)=18/7=2.5714\), rounded to four decimals.

Step 3: Calculate and interpret the dispersion index. \(D=(18/7)/2.5=36/35=1.0286\), rounded to four decimals. The sample variance is close to the mean, so these data do not show a large departure from the Poisson variance assumption. This is a descriptive conclusion, not proof that the Poisson model is correct.

Method and trap avoided. This is the equal-exposure sample-variance-to-mean diagnostic. Dividing the sum of squared deviations by 8 instead of 7 would calculate a variance with the wrong denominator for the usual sample-variance definition and would change the index.

Why the Poisson Mean and Variance Are Equal

For a Poisson random variable \(N\) with parameter \(\lambda>0\), the probability of count \(k\), for \(k=0,1,2,\ldots\), is \(P(N=k)=e^{-\lambda}\lambda^k/k!\). Its probability generating function is \(G(t)=E[t^N]=e^{\lambda(t-1)}\). Differentiating gives \(G'(1)=E[N]=\lambda\). The second derivative gives \(G''(1)=E[N(N-1)]=\lambda^2\). Therefore \(E[N^2]=E[N(N-1)]+E[N]=\lambda^2+\lambda\), and

\operatorname{Var}(N)=E[N^2]-E[N]^2 =(\lambda^2+\lambda)-\lambda^2 =\lambda.

This identity is the reason a variance comparison is relevant to frequency credibility. With exposure \(E\), a constant frequency \(\lambda\) per exposure unit gives a total count with mean and variance \(\lambda E\). The estimated frequency \(N/E\) then has variance \(\lambda/E\), and its coefficient of variation is \(1/\sqrt{\lambda E}\). The square-root relationship used in standard frequency credibility calculations depends on this Poisson variation assumption.

If the variance is instead approximately \(\phi\) times the mean, with \(\phi>1\), then the frequency estimate has approximately \(\phi\) times the Poisson variance and \(\sqrt{\phi}\) times its standard deviation. This explains why unexamined overdispersion can make a Poisson-based credibility standard too optimistic. A diagnostic does not, on its own, specify a replacement credibility model.

Unequal Exposures and the Pearson Dispersion Statistic

When group \(i\) has known exposure \(e_i\), the common-rate Poisson model is \(N_i\sim\operatorname{Poisson}(\lambda e_i)\), independently across groups. The expected count is now different for each group. Estimate the rate by total claims divided by total exposure, then compare each observed count with its own fitted expected count.

\hat{\lambda}=\frac{\sum_{i=1}^{n}N_i}{\sum_{i=1}^{n}e_i}, \qquad \widehat{N}_i=\hat{\lambda}e_i, \qquad Q=\sum_{i=1}^{n}\frac{(N_i-\widehat{N}_i)^2}{\widehat{N}_i}.

The statistic \(Q\) is the Pearson statistic for count variation. When the common rate is estimated from the same data, there is one fitted parameter, so \(Q\) is commonly compared with a chi-square distribution with \(n-1\) degrees of freedom as an approximate diagnostic. That approximation is most useful when expected counts are not too small. Sparse counts, particularly many fitted expectations well below 1, make the approximation less dependable.

A convenient summary is \(Q/(n-1)\). A value near 1 is broadly consistent with the fitted Poisson variation; a value substantially above 1 suggests overdispersion. The statistic should be examined with the underlying counts and exposure pattern, not interpreted as a mechanical pass-or-fail rule.

Worked Example: Checking Counts With Different Exposures

Hypothetical data and model. Four independent groups have exposures \(1,2,3,4\) policy-years and counts \(4,9,15,20\). Assume a common Poisson frequency per policy-year, with expected count \(\lambda e_i\). No interest-rate or mortality assumption applies; the rate is estimated from these counts and exposures.

Step 1: Estimate the frequency. Total claims are \(4+9+15+20=48\), and total exposure is \(1+2+3+4=10\) policy-years. Thus \(\hat{\lambda}=48/10=4.8\) claims per policy-year.

Step 2: Calculate fitted expected counts. The four fitted counts are \(4.8(1)=4.8\), \(4.8(2)=9.6\), \(4.8(3)=14.4\), and \(4.8(4)=19.2\). The observed-minus-expected differences are \(-0.8,-0.6,0.6,0.8\).

Step 3: Calculate the Pearson statistic. The four contributions are \(0.64/4.8=0.1333\), \(0.36/9.6=0.0375\), \(0.36/14.4=0.0250\), and \(0.64/19.2=0.0333\). Their sum is \(Q=0.2292\), rounded to four decimals. With \(4-1=3\) degrees of freedom, the dispersion summary is \(Q/3=0.0764\), rounded to four decimals. This small statistic does not indicate excess variation in these observations; with only four groups, it is not strong evidence that the underlying model is correct.

Method and trap avoided. This is the unequal-exposure Pearson diagnostic. Treating the four counts as equal-exposure observations and comparing their raw-count variance with their mean would ignore the fact that expected counts increase with exposure.

Recognizing Overdispersion and Its Limits

Overdispersion means that observed count variation exceeds what the fitted Poisson model predicts. It can be visible in a high dispersion index or a high Pearson statistic relative to its degrees of freedom. It is a property of the observed variation relative to a specified model, not a diagnosis of the mechanism behind that variation.

For example, suppose risks have different underlying frequencies even after they have been grouped together. Counts may then vary more than a single common-rate Poisson model predicts. A changing rate over time, clustering of claims, or dependence between observations can also create excess variation. In each case, the common-rate independent Poisson model may not describe the data adequately. The right next step is to examine the exposure definition, classification, time period, and dependence structure before changing a credibility calculation.

A high value is also not conclusive evidence of a structural problem. A finite sample can have unusually large variation by chance. Conversely, a low value does not prove independence, a stable rate, or a correct Poisson model. The diagnostic checks a particular implication of the model; it does not establish every assumption.

Worked Example: Interpreting a Clear Overdispersion Signal

Hypothetical data and model. Eight independent, equal-exposure groups have counts \(0,0,1,1,5,5,6,6\). For the diagnostic, fit one common Poisson parameter to all groups. There is no interest-rate or mortality assumption; the fitted Poisson parameter is the mean count per group.

Step 1: Calculate the fitted mean. The total count is 24, so \(\bar{N}=24/8=3\).

Step 2: Calculate the sample variance. Four counts are 3 below the mean and four are 2 above it. The sum of squared deviations is \(4(3^2)+4(2^2)=36+16=52\). Therefore \(s_N^2=52/7=7.4286\), rounded to four decimals.

Step 3: Calculate the dispersion index and Pearson statistic. The index is \(D=(52/7)/3=52/21=2.4762\), rounded to four decimals. The Pearson statistic is \(Q=52/3=17.3333\), rounded to four decimals, with 7 degrees of freedom. Both summaries indicate considerably more variation than the fitted Poisson mean-variance relationship predicts.

Interpretation. These data warrant investigation of group differences, time effects, and possible dependence before applying a Poisson-based frequency credibility standard. The calculation does not establish which explanation is responsible.

Method and trap avoided. This is an equal-exposure dispersion check using the fitted common mean. A large index is evidence of a model mismatch to investigate, not automatic proof that a particular alternative model should be used.

A Practical Diagnostic Sequence

Verify the observations.
Confirm that each count covers the intended claim definition, period, and exposure. Check for duplicate claims, missing periods, and inconsistent exposure units.

Choose the exposure structure.
Use the equal-exposure comparison only for comparable exposure amounts. For differing exposures, model the expected count as a common frequency multiplied by exposure.

Compare variation with the fitted expectation.
Calculate the sample variance-to-mean index for equal exposures or the Pearson statistic for unequal exposures. Report the sample size and fitted expectations with the result.

Investigate rather than overinterpret.
Consider exposure, risk mix, time variation, and dependence. A dispersion statistic flags a possible mismatch; it does not determine its cause or prescribe a credibility adjustment.

Common Exam Traps

Using raw counts when exposure differs. Expected counts are proportional to exposure under a common rate. Compare each count with its fitted expectation rather than using an equal-exposure variance calculation.
Confusing a sample variance with a population variance. The standard sample variance divides by \(n-1\). The Pearson statistic instead sums squared residuals divided by their fitted expected counts.
Reading a dispersion index of 1 as proof. A value near 1 is compatible with the Poisson variance relationship, but does not establish the other assumptions or rule out chance.
Concluding that every value above 1 is significant. The size of the departure, the number of observations, and the reliability of any chi-square approximation all matter.
Applying a Poisson credibility standard without checking its premise. If observed variability is materially greater than Poisson variability, the standard calculation may understate uncertainty. A separate, justified model is needed to quantify any adjustment.

Key takeaway: The Poisson model predicts equal mean and variance for counts at equal exposure, and a variance proportional to expected count when exposure differs. Dispersion diagnostics test that prediction. They are evidence about model fit—not proof of a cause and not an automatic replacement for the model.

Check Your Understanding

For each problem, identify the relevant exposure structure before interpreting the calculation.

Five independent, equal-exposure groups have counts \(1,2,4,1,2\). Calculate the mean, sample variance, and dispersion index. State whether the index suggests overdispersion. (Answer: Mean \(=2\); sample variance \(=1.5\); dispersion index \(=0.75\), so the data are underdispersed relative to the Poisson benchmark.)
Six equal-exposure groups have counts \(1,1,1,5,5,5\). Calculate the mean, sample variance, and dispersion index. (Answer: Mean \(=3\); sample variance \(=4.8\); dispersion index \(=1.6\), suggesting overdispersion.)
Four independent groups have exposures \(1,2,2,3\) and counts \(2,3,5,8\). Under a common Poisson frequency per exposure unit, calculate the estimated frequency, fitted counts, and Pearson statistic. (Answer: \(\hat{\lambda}=18/8=2.25\); fitted counts \(2.25,4.5,4.5,6.75\); \(Q=0.8148\) to four decimals, with 3 degrees of freedom.)
A Poisson-based full-credibility standard requires 400 claims. An experience has 100 claims, and a credible dispersion assessment supports a variance-to-mean factor \(\phi=2.25\). Using the square-root relationship, calculate the adjusted full standard and partial credibility. (Answer: Adjusted full standard \(=400(2.25)=900\) claims; partial credibility \(=\sqrt{100/900}=1/3\), or about \(0.3333\).)
Two equally exposed groups have different underlying Poisson frequencies: 2 and 6 claims per exposure unit. For one exposure unit in each group, find the combined mean count and variance of the total count. Compare the variance with a single common-rate Poisson model fitted to that combined mean. (Answer: Combined mean \(=2+6=8\), and variance \(=2+6=8\). The fitted common-rate Poisson model also has variance 8 for this total, so this particular total-count comparison alone does not reveal the between-group difference.)

What You'll Learn