Variance-To-Mean Ratios And Overdispersion

Why Compare Variance With the Mean?

A frequency model describes both the expected number of claims and how much claim counts vary around that expectation. For a Poisson count, the variance equals the mean. Comparing observed or model-implied variance with the mean therefore gives a compact way to assess whether the Poisson model’s level of variation is plausible. A ratio materially above one indicates greater variation than the Poisson benchmark, but a ratio from a small sample is not by itself proof that the Poisson model is wrong.

The comparison is meaningful only when counts are on a comparable basis. Risks with different exposure periods generally have different expected counts, so comparing the raw variance and mean of their counts can confuse exposure differences with frequency variation. The exposure-adjusted approach below compares each observed count with its own Poisson expectation.

Definition: For a count random variable \(N\) with positive mean, its variance-to-mean ratio is \(\operatorname{Var}(N)/E[N]\). A Poisson random variable with mean \(\mu\) has \(\operatorname{Var}(N)=\mu\), so its model-based variance-to-mean ratio is 1. A ratio greater than 1 is called overdispersion relative to the Poisson model; a ratio less than 1 is called underdispersion relative to it.

Observed Ratios for Equal Exposures

Suppose \(N_1,\ldots,N_n\) are independent claim counts observed over equal exposure periods and are treated as having a common mean. The sample mean and sample variance are

\bar{N}=\frac{1}{n}\sum_{i=1}^{n}N_i, \qquad s^2=\frac{1}{n-1}\sum_{i=1}^{n}(N_i-\bar{N})^2.

The observed variance-to-mean ratio is \(s^2/\bar{N}\), provided \(\bar{N}>0\). The denominator \(n-1\) is the conventional sample-variance denominator. It matters: using \(n\) instead would give a different variance and ratio. With small \(n\), the ratio can fluctuate substantially, even when a Poisson model is correct.

Worked Example: A Small Equal-Exposure Sample

A hypothetical insurer observes annual claim counts of \(1,1,2,2\) for four policies with equal annual exposure. Calculate the sample variance-to-mean ratio and interpret it relative to a Poisson model. Use the sample variance denominator \(n-1\).

Method. Calculate the sample mean, then the sample variance using \(n-1\), and finally their ratio. This method avoids confusing the total count with the mean or dividing the squared deviations by \(n\).

The number of policies is \(n=4\), and the total count is \(1+1+2+2=6\). Therefore,

\bar{N}=\frac{6}{4}=1.5.

The deviations from the mean are \(-0.5,-0.5,0.5,0.5\), so the sum of their squares is \(0.25+0.25+0.25+0.25=1\). Thus,

s^2=\frac{1}{4-1}=\frac{1}{3}, \qquad \frac{s^2}{\bar{N}} =\frac{1/3}{3/2} =\frac{2}{9} \approx 0.2222.

The observed ratio is below the Poisson benchmark of 1. This four-policy sample does not establish that the underlying process is underdispersed; it simply has relatively little observed variation. The calculation is checked by noting that the counts sum to 6, giving mean 1.5, and the squared deviations sum to 1.

Why the Poisson Benchmark Is One

The Poisson probability model with parameter \(\mu>0\) assigns probability \(e^{-\mu}\mu^k/k!\) to each nonnegative integer count \(k\). Its mean and variance are both \(\mu\). The ratio is therefore one, regardless of the value of the Poisson mean.

The equality follows from the Poisson moments. If \(N\) has a Poisson distribution with mean \(\mu\), then \(E[N]=\mu\). Also, using \(N^2=N(N-1)+N\), its second moment is

E[N^2]=E[N(N-1)]+E[N]=\mu^2+\mu.

Consequently, \(\operatorname{Var}(N)=E[N^2]-(E[N])^2=(\mu^2+\mu)-\mu^2=\mu\). This explains why a variance-to-mean ratio above one signals variation beyond the Poisson model’s prediction, rather than simply a high average claim count.

Adjusting for Unequal Exposure

Let policy \(i\) have exposure \(e_i>0\), measured in a consistent unit such as policy-years, and let \(N_i\) be its claim count. Under a Poisson frequency model with a common rate \(\lambda\) per unit of exposure, assume the counts are independent and

N_i\sim\operatorname{Poisson}(\lambda e_i), \qquad \mu_i=E[N_i]=\lambda e_i.

The expected count \(\mu_i\) differs across policies when their exposures differ. A useful measure of dispersion is the Pearson statistic divided by its degrees of freedom. If the common rate is estimated from the data, \(\hat{\lambda}=\sum_i N_i/\sum_i e_i\), then \(\hat{\mu}_i=\hat{\lambda}e_i\). With one fitted rate and \(n\) observations, the usual dispersion estimate is

\hat{\phi} = \frac{1}{n-1} \sum_{i=1}^{n} \frac{(N_i-\hat{\mu}_i)^2}{\hat{\mu}_i}.

This is a diagnostic summary, not an exact test in every sample. Under a correctly specified Poisson model, the Pearson statistic is often compared with \(n-1\), particularly when expected counts are not too small. Sparse counts can make that approximation unreliable. A value above one suggests extra variation relative to the fitted Poisson model; sample size and the individual residual contributions still matter.

Worked Example: Unequal Exposures and an Adjusted Dispersion Estimate

In a hypothetical portfolio, four policies have exposures \(0.5,1,1.5,2\) policy-years and observed annual claim counts \(0,1,1,4\), respectively. Estimate the common Poisson rate and calculate the Pearson dispersion estimate. Also show what the unadjusted raw-count ratio would suggest.

Method. Estimate the rate using total claims divided by total exposure, then compare each count with its own expected count. This avoids treating a two-year policy as though it had the same expected count as a half-year policy.

Total claims are \(0+1+1+4=6\), and total exposure is \(0.5+1+1.5+2=5\) policy-years. Hence

\hat{\lambda}=\frac{6}{5}=1.2 \quad\text{claims per policy-year}.

The fitted expected counts are \(1.2(0.5)=0.6\), \(1.2(1)=1.2\), \(1.2(1.5)=1.8\), and \(1.2(2)=2.4\). The four Pearson contributions are

\frac{(0-0.6)^2}{0.6}=0.6,\quad \frac{(1-1.2)^2}{1.2}=\frac{0.04}{1.2}=0.0333, $$ $$ \frac{(1-1.8)^2}{1.8}=\frac{0.64}{1.8}=0.3556,\quad \frac{(4-2.4)^2}{2.4}=\frac{2.56}{2.4}=1.0667.

Their sum is approximately \(2.0556\). One rate was estimated, so the degrees of freedom are \(4-1=3\). The adjusted estimate is

\hat{\phi}=\frac{2.0556}{3}\approx0.6852.

For comparison, the raw-count mean is \(6/4=1.5\). The deviations from 1.5 are \(-1.5,-0.5,-0.5,2.5\), whose squares sum to \(9\). Thus the raw sample variance is \(9/3=3\), and the raw variance-to-mean ratio is \(3/1.5=2\). That ratio appears overdispersed, while the exposure-adjusted estimate is below one. The disagreement occurs because the counts came from unequal exposure periods. The adjusted calculation is checked by confirming that fitted expected counts sum to \(0.6+1.2+1.8+2.4=6\), matching total observed claims.

Model-Based Overdispersion

A variance-to-mean comparison can also be made from a specified model, rather than estimated from a sample. For example, under a negative binomial parameterization with mean \(\mu\) and dispersion parameter \(\alpha>0\), suppose

\operatorname{Var}(N)=\mu+\alpha\mu^2.

Dividing by \(\mu>0\) gives a model-based ratio of \(1+\alpha\mu\), which exceeds one. This formula states how much variance the specified model predicts; it does not establish that the model is appropriate for any particular data set. The variance-to-mean ratio depends on the mean as well as on \(\alpha\), so it need not be the same at different exposure levels.

Worked Example: Comparing Poisson and Negative Binomial Variation

For a hypothetical policy, suppose the expected claim count is \(\mu=1.2\). Compare the Poisson variance-to-mean ratio with the negative binomial ratio when \(\alpha=0.5\), using \(\operatorname{Var}(N)=\mu+\alpha\mu^2\).

Method. Calculate each model’s variance first and then divide by its mean. This keeps the variance parameterization explicit and avoids treating \(\alpha\) itself as the variance-to-mean ratio.

Under the Poisson model, the variance equals the mean:

\operatorname{Var}(N)=1.2, \qquad \frac{\operatorname{Var}(N)}{E[N]}=\frac{1.2}{1.2}=1.

Under the stated negative binomial model,

\operatorname{Var}(N) =1.2+0.5(1.2)^2 =1.2+0.5(1.44) =1.2+0.72 =1.92.

Its ratio is \(1.92/1.2=1.6\), equivalently \(1+\alpha\mu=1+0.5(1.2)=1.6\). Thus the specified negative binomial model predicts 60% more variance than the Poisson model at this mean. The arithmetic checks because \(1.6(1.2)=1.92\).

Interpreting the Ratio Carefully

A variance-to-mean ratio is a concise diagnostic, but its interpretation depends on how counts were collected and what model was fitted. In particular:

Check the unit of exposure. Counts from different time periods should not be compared as if the periods were equal. For unequal exposures, compare observations with their exposure-specific expected counts.
Check the variance convention. The sample variance uses denominator \(n-1\). Replacing it with \(n\) changes the observed ratio.
Account for estimated parameters. When a common Poisson rate is estimated from the observations, the Pearson statistic is ordinarily divided by \(n-1\), not \(n\).
Respect sample size. A ratio based on few policies is highly variable. A value just above one is not compelling evidence of overdispersion on its own, and a value below one does not prove underdispersion.
Do not interpret the ratio as a cause. It summarizes variation relative to a Poisson benchmark. The ratio alone does not identify why the counts differ from that benchmark.

The Poisson model remains a useful reference even when another frequency model is being considered: its variance-to-mean ratio is exactly one. The comparison is most informative when the exposure basis, mean assumptions, and degrees of freedom are all stated clearly.

Key takeaway: For equal exposures, compare the sample variance with the sample mean. For unequal exposures under a common Poisson rate, compare each count with its own fitted expected count and use a degrees-of-freedom-adjusted Pearson statistic. Ratios above one indicate more variation than the Poisson benchmark predicts, subject to sampling uncertainty.

Check Your Understanding

For each problem, identify the relevant comparison and check whether exposure or sample size affects the interpretation.

Four equal-exposure policies have counts \(0,1,2,3\). Using the sample variance denominator \(n-1\), calculate the mean, sample variance, and variance-to-mean ratio. (Answer: mean \(=1.5\), variance \(=5/3\), ratio \(=10/9\approx1.1111\).)
Three policies have exposures \(0.5,1,1.5\) policy-years and counts \(0,1,3\). Fit a common Poisson rate and calculate the Pearson dispersion estimate, using \(n-1\) in the denominator. (Answer: fitted rate \(=4/3\); fitted means \(=2/3,4/3,2\); Pearson sum \(=1.25\); dispersion estimate \(=0.6250\).)
A negative binomial model has mean \(\mu=2\) and variance \(\mu+0.25\mu^2\). Calculate its variance and variance-to-mean ratio, and compare the ratio with Poisson. (Answer: variance \(=3\), ratio \(=1.5\), which is above the Poisson benchmark of 1.)
Five equal-exposure policies have counts \(2,3,3,4,4\). Calculate the sample variance-to-mean ratio using denominator \(n-1\). (Answer: mean \(=3.2\), variance \(=0.7\), ratio \(=0.21875\).)
Three policies have exposures \(1,2,5\) policy-years and counts \(0,2,7\). Estimate a common Poisson rate and calculate the Pearson dispersion estimate using \(n-1\) degrees of freedom. (Answer: fitted rate \(=9/8=1.125\); expected counts \(=1.125,2.25,5.625\); Pearson sum \(=1.4889\); dispersion estimate \(\approx0.7444\).)

What You'll Learn