Why Compare Claim-Count Variance With Its Mean?
A claim-count model is not assessed by its average alone. Two groups of policyholders may have the same average number of claims but very different spreads of observed counts. Conversely, a pooled portfolio can show a large spread because it combines risks with different underlying frequencies, even when variation within each risk group is modest. Separating these possibilities is a practical first step in diagnosing frequency data.
For a Poisson count with mean \(\lambda\), the variance is also \(\lambda\). This equality is a useful reference point, not a rule that every finite data set must satisfy exactly. An empirical variance above the empirical mean is a signal to investigate; by itself, it does not establish why the data vary or prove that a particular model is correct.
The denominator convention matters. The definition above uses \(m\), so it describes the variance of the empirical distribution that assigns probability \(1/m\) to each observation. Some statistical summaries instead report the sample variance \(s^2=\frac{1}{m-1}\sum_j(n_j-\bar n)^2\), which is an unbiased estimator of a population variance under independent, identically distributed sampling with finite variance. Do not compare one version with the mean and another version across groups without saying which convention is being used. The examples here use \(s_m^2\) throughout.
What Pooling Does to Variance
Suppose observations are partitioned into groups, such as risk classes. Let group \(g\) contain \(m_g\) counts, have empirical mean \(\bar n_g\), and empirical variance \(s_{m,g}^2\). The group weight is \(w_g=m_g/m\), where \(m=\sum_g m_g\). The pooled mean is the weighted average of group means. The pooled variance has an additional component reflecting how far the group means lie from the pooled mean.
The first variance term is the weighted average of within-group variances. The second is the variance of the group means around the pooled mean, weighted by group size. It is nonnegative, and it is positive whenever group means differ. Therefore, pooling can increase the observed variance even if the within-group variances are not unusually large.
Here is a short justification. For an observation \(n_{gj}\) in group \(g\), write its deviation from the pooled mean as \((n_{gj}-\bar n_g)+(\bar n_g-\bar n)\). Squaring and summing within group gives a within-group squared-deviation sum, a group-mean squared-deviation sum, and a cross-product. The cross-product is zero because \(\sum_j(n_{gj}-\bar n_g)=0\). Dividing the resulting total by \(m\) gives the displayed identity.
The connection with a Poisson model makes the diagnosis concrete. Suppose each observation comes from a Poisson distribution conditional on its group, with group mean \(\lambda_g\). In the population, the pooled mean is \(E[\lambda_G]\), while the pooled variance is \(E[\lambda_G]+\operatorname{Var}(\lambda_G)\). The first term is average within-group Poisson variance; the second is variation in the group-specific means. Thus, even if every group follows a Poisson model, a mixture of groups with different means has variance greater than its mean.
This is a population result under the stated conditional Poisson assumptions. Empirical group means and variances are estimates from finite data, so they fluctuate. A high pooled ratio can motivate stratification and model checking, but it is not proof that the conditional counts are Poisson or that group membership is the only source of variation.
Worked Example: Separating Within-Group and Between-Group Variation
A hypothetical insurer records one-year claim counts for four policies in each of two risk groups. Group A has counts \(0,1,1,2\); Group B has counts \(1,3,3,5\). Calculate the two group means and empirical variances, then calculate the pooled mean, pooled variance, and variance-to-mean ratio. Use denominator \(m_g\) within each group and denominator \(m\) for the pooled variance.
Method. Calculate each group summary first, then use the variance decomposition. This avoids attributing all pooled variation to variation among otherwise similar policies.
For Group A, the total count is \(0+1+1+2=4\), so \(\bar n_A=4/4=1\). Its squared deviations from 1 are \(1,0,0,1\), giving \(s_{m,A}^2=(1+0+0+1)/4=0.5\).
For Group B, the total is \(1+3+3+5=12\), so \(\bar n_B=12/4=3\). Its squared deviations from 3 are \(4,0,0,4\), giving \(s_{m,B}^2=(4+0+0+4)/4=2\).
There are eight policies, with equal group weights \(w_A=w_B=4/8=0.5\). The pooled mean is \(\bar n=0.5(1)+0.5(3)=2\). The within-group component is \(0.5(0.5)+0.5(2)=1.25\). The between-group component is \(0.5(1-2)^2+0.5(3-2)^2=1\). Therefore, the pooled variance is \(1.25+1=2.25\), and the ratio is \(2.25/2=1.125\).
Check. Directly, the pooled squared deviations from 2 sum to \(18\), so the pooled variance is \(18/8=2.25\), as obtained from the decomposition. The pooled data show variance above mean, while each group’s empirical variance is below its own mean. The between-group component explains how pooling can create that signal.
Common mistake avoided. It would be incorrect to say that the pooled ratio proves excess variation within each risk group. The group calculations show that the pooled variance includes variation between the group means.
Exposure Must Be Considered Before Comparing Counts
A policy observed for two years has more opportunity to generate claims than a policy observed for one year. Thus, raw counts are not directly comparable across unequal observation periods. If \(e_j\) is the exposure of observation \(j\), measured in consistent exposure units, a common-rate Poisson model specifies \(N_j\sim\operatorname{Poisson}(\lambda e_j)\). Its mean and variance are both \(\lambda e_j\), not a common value for every observation.
Even with a common underlying rate, pooled counts across unequal exposures have different conditional means. In the population, if exposure \(E\) varies and \(N\mid E\sim\operatorname{Poisson}(\lambda E)\), then \(E[N]=\lambda E[E]\) and \(\operatorname{Var}(N)=\lambda E[E]+\lambda^2\operatorname{Var}(E)\). The second term arises from exposure variation. This is another reason a pooled raw-count variance can exceed its mean without establishing differing risk rates.
Worked Example: Checking Counts From Unequal Exposure Periods
A hypothetical portfolio has four observations with exposure 1 unit and counts \(0,1,0,1\), and four observations with exposure 2 units and counts \(0,2,0,2\). Find each exposure stratum’s estimated rate, the pooled rate, and the pooled raw-count mean and empirical variance. Explain why the raw-count comparison alone is not a valid common-mean Poisson diagnostic.
Method. Estimate each stratum’s rate by total claims divided by total exposure. Then summarize the eight raw counts separately, keeping their differing exposure periods in view.
For the first stratum, claims total \(0+1+0+1=2\) over exposure \(1+1+1+1=4\), so its estimated rate is \(2/4=0.5\) claims per exposure unit. For the second, claims total \(0+2+0+2=4\) over exposure \(2+2+2+2=8\), so its rate is \(4/8=0.5\).
Across both strata, total claims are \(2+4=6\), and total exposure is \(4+8=12\). The pooled rate is therefore \(\hat\lambda=6/12=0.5\) claims per exposure unit. The rates agree in this data set.
The eight raw counts sum to 6, so their mean is \(6/8=0.75\). Their squared values sum to \(1+1+4+4=10\). Using \(s_m^2=\frac{1}{m}\sum_j n_j^2-\bar n^2\), the empirical variance is \(10/8-(0.75)^2=1.25-0.5625=0.6875\).
Check. The variance-to-mean ratio is \(0.6875/0.75=0.9167\) to four decimal places. This ratio describes these pooled raw counts, but the observations do not have a common exposure or a common Poisson mean. Under the fitted common-rate model, expected counts are \(0.5\) for exposure 1 and \(1\) for exposure 2. A count-level assessment should respect those different expected values, rather than treating all eight counts as identically distributed with mean \(0.75\).
Common mistake avoided. Dividing every count by exposure and then comparing the variance of those rates with their mean would not restore a Poisson count model: a count divided by exposure is a rate observation, not a Poisson count. Use the exposure-adjusted rate for estimating frequency, and account for exposure when diagnosing count variability.
A Practical Diagnostic Sequence
A useful analysis proceeds from the data structure to the variance comparison, rather than starting with a single pooled ratio. The following sequence helps identify whether a result is being driven by unequal observation periods, differences between groups, or variation remaining within groups.
Confirm that each count refers to a consistent risk unit and observation window, and identify the exposure units recorded for it.
For comparable observations, state the empirical mean, empirical variance convention, and variance-to-mean ratio. A Poisson count model supplies a mean-equals-variance reference.
Compare groups with similar exposure periods and relevant risk characteristics. Calculate each group’s mean and variance rather than relying only on the pooled summaries.
Use the within/between decomposition to see whether group means account for much of the pooled variance. Treat the result as a diagnostic, not a causal conclusion.
The choice of groups matters. Groups should be defined using information relevant to expected claim frequency and known at the appropriate observation point. Creating many tiny groups can make empirical means and variances unstable; combining genuinely different risks can conceal useful structure. In small data sets, report the group sizes alongside the summaries so that a striking ratio based on only a few observations is not mistaken for a precise estimate.
Also check data quality before interpreting dispersion. Inconsistent exposure units, missing zero-claim policies, duplicate records, and differing claim-reporting windows can distort both the mean and variance. A comparison is meaningful only when the count definition and observation basis are consistent.
Common Traps and Interpretation
- Mixing variance conventions. The empirical distribution variance divides by \(m\); the conventional sample variance divides by \(m-1\). State and use one convention consistently.
- Comparing counts with unequal exposure as if they had the same mean. A longer observation period allows more claims. Estimate rates per exposure unit and retain the exposure structure in a count-model diagnostic.
- Calling any ratio above 1 proof of heterogeneity. A finite sample fluctuates, and other features of the data or model may also explain the signal. The ratio is descriptive evidence for further investigation.
- Assuming stratification must remove all excess variation. Groups can still contain heterogeneous risks, and empirical results within a small group can be noisy.
For a first-pass frequency assessment, the key distinction is between variation among observations within comparable groups and variation caused by differences between group means. Exposure adjustment establishes a fairer basis for comparison; stratification reveals structure that a pooled variance can hide or exaggerate. These calculations support model assessment, but do not replace a reasoned choice of model or an examination of data quality.
Check Your Understanding
Use the empirical distribution variance with denominator equal to the number of observations unless a question states otherwise.
- A hypothetical group has observed counts \(0,1,1,2,2\). Calculate its empirical mean, empirical variance, and variance-to-mean ratio. (Answer: mean \(1.2\), variance \(0.56\), ratio \(0.4667\).)
- Two equally sized groups have means 1 and 3, and empirical variances 0.5 and 2. Find the pooled mean and variance. How much of the pooled variance is the between-group component? (Answer: pooled mean \(2\), pooled variance \(2.25\), between-group component \(1\).)
- Four observations each have exposure 1 and total 2 claims; another four each have exposure 2 and total 4 claims. Find the two stratum rates and the pooled rate. (Answer: \(0.5\), \(0.5\), and \(0.5\) claims per exposure unit, respectively.)
- Two equally sized groups have means 1 and 4 and empirical variances 1 and 2. Find the pooled mean, pooled variance, and variance-to-mean ratio. (Answer: pooled mean \(2.5\), pooled variance \(3.75\), ratio \(1.5\).)
- A portfolio contains 12 claims over 12 exposure units in one stratum and 16 claims over 16 exposure units in another. Under a common-rate model, what is the pooled rate, and why should the two raw count totals not be compared as if they were observations with equal exposure? (Answer: pooled rate \(28/28=1\) claim per exposure unit; the totals have different numbers of contributing observations or exposure structures and are not directly comparable without exposure information.)