Why Grouping Matters for Insurance Claims
Insurance policies are often nested within groups such as agencies, regions, or underwriting portfolios. Policies in the same group may share unmeasured features: local risk, sales practices, or claim-reporting procedures. If those shared features are ignored, claim counts can vary more across policies than a simple Poisson model allows. But a high number of zero-claim policies does not, by itself, identify the reason for that variation. Shared group risk, additional policy-level heterogeneity, and a structural-zero process imply different patterns in the data.
This tutorial compares those mechanisms using grouped claim counts. The calculations are model implications, not evidence that one mechanism is correct for any particular insurer. In the examples, all policies and numerical values are hypothetical.
The conditional mean is \(\lambda_{ij}\), given the group effect. The marginal mean averages over that effect. These are different quantities: the former describes policies at a specified group risk level, while the latter averages over the model's population of groups. In the calculations below, a policy's exposure and covariates are held fixed when comparing models.
What a Shared Poisson Random Intercept Implies
For a policy with fixed covariates and exposure, write \(a_{ij}=E_{ij}\exp(\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta})\), so \(\lambda_{ij}=a_{ij}\exp(b_j)\). The normal moment-generating function gives \(E[\exp(t b_j)]=\exp(t^2\sigma_b^2/2)\). It follows that the marginal mean is \(a_{ij}\exp(\sigma_b^2/2)\). Applying the law of total variance gives the central implication:
The first variance term is Poisson variation conditional on group risk. The second is variation in the conditional mean across groups. For two distinct policies in the same group, conditional independence does not make the marginal counts independent: both depend on \(b_j\). Their covariance is
Thus a random intercept produces both extra marginal variance and positive within-group association. It does not create a separate structural-zero class. It can produce many zeros because low-risk groups have small conditional means, but every policy still has a Poisson count distribution conditional on its group effect.
Worked Example: Marginal Variation and Dependence From Group Risk
Suppose a hypothetical insurer models one-year policies with a baseline rate of \(0.20\) claims per policy-year and \(\sigma_b^2=0.25\). For clarity, take \(E_{ij}=1\) and no covariates beyond the intercept. What marginal mean and variance does the random-intercept Poisson model imply, and how strongly are two policies in the same group associated?
The conditional mean at group effect \(b_j\) is \(0.20\exp(b_j)\). Averaging over \(b_j\sim N(0,0.25)\) gives $$ \mu=0.20\exp(0.25/2)=0.20\exp(0.125)\approx0.2266 $$ claims per policy-year.
Using the unrounded mean in the variance calculation, $$ \operatorname{Var}(Y)=\mu+\mu^2\{\exp(0.25)-1\} \approx0.22663+(0.20\exp(0.125))^2(0.28403) \approx0.2412 $$ claims-squared per policy-year count. The variance exceeds the mean because the group effect varies across groups.
For two policies with these same covariates and exposure in one group, $$ \operatorname{Cov}(Y_{ij},Y_{kj}) =[0.20\exp(0.125)]^2\{\exp(0.25)-1\} \approx0.01459 $$ claims-squared. The marginal variance of each count is about \(0.2412\), so their correlation is approximately \(0.01459/0.2412=0.0605\). This is positive but modest dependence. The covariance is not obtained by treating the policies as independent after averaging over their shared group effect.
Negative-Binomial Variation: Extra Spread Within Groups
A negative-binomial GLMM can retain the same normal random intercept while allowing additional conditional variation. Use the NB2 parameterization: conditional on \(b_j\), the count has mean \(\lambda_{ij}\) and variance \(\lambda_{ij}+\alpha\lambda_{ij}^2\), where \(\alpha\geq0\) is the dispersion parameter. At \(\alpha=0\), this conditional variance reduces to the Poisson variance. The conditional negative-binomial distribution can be understood as allowing policy-specific rate variation beyond the shared group effect.
This extra component changes marginal variance but does not replace shared group dependence. Since the conditional mean remains \(\lambda_{ij}\), the covariance of two distinct policies in a group is still the covariance induced by the common \(b_j\). The marginal variance becomes
Worked Example: Adding Negative-Binomial Dispersion
Continue with the hypothetical one-year policy in the preceding calculation, but suppose the conditional count follows an NB2 distribution with \(\alpha=0.80\). Keep the same mean structure and group-effect variance. Calculate the marginal variance and compare the two sources of overdispersion.
The marginal mean remains \(\mu=0.20\exp(0.125)\approx0.22663\) claims per policy-year. The variance multiplier is $$ (1+\alpha)\exp(\sigma_b^2)-1 =1.80\exp(0.25)-1 \approx1.31125. $$ Therefore, $$ \operatorname{Var}(Y) =\mu+\mu^2(1.31125) \approx0.22663+(0.20\exp(0.125))^2(1.31125) \approx0.2940 $$ claims-squared per policy-year count.
The random intercept alone gave variance about \(0.2412\); adding \(\alpha=0.80\) raises it to about \(0.2940\). For two distinct policies in the same group, the covariance is still about \(0.01459\) under these assumptions. The additional NB2 variation increases each policy's variance without adding the same amount to the covariance between different policies. Consequently, within-group correlation is lower than under the random-intercept Poisson model with these same marginal means and group effects.
The negative-binomial mechanism is therefore not interchangeable with a random intercept. If excess variation is mainly shared by policies in the same group, the random intercept predicts corresponding within-group covariance. If there is substantial extra variation among policies even after accounting for group risk, NB2 dispersion can represent it. The two mechanisms may both be needed, but a more complex model should be supported by patterns that the simpler model fails to capture.
Zero Inflation: A Separate Structural-Zero Mechanism
A zero-inflated model adds a latent state in which a policy is not at risk of producing a count under the ordinary count process. Let \(S_{ij}=1\) indicate the structural-zero state, with probability \(\pi\). If \(S_{ij}=1\), then \(Y_{ij}=0\); otherwise the count follows a Poisson component. A zero-inflated random-intercept Poisson model can use \(\lambda_{ij}=E_{ij}\exp(\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta}+b_j)\) for that count component. For the formulas here, assume the structural-zero indicators are independent across policies and independent of \(b_j\), with a common probability \(\pi\). More elaborate models can allow \(\pi\) to depend on covariates or group effects.
Conditional on the group effect, the observed mean is \((1-\pi)\lambda_{ij}\), and the conditional variance is \((1-\pi)\lambda_{ij}+\pi(1-\pi)\lambda_{ij}^2\). When \(b_j\) is also integrated out, the marginal mean is \((1-\pi)\mu^*_{ij}\), where \(\mu^*_{ij}=E(\lambda_{ij})\) is the marginal mean of the count component. The total variance can be derived by conditioning first on the structural state and then on the group effect.
where \(Y^*_{ij}\) denotes the count-component outcome, including its random intercept. If that component is the Poisson random-intercept model, then \(\operatorname{Var}(Y^*_{ij})=\mu^*_{ij}+(\mu^*_{ij})^2\{\exp(\sigma_b^2)-1\}\). The probability of an observed zero is \(\pi+(1-\pi)E[\exp(-\lambda_{ij})]\), not simply \(\pi+(1-\pi)\exp(-\mu^*_{ij})\) when group effects vary. The expectation of an exponential is not generally the exponential of the expectation.
Worked Example: Mean and Variance Under a Zero-Inflated Group Model
Suppose a hypothetical zero-inflated model has a structural-zero probability \(\pi=0.25\). Its Poisson random-intercept count component has marginal mean \(\mu^*=0.30\) claims per policy-year and \(\sigma_b^2=0.25\). Find the observed marginal mean and variance.
A policy enters the count component with probability \(1-\pi=0.75\), so the observed mean is $$ E(Y)=0.75(0.30)=0.225 $$ claims per policy-year.
The count-component variance is $$ \operatorname{Var}(Y^*)=0.30+(0.30)^2\{\exp(0.25)-1\} =0.30+0.09(0.28403)\approx0.32556. $$ The mixture variance is therefore $$ \operatorname{Var}(Y) =0.75(0.32556)+(0.25)(0.75)(0.30)^2 \approx0.2610 $$ claims-squared per policy-year count. The extra mixture term reflects variation between the structural-zero state and the count-component state. This variance calculation alone does not establish that structural zeros exist; the observed zero pattern must be assessed against plausible count and group-risk variation.
Reading Grouped Claim Patterns
Consider the following hypothetical counts for four groups, with five one-year policies in each group. These small data are useful for illustrating what to inspect, not for choosing a model reliably.
| Group | Claims for the five policies | Total claims | Zero-claim policies |
|---|---|---|---|
| A | 0, 0, 0, 1, 0 | 1 | 4 |
| B | 0, 1, 0, 0, 2 | 3 | 3 |
| C | 0, 0, 3, 0, 1 | 4 | 3 |
| D | 0, 4, 0, 0, 0 | 4 | 4 |
There are 12 claims in 20 policy-years, for a sample mean of \(12/20=0.60\) claims per policy-year, and 14 of 20 observations are zero. Both the concentration of claims in a few policies and the differences in group totals are potentially relevant. However, four groups are far too few to separate mechanisms convincingly, and a high zero proportion alone is not proof of zero inflation. A random intercept can generate low-risk groups with many zeros; an NB distribution can generate high policy-level variation; and a structural-zero process can also increase zeros. The models differ in their full distributions and dependence patterns, not just in their zero probabilities.
Confirm that \(Y_{ij}\) is a nonnegative count, exposure is positive and measured consistently, group membership is meaningful, and the offset proportionality assumption is defensible.
Compare group totals, zero proportions, and variation among policies with similar exposure. Persistent group-level clustering is a reason to examine a shared random effect; it is not by itself proof that a particular random-effects distribution is correct.
Check whether the fitted model reproduces the distribution of zeros, positive counts, group totals, and within-group association. A model that matches only the overall mean can still miss important features.
Use fit comparisons and predictive checks alongside substantive knowledge. A more flexible model may fit better simply because it has more parameters, and small numbers of groups can make distinctions particularly uncertain.
Comparison, Assumptions, and Limitations
| Model | Source of extra variation | Distinctive implication | Important limitation |
|---|---|---|---|
| Random-intercept Poisson GLMM | Shared group risk | Marginal overdispersion and positive covariance for policies in the same group | Conditional Poisson variation may be too restrictive if policy-level spread remains |
| Negative-binomial GLMM | Shared group risk plus conditional NB2 dispersion | Extra policy-level variance as well as group-induced dependence | Dispersion may be weakly identified alongside random-effect variation in sparse data |
| Zero-inflated Poisson GLMM | Shared group risk plus a structural-zero state | A separate probability of being outside the count-generating process | The latent zero state may not be identifiable from observed zeros without adequate information or substantive support |
The calculations assume correctly specified fixed effects, a log link, a normal random intercept with finite variance, conditional independence given the latent effects, and the stated conditional count distribution. An omitted predictor, inaccurate exposure, dependence between groups, or a poorly chosen random-effects distribution can produce apparent overdispersion or excess zeros. In particular, sparse counts may provide little information to distinguish a large random-effect variance, NB dispersion, and a zero-inflation probability.
A practical diagnostic is to compare observed and model-predicted distributions at both the policy and group levels, including zeros, upper-tail counts, and group totals. Persistent discrepancies in group totals point toward a failure to represent shared variation; excess within-group policy spread after accounting for groups is more compatible with extra conditional dispersion. Neither pattern uniquely determines a mechanism. Zero-inflation claims require more than observing many zeros: assess whether a structural state makes sense and whether the fitted model predicts the full count pattern adequately.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In a random-intercept Poisson GLMM, why can two policies be marginally associated even when they are conditionally independent?
- Suppose \(\sigma_b^2=0\) but \(\alpha>0\) in an NB2 model. Which source of variation remains, and is there random-intercept-induced covariance between distinct policies?
- A zero-inflated count component has marginal mean \(0.40\) and \(\pi=0.20\). What is the observed marginal mean?
- Does a high observed zero proportion prove structural zeros? Give one alternative explanation from the models considered here.
Solutions. (1) Both policies depend on the same random intercept \(b_j\); averaging over it yields covariance \(\mu_{ij}\mu_{kj}\{\exp(\sigma_b^2)-1\}\), which is positive when \(\sigma_b^2>0\). (2) Conditional NB2 overdispersion remains, with variance \(\lambda+\alpha\lambda^2\); because there is no shared random intercept, this mechanism alone does not induce covariance between distinct policies. (3) The observed mean is \((1-\pi)\mu^*=0.80(0.40)=0.32\) counts per stated exposure unit. (4) No. Shared group risk can create many low-risk groups with zero claims, and NB variation can also change the zero frequency; the full grouped count pattern and substantive mechanism must be considered.