Grouping Rare Categorical Levels

Why Group Rare Categorical Levels?

A rating factor may have categories with very little exposure or few claims. A frequency estimate based on a small amount of data can be highly variable; if a level has no claims, its unadjusted Poisson maximum-likelihood estimate is zero. Yet a rare level should not automatically be merged with another. Pooling is most defensible when the categories are also plausibly similar in risk and can be treated consistently in underwriting, pricing, and administration.

Grouping changes the model, not just the display of its results. When two levels are pooled, the model requires them to share a rating effect. The resulting indication is more stable when their underlying risks are similar, but it can conceal a genuine difference when they are not. The central task is therefore to balance data support with the meaning of the categories.

Definition: Grouping rare categorical levels means replacing two or more original levels of a rating factor with a common level and fitting the model under the constraint that those original levels have the same effect. A level is rare relative to the available data—often because it has low exposure, few claims, or both—not because it falls below one universal numerical threshold.

Exposure and the Pooled Frequency Indication

For a simple frequency model, let \(N_j\) be the claim count and \(E_j\) the earned exposure for level \(j\). Assume \(N_j\) has a Poisson distribution with mean \(E_j\lambda_j\), where \(\lambda_j\) is the claim frequency per exposure unit. This corresponds to a log-link model with \(\log(E_j)\) as an offset. In this simple setting, the separate maximum-likelihood estimate is \(N_j/E_j\) when \(E_j>0\).

If levels in a proposed group are assumed to have one common frequency \(\lambda\), and their claim counts are conditionally independent, their combined count is Poisson with mean \(\lambda\sum_j E_j\). Provided total exposure is positive, the pooled frequency estimate is the combined claim count divided by combined exposure. Importantly, it is an exposure-weighted average of the separate rates, not their unweighted average.

\widehat{\lambda}_{\text{pool}} = \frac{\sum_j N_j}{\sum_j E_j}, \qquad \sum_j E_j>0.

Here is the likelihood justification, including the boundary case. Ignoring factors that do not depend on \(\lambda\), the likelihood is proportional to \(\lambda^{\sum_j N_j}\exp(-\lambda\sum_j E_j)\), for \(\lambda\geq 0\). If total claims are positive, differentiating the log-likelihood for \(\lambda>0\) gives \(-\sum_j E_j+(\sum_j N_j)/\lambda\). Setting this equal to zero gives the ratio above; the log-likelihood is concave, so this stationary point is the maximum. If total claims are zero, the log-likelihood is \(-\lambda\sum_j E_j\), which decreases as \(\lambda\) increases. Its maximum over \(\lambda\geq0\) is therefore at the boundary, \(\widehat{\lambda}_{\text{pool}}=0\). Thus the ratio formula remains valid when total exposure is positive, including the zero-claim case.

\widehat{\lambda}_j=\frac{N_j}{E_j}, \qquad \widehat{\lambda}_{\text{pool}} =\frac{\sum_j E_j\widehat{\lambda}_j}{\sum_j E_j}.

These formulas apply directly to the stated Poisson model with exposure as the only difference among observations. In a multivariable GLM, other predictors may vary across levels. Then raw totals divided by exposure need not equal the adjusted fitted indications. Pooling still imposes a common categorical effect, but the model should be refitted with its other predictors and offsets intact.

Worked Example: Pooling Two Low-Volume Levels

A hypothetical insurer studies annual claim frequency for two vehicle-use levels within an otherwise identical rating stratum. Level A has 40 earned vehicle-years and 8 claims; Level B has 60 earned vehicle-years and 3 claims. Calculate the separate rates and the pooled rate, then compare the indications. Assume independent Poisson counts with mean equal to exposure times frequency.

Separate indications. For A, \(8/40=0.2000\) claims per vehicle-year. For B, \(3/60=0.05000\) claims per vehicle-year. The separate indications differ by \(0.2000-0.05000=0.1500\).

Pooled indication. The combined claim count is \(8+3=11\), and combined exposure is \(40+60=100\) vehicle-years. Therefore, the pooled estimate is \(11/100=0.1100\) claims per vehicle-year.

Check. The exposure-weighted calculation gives \((40(0.2000)+60(0.05000))/100=(8+3)/100=0.1100\), matching the pooled count-over-exposure calculation. Pooling lowers A's indication and raises B's. It is not automatically appropriate: the observed separation may reflect genuine risk differences, random variation, or omitted characteristics. A common effect is a modeling assumption to assess, not a consequence of low volume alone.

Method and mistake avoided. The method is the exposure-offset Poisson estimate. Averaging the two rates without weights would give \((0.2000+0.05000)/2=0.1250\), which is wrong because the levels have different exposures.

Choosing Levels to Combine

A sound grouping decision uses several kinds of evidence. Exposure volume indicates how much policy or risk time supports an estimate. Claim counts show how much observed experience informs it. Neither measure alone determines whether two levels belong together. A high-frequency level with substantial exposure may be well supported yet genuinely distinct; a level with no claims may have too little exposure to establish that its true frequency is zero.

Operational relevance asks whether the categories represent meaningfully similar risks and can reasonably receive the same treatment. Consider whether they have similar definitions, underwriting characteristics, and plausible loss mechanisms. Also consider whether a proposed combined category can be explained and administered clearly. Do not group categories simply because their point estimates happen to be close, or simply because a test fails to detect a difference: small samples can make meaningful differences hard to detect.

Measure data support.
Review earned exposure and claim counts for each level, along with the stability and uncertainty of the indications. Flag sparse experience without treating any single cutoff as universal.

Assess substantive similarity.
Use the level definitions and relevant risk characteristics to decide whether a shared effect is plausible. A data-driven grouping that contradicts the category meanings needs a clear justification.

Fit and compare.
Refit the model with the proposed grouping, then compare fitted indications and uncertainty with those from separate levels. Keep other predictors and exposure definitions consistent.

Review consequences.
Check whether the combined indication is interpretable and operationally usable, and whether the grouping obscures a material difference or creates an unsuitable classification.

Worked Example: A Zero-Claim Level

A hypothetical insurer observes 20 earned units of exposure and 0 claims in a small category, and 30 earned units and 2 claims in a neighboring category. Assume independent Poisson counts with one common frequency if the categories are pooled. Find the separate and pooled maximum-likelihood frequency indications, and explain what the zero count does and does not establish.

Separate indications. The first category has \(0/20=0.0000\) claims per exposure unit. Its estimate is a boundary estimate: with zero claims and positive exposure, the likelihood is maximized at frequency zero. The second category has \(2/30=0.06667\) claims per exposure unit.

Pooled indication. There are \(0+2=2\) claims over \(20+30=50\) exposure units, giving \(2/50=0.04000\) claims per exposure unit. The pooled estimate is positive because the combined count is positive.

Check and interpretation. The exposure-weighted calculation is \((20(0)+30(0.06667))/50=2/50=0.04000\), subject to rounding. The zero estimate for the first category does not prove its underlying frequency is zero; it is the likelihood estimate from a small observed sample. If both categories had zero claims, their pooled estimate would also be zero, attained at the boundary, rather than obtained from an interior derivative solution.

Method and mistake avoided. The method is the Poisson maximum-likelihood estimate with nonnegative frequency. A common error is to use a derivative equation that divides by the total claim count and then apply it when that count is zero. The derivative has no positive-frequency root in that case; maximize the likelihood at the boundary instead.

How Grouping Changes Rating Indications

In a multiplicative frequency model with a log link, a categorical level contributes a factor to the fitted frequency, conditional on the other predictors. Grouping two levels constrains their factors to be equal. It does not generally set the shared factor to the arithmetic average of the original fitted factors. The result depends on the likelihood, exposures, other covariates, and the rest of the refitted model.

The following simple example isolates the categorical factor by holding the rest of the rating profile fixed. Its rates can therefore be compared directly as frequency relativities to the stated reference rate. In a more complex model, the same comparison should be made using refitted adjusted predictions on a consistent basis.

Worked Example: Separate Versus Shared Relativities

For a hypothetical product, the reference category has 1,000 exposure units and 100 claims. Two rare levels have the following experience: level A has 40 exposure units and 8 claims; level B has 60 exposure units and 3 claims. Assume Poisson frequency, a common baseline profile apart from this factor, and no other differences. Compare separate and pooled rate relativities to the reference.

Reference rate. The rate is \(100/1{,}000=0.1000\) claims per exposure unit.

Separate indications. A's rate is \(8/40=0.2000\), so its relativity is \(0.2000/0.1000=2.000\). B's rate is \(3/60=0.05000\), so its relativity is \(0.05000/0.1000=0.5000\).

Pooled indication. The levels together have \(8+3=11\) claims and \(40+60=100\) exposure units, so their shared rate is \(11/100=0.1100\). Relative to the reference, their shared relativity is \(0.1100/0.1000=1.100\).

Check and interpretation. The pooled rate is above the reference by \(0.1100-0.1000=0.01000\), while the separate indications place A well above and B below it. The combined indication is an exposure-weighted result, not evidence that both levels individually have frequency 0.1100. Pooling would substantially change both indications in this example, so operational similarity and further risk information matter before imposing a common effect.

Method and mistake avoided. The method compares fitted rates on the same reference basis. Averaging the separate relativities gives \((2.000+0.5000)/2=1.250\), not the pooled indication of 1.100, because the exposures are unequal.

Common Traps and Practical Checks

Using an unweighted average. For a simple Poisson model, pool claims and exposure, or weight separate rates by exposure. Do not give a small level the same weight as a large one unless that weighting is explicitly part of a different method.
Treating zero observed claims as zero risk. A zero-count estimate is a boundary maximum-likelihood estimate, not proof of zero underlying frequency. Its stability depends on the exposure and the model.
Grouping only to make estimates look smoother. Smoother indications are not necessarily more accurate. A common effect can bias the indication if the levels are materially different.
Comparing unadjusted and adjusted quantities as if they were identical. Raw claim counts divided by exposure are direct estimates only under the simple model conditions stated above. With other predictors, compare refitted adjusted predictions.
Forgetting the operational definition. The grouped category must have a coherent meaning that can be applied consistently. If it does not, a statistically convenient grouping may be difficult to use responsibly.

Key takeaway: Group levels when limited data support and substantive similarity jointly justify a shared effect. In the simple exposure-offset Poisson model, the pooled frequency is total claims divided by total exposure; when total claims are zero, the maximum is at the boundary frequency of zero. In a fuller GLM, refit the model and evaluate the resulting indications under the same rating profile.

Check Your Understanding

For each problem, use the stated assumptions and distinguish pooled indications from separate ones.

A hypothetical category has 80 exposure units and 6 claims; a second has 20 exposure units and 4 claims. Under an exposure-offset Poisson model, find both separate rates and the pooled rate. (Answer: separate rates 0.07500 and 0.2000; pooled rate 0.1000 claims per exposure unit.)
A reference level has 500 exposure units and 45 claims. Two rare levels have 30 exposure units and 4 claims, and 70 exposure units and 5 claims. With the same profile apart from this factor, calculate their separate relativities and their pooled relativity to the reference. (Answer: separate relativities 1.481 and 0.7937; pooled rate 0.09000 and pooled relativity 1.000.)
Two levels have 25 and 35 exposure units, respectively, and both have zero claims. What is the pooled maximum-likelihood frequency, and is it an interior solution? (Answer: 0 claims per exposure unit; no, it is the boundary maximum at frequency zero.)
Two levels have rates 0.1600 and 0.08000, based on 25 and 75 exposure units. Calculate the pooled rate and explain why the arithmetic mean is not the pooled estimate. (Answer: pooled rate \((25(0.1600)+75(0.08000))/100=0.1000\); the unweighted mean is 0.1200 and gives the unequal exposures equal weight.)
Two operationally distinct categories each have sparse data and estimated rates 0.09000 and 0.1500. State one reason data scarcity may support investigating a grouping and one reason it does not, by itself, justify pooling. (Answer: scarcity can make separate estimates unstable; it does not establish that the categories have the same underlying risk or should receive a common effect.)

What You'll Learn