High-Cardinality Rating Factors

Why High-Cardinality Factors Need Special Care

A rating factor is high-cardinality when it has many possible levels. Examples might include a detailed vehicle classification, a large set of territories, or a product code with numerous categories. A model can estimate a separate effect for each level, but the existence of a code does not mean there is enough experience to estimate its effect reliably.

The central issue is the relationship between model complexity and level volume. Each additional freely estimated level effect uses information. When a level has little exposure or few claims, its fitted relativity can be driven by a small number of observations rather than a persistent risk difference. At the same time, merging levels indiscriminately can conceal real differences or create groups that are difficult to explain. The goal is not to minimize the number of levels at any cost; it is to make distinctions that are adequately supported and defensible.

Measure Support in the Context of the Model

For a claim-frequency model, policies or exposure units are not interchangeable with claims as measures of precision. A level may have substantial exposure but few claims, leaving its observed frequency imprecise. Conversely, a large number of claims can support a more precise frequency estimate, subject to the model assumptions and the exposure definition. For a severity model, claim count is usually more directly relevant than policy count because severities are observed on claims.

To make the frequency calculations precise, suppose claim counts \(N_j\) for levels \(j=1,\ldots,K\) are independent Poisson random variables, with mean \(e_j\lambda_j\). Here \(e_j\) is earned exposure in consistent units and \(\lambda_j\) is the expected claim frequency per exposure unit. This model assumes a constant frequency within each level over the period being modeled, conditional on the model predictors, and independent Poisson counts. If these assumptions are unsuitable—for example, because the data have material overdispersion—the uncertainty calculations below need an appropriate alternative model or variance adjustment.

Under these assumptions, \(\operatorname{Var}(N_j)=e_j\lambda_j\), so the estimated variance of \(\widehat{\lambda}_j\) is approximately \(\widehat{\lambda}_j/e_j\). For a positive count, the estimated standard error of the log frequency is approximately \(1/\sqrt{N_j}\). This is a useful diagnostic of relative uncertainty: it depends on claim count, not exposure alone.

Pooling Levels Without Losing the Meaning of the Factor

Pooling replaces several level-specific effects with one effect for a combined group. A defensible grouping may reflect a known classification structure, similar observed experience, or a documented business reason. The earlier material on grouping rare categorical levels addresses the mechanics of rare-level grouping; here the emphasis is on evaluating whether a factor with many levels has sufficient support overall and whether its distinctions remain credible.

When combining frequency experience, add claims and exposure, then calculate the pooled frequency. Do not take an unweighted average of the individual frequencies unless the exposures happen to be equal. The pooled estimate is exposure-weighted because it is the total number of claims divided by the total exposure.

Why the Poisson Precision Formula Holds

The approximate log-frequency standard error follows from the Poisson model, rather than from a rule of thumb about how many policies a level contains. For one level with exposure \(e\) and count \(N\), the log-likelihood, omitting terms that do not involve \(\lambda\), is \(N\log(\lambda)-e\lambda\). Its maximum occurs at \(\widehat{\lambda}=N/e\). The negative second derivative of the log-likelihood at that estimate is \(N/\widehat{\lambda}^{\,2}\), so the usual likelihood-based variance approximation for \(\widehat{\lambda}\) is \(\widehat{\lambda}^{\,2}/N\). Applying the delta method to \(\log(\widehat{\lambda})\) gives variance approximately \(1/N\), and hence standard error \(1/\sqrt{N}\).

This is a local, large-sample approximation. It is not reliable as a substitute for judgment when a level has very few claims, and it does not yield a finite log-rate estimate when \(N=0\). In a GLM with other predictors, the uncertainty of a level effect also depends on the model design and the distribution of experience across combinations of predictors.

Separate Effects, Complexity, and Validation

For a factor with \(K\) levels, an ordinary categorical effect generally requires \(K-1\) independent contrasts relative to a reference or under an equivalent constraint. A high-cardinality factor can therefore consume many parameters. If each level has strong, broad experience, this may be appropriate. If many levels are weakly supported, the fitted effects can be unstable, and apparent in-sample improvement may not persist in new data.

A useful comparison is between a model that pools a proposed set of levels and one that estimates separate effects. Under the Poisson assumptions, the likelihood-ratio deviance compares their fit. For a pre-specified split into two levels with counts \(N_1,N_2\) and exposures \(e_1,e_2\), the deviance against a pooled-rate model is \(2\sum_{j=1}^2 N_j\log(N_j/\widehat{N}_j)\), where \(\widehat{N}_j=e_j(N_1+N_2)/(e_1+e_2)\). A good in-sample fit improvement is evidence to investigate, not automatic proof that the distinction is stable or worth retaining.

Choosing and Defending a Level Structure

A sound assessment combines numerical support with context. Before retaining or combining levels, examine the volume relevant to the response, the stability of estimates over time or across data partitions, and whether the proposed distinctions have a defensible interpretation. Useful checks include:

• Exposure and response volume: inspect both the amount of exposure and the number of claims or payments supporting each level.
• Parameter burden: compare the number of estimated contrasts with the information available, including whether experience is concentrated in only a few levels.
• Stability: assess whether estimated effects change substantially across reasonable time periods or validation samples.
• Multiple comparisons: treat striking differences found after examining many levels cautiously; some may appear by chance.
• Defensibility: document why retained levels or pooled groups make sense for the classification and intended use.

Regularization or hierarchical modeling can shrink noisy level estimates toward a common pattern rather than forcing a complete choice between separate estimates and full pooling. These approaches require an explicit penalty or distributional assumption and suitable validation; they do not create information where the data are sparse. Likewise, an impressive training-fit statistic is not sufficient justification for a complicated factor. A practical evaluation should consider predictive performance on data not used to choose the grouping or tune the model.

The common exam trap is to equate a large number of policies with reliable evidence for every level. For frequency, the claim count also matters; for severity, the number and variability of claim payments matter. Another trap is to merge levels solely because their raw rates look similar, without considering uncertainty, exposure weights, or a defensible classification rationale. Use the response model to quantify support, and use validation and context to decide whether the resulting distinctions should remain.