Combining Sparse Categories Correctly

Why Sparse Categories Need Careful Pooling

A Pearson chi-square goodness-of-fit test compares observed category counts with the counts expected under a specified null model. When a category has a very small expected count, the chi-square approximation used to assess the statistic may be unreliable. Combining sparse categories can address this issue, but the grouping must preserve the null probabilities and must not be chosen after inspecting the data in a way that favors a desired result.

Pooling is not a matter of deleting a small category or shifting its probability elsewhere. For a valid grouping, combine neighboring categories in the model’s natural order, add their null probabilities, and add their observed counts. The grouped categories must still cover all possible outcomes, with total null probability one.

Definition: Suppose a sample of \(n\) independent observations falls into \(k\) mutually exclusive, exhaustive categories. Under the null hypothesis, category \(j\) has probability \(p_j\), where \(\sum_{j=1}^k p_j=1\). Its expected count is \(E_j=np_j\). If adjacent categories \(a,\ldots,b\) are pooled, the new category has probability \(p^*=\sum_{j=a}^b p_j\), expected count \(E^*=np^*=\sum_{j=a}^b E_j\), and observed count \(O^*=\sum_{j=a}^b O_j\).

The common practical target of expected counts at least 5 is a useful convention, not a universal mathematical boundary. The appropriate guidance can depend on the testing context. In the examples here, use the explicit rule “pool until every final expected count is at least 5.” A count exactly equal to 5 meets that rule and does not, by itself, require pooling.

What Pooling Changes in the Test

For the final set of \(g\) categories, the Pearson statistic is calculated from the grouped observed and expected counts:

X^2=\sum_{r=1}^{g}\frac{(O_r-E_r)^2}{E_r}.

Here each \(O_r\) and \(E_r\) refers to a final category after any pooling. Do not calculate contributions for the original categories and then keep those contributions as though the categories had not been combined. Pooling changes the terms in the statistic and the number of categories used in the test.

Under the usual large-sample conditions, when the null probabilities are fully specified and the final grouping is fixed independently of the observed counts, the reference degrees of freedom are \(g-1\). When parameters are estimated from the data, the degrees of freedom require the corresponding adjustment for the fitting procedure. Pooling does not remove that issue; the final grouped categories and the parameter estimation must both be accounted for.

Key principle: Choose pooling from the null probabilities and sample size, using a stated rule, before using the observed counts to assess fit. Combine adjacent categories, preserve their total probability, and compute the statistic from the resulting grouped counts.

Why Grouped Probabilities and Counts Add

For each observation, define an indicator that equals 1 when it falls in a particular category and 0 otherwise. The count in that category is the sum of those indicators over the \(n\) observations, so its expectation is \(np_j\). For a pooled category, an observation is in the pool if it falls in any of the constituent categories. Those categories are mutually exclusive, so the pooled probability is their sum. Consequently, the expected pooled count is the sum of their expected counts.

p^*=\sum_{j=a}^b p_j \quad\Longrightarrow\quad E^*=np^* =n\sum_{j=a}^b p_j =\sum_{j=a}^b np_j =\sum_{j=a}^b E_j.

The observed pooled count is also the sum of the original counts: every observation in the pooled range is counted once, and observations outside it are not counted. This is why pooling preserves the total sample size and the null probability, rather than changing the underlying null model.

Pooling does, however, discard information about how counts are distributed within the pool. For example, a pooled total can match its expected value even if one constituent category is above expectation and another is below. The test after pooling assesses fit at the resolution of the final categories, not at the finer resolution that was removed.

Worked Pooling Procedures

Worked Example: Pooling a Sparse Lower-Tail Pair

A hypothetical severity study has \(n=50\) independent observations assigned to five ordered loss bands. Under a fully specified null model, the probabilities are \(0.02, 0.08, 0.20, 0.30,\) and \(0.40\). The observed counts are \(0, 5, 11, 14,\) and \(20\). Using the rule that every final expected count must be at least 5, determine the pooling and calculate the Pearson statistic. Assume the grouping is selected from the null expected counts before assessing the observed fit.

Method and expected counts. Multiply each null probability by \(n=50\):

$$ (E_1,E_2,E_3,E_4,E_5) =50(0.02,0.08,0.20,0.30,0.40) =(1,4,10,15,20). $$

The first two adjacent categories have expected counts below 5. Pool them: their combined probability is \(0.02+0.08=0.10\), so their combined expected count is \(50(0.10)=5\). The remaining expected counts are already at least 5. The final expected counts are therefore \(5,10,15,20\).

Grouped observed counts. Add the observed counts in the same first two categories: \(0+5=5\). The final observed counts are \(5,11,14,20\). Both sets total 50.

Pearson statistic. Calculate one contribution for each final group:

X^2 =\frac{(5-5)^2}{5} +\frac{(11-10)^2}{10} +\frac{(14-15)^2}{15} +\frac{(20-20)^2}{20} =0+0.1000+0.0667+0 =0.1667.

The result is \(X^2\approx0.1667\). As a check, the pooled probability is \(0.10\), the pooled expectation is 5, and the final observed counts still sum to 50. The common mistake avoided here is treating the original first two cells as separate terms in the final statistic after pooling them.

Worked Example: An Expected Count Exactly at the Target

A hypothetical portfolio has \(n=100\) losses classified into six ordered bands. The specified null probabilities are \(0.10, 0.20, 0.30, 0.25, 0.10,\) and \(0.05\). The observed counts are \(8,22,29,26,9,\) and \(6\). Apply the “at least 5” rule and calculate the Pearson statistic. Treat the probabilities as fully specified and the categories as fixed before observing the counts.

Expected counts and grouping decision. Multiplication by 100 gives expected counts \(10,20,30,25,10,\) and \(5\). Each is at least 5. In particular, the final category’s expected count is exactly 5, so the stated rule does not require pooling.

Statistic. Keep the six categories separate and compute each contribution:

\begin{aligned} X^2 &=\frac{(8-10)^2}{10} +\frac{(22-20)^2}{20} +\frac{(29-30)^2}{30} +\frac{(26-25)^2}{25} +\frac{(9-10)^2}{10} +\frac{(6-5)^2}{5}\\ &=0.4000+0.2000+0.0333+0.0400+0.1000+0.2000\\ &=0.9733. \end{aligned}

The observed counts sum to \(100\), as do the expected counts, and the statistic is approximately \(0.9733\). The common mistake is to call the final category sparse because it is small or at the boundary, then pool it automatically. Under the rule in this problem, “at least 5” includes 5; no pooling is required.

Worked Example: Pooling More Than Two Adjacent Categories

A hypothetical loss sample contains \(n=120\) observations in seven ordered severity bands. The null probabilities are \(0.01,0.03,0.06,0.15,0.25,0.30,\) and \(0.20\). The observed counts are \(1,3,9,16,32,35,\) and \(24\). Pool from the lower tail inward until each final expected count is at least 5, then calculate the grouped Pearson statistic. Assume this pooling rule was set using the null model and sample size, not selected to improve the observed fit.

Expected counts. Multiplying each probability by 120 gives \((1.2,3.6,7.2,18,30,36,24)\). Pool the first two categories first: \(1.2+3.6=4.8\), which is still below 5. Add the next adjacent category, giving \(4.8+7.2=12\). All remaining expected counts are at least 5, so the final groups are categories 1–3, then categories 4, 5, 6, and 7 separately.

Grouped counts. The first group has observed count \(1+3+9=13\). The final observed counts are therefore \(13,16,32,35,24\), with expected counts \(12,18,30,36,24\). Both totals equal 120.

Statistic and check. Applying the Pearson formula to the five final groups:

\begin{aligned} X^2 &=\frac{(13-12)^2}{12} +\frac{(16-18)^2}{18} +\frac{(32-30)^2}{30} +\frac{(35-36)^2}{36} +\frac{(24-24)^2}{24}\\ &=\frac{1}{12}+\frac{4}{18}+\frac{4}{30}+\frac{1}{36}+0\\ &=0.0833+0.2222+0.1333+0.0278 =0.4667. \end{aligned}

Thus the grouped statistic is approximately \(0.4667\). The first two expected counts alone sum to \(4.8\), so stopping after pooling only those two would fail the stated target; the third adjacent category is needed. The check also confirms that the final pooled expected count is \(12\), and that no probability or observation has been omitted.

Pre-Specification, Adjacency, and Test Calibration

The reason for using expected counts to guide pooling is that those counts are determined by the null probabilities and \(n\), not by the particular random sample observed. If a grouping plan is chosen after looking at the observed counts—for example, repeatedly merging categories until the observed departures appear small—the resulting statistic is selected using the same data that it is meant to test. Its distribution is generally not the ordinary chi-square reference distribution for the final number of groups. The nominal p-value can then be misleading.

If observed counts have already been examined and the analyst wants to choose among several possible groupings, the choice cannot simply be treated as though it had been fixed in advance. The selection procedure must be reflected in the calibration of the test, or the grouping should be justified independently of the observed fit. The standard Pearson calculation with the usual reference degrees of freedom assumes the final categories have been set independently of the data used to calculate the statistic.

Adjacency matters when categories have a natural order, such as consecutive loss intervals. Pooling categories 1 and 4 while leaving categories 2 and 3 separate creates a disconnected group and usually has no coherent interval interpretation. Likewise, a sparse lower-tail category should generally be combined inward with its nearest neighbor, not with a distant upper-tail category. If both tails need pooling, pool each tail inward according to a stated rule while keeping the intervening categories in order.

Pooling is not automatically required just because it is possible. If all expected counts meet the chosen criterion, retaining the original categories preserves more information. Conversely, if pooling is necessary, report the final category boundaries or group definitions, the grouped expected counts, and the grouped observed counts. This makes the calculation auditable and clarifies exactly what departures the test can detect.

Common Mistakes

Using observed counts to decide which cells are sparse. Sparsity for this procedure is assessed from expected counts under the null model, not from whether an observed count happens to be small.
Pooling a category with expected count exactly 5 under an “at least 5” rule. It meets that rule. Do not replace “at least 5” with “greater than 5.”
Combining probabilities but not counts, or counts but not probabilities. A pooled category must use the sum of both its null probabilities and its observed counts.
Adding the original Pearson contributions after pooling. Recalculate the contribution from the pooled observed and expected totals; the original contributions do not remain separate.
Combining nonadjacent categories without a defensible grouping rule. Preserve the natural order and keep the final groups mutually exclusive and exhaustive.
Ignoring the effect of fitting parameters. Pooling determines the final number of categories, but parameter estimation can also affect the reference degrees of freedom.

Key takeaway: Pool adjacent categories when their null expected counts require it under the stated criterion. Add their probabilities, expected counts, and observed counts; then calculate the Pearson statistic from the final groups. Do not select groups after inspecting the observed fit and assume the ordinary chi-square calibration still applies.

Check Your Understanding

For each problem, use the stated “at least 5” expected-count target and assume the null probabilities and grouping rule are set independently of the observed counts.

A sample of 75 observations has ordered null probabilities \(0.04,0.16,0.30,\) and \(0.50\). Which categories must be pooled, and what are the final expected counts? (Answer: Pool the first two; the final expected counts are \(15,22.5,37.5\).)
For \(n=100\), five ordered categories have probabilities \(0.20,0.25,0.30,0.20,\) and \(0.05\). Which categories need pooling under the stated target? (Answer: None; the expected counts are \(20,25,30,20,5\), all at least 5.)
For \(n=80\), ordered probabilities are \(0.025,0.075,0.20,0.30,\) and \(0.40\), and the observed counts are \(1,7,15,25,\) and \(32\). Pool as needed and calculate the grouped Pearson statistic. (Answer: Pool the first two categories; the grouped counts are \(8,15,25,32\), the expected counts are \(8,16,24,32\), and \(X^2\approx0.1042\).)
For \(n=60\), four ordered categories have probabilities \(0.10,0.20,0.30,\) and \(0.40\), and observed counts \(4,14,18,\) and \(24\). Is pooling required, and what is the unpooled Pearson statistic? (Answer: Pooling is not required; expected counts are \(6,12,18,24\), and \(X^2=1.0000\).)
For \(n=200\), seven ordered categories have probabilities \(0.01,0.04,0.15,0.30,0.25,0.20,\) and \(0.05\). Pool the lower tail inward as needed. What is the first final group’s probability and expected count, and are further groups needed? (Answer: Pool the first two categories; their probability is \(0.05\) and expected count is \(10\). Every remaining category also has expected count at least 5, so no further pooling is needed.)

What You'll Learn