Why Sparse Expected Counts Matter
A chi-square test of independence or homogeneity compares observed cell counts with the counts expected under a null hypothesis. The Pearson chi-square statistic can be calculated even when some expected counts are small. The concern is different: when counts are sparse, the statistic’s reference chi-square distribution may not approximate its actual null distribution well, so the resulting p-value can be inaccurate.
A small expected count is not evidence by itself that the null hypothesis is false, nor does it mean that the data are unusable. It is a warning to check whether the usual approximation is suitable and, if not, to consider a method that fits the data and question more appropriately.
For a cell in row \(i\) and column \(j\), the expected count under independence is \(E_{ij}=(\text{row total}_i)(\text{column total}_j)/n\). In a homogeneity test, the same calculation uses the group totals and the overall category totals. The chi-square statistic is \(X^2=\sum (O_{ij}-E_{ij})^2/E_{ij}\), where \(O_{ij}\) is the observed count. These formulas describe the discrepancy being measured; they do not, by themselves, guarantee that the chi-square reference distribution gives a reliable p-value.
What the Approximation Relies On
The Pearson chi-square test uses a large-sample approximation. Under the null hypothesis, differences between observed and expected counts, after standardization, behave approximately like a multivariate normal quantity when sample sizes are sufficiently large. The sum of their squared components is then approximated by a chi-square distribution.
This approximation becomes less dependable when expected counts are very small. A count cannot be negative and changes only in whole-number steps; with a small expected count, its distribution can be strongly discrete and asymmetric rather than well approximated by a normal distribution. A rare category may produce many zero counts, for example, even when the null hypothesis is true. As a result, the tail areas of the chi-square statistic may not match the actual probabilities of equally or more extreme data.
The degrees of freedom also reflect constraints in the table. Once the row and column totals are fixed, the cell deviations \(O_{ij}-E_{ij}\) must sum to zero within every row and column. Those constraints leave \((r-1)(c-1)\) independent deviations in an \(r\)-by-\(c\) table. The chi-square reference distribution with that many degrees of freedom follows from the large-sample approximation to those independent directions. Sparse counts can undermine the approximation even when the degrees of freedom have been calculated correctly.
Worked Example: Diagnosing a Sparse 3-by-2 Table
Suppose a hypothetical survey records two responses from people in three groups. The observed counts are:
| Group | Response 1 | Response 2 | Row total |
|---|---|---|---|
| A | 10 | 2 | 12 |
| B | 7 | 5 | 12 |
| C | 7 | 5 | 12 |
| Column total | 24 | 12 | 36 |
Under the null hypothesis of independence, the expected count for a cell is its row total multiplied by its column total, divided by the total sample size. For Group A and Response 1:
For Group A and Response 2, \(E_{A,2}=12(12)/36=4\). The row totals are all 12, so the expected counts in each row are 8 and 4. The resulting three-by-two table has expected counts of 8 and 4 in each row. Every expected count is at least 1, but only three of the six counts (the three counts of 8) are at least 5. Thus, 50% of the cells are below 5, which does not meet the common guideline that at least 80% be 5 or greater.
The table’s degrees of freedom are \((3-1)(2-1)=2\), but having the correct degrees of freedom does not repair sparse counts. The expected-count check raises a concern about using the ordinary chi-square approximation for the p-value. The next step is to consider whether a substantively justified category combination, a larger sample, or an appropriate exact method is available.
Choosing a Response to Sparse Counts
There is no single remedy for every sparse table. The choice should be driven by the study design and the question being asked, not by which option produces a smaller p-value. Three possibilities are particularly useful.
Combine categories only when the resulting groups make sense for the subject and answer a useful question. Recalculate the expected counts for the revised table.
If a larger sample is feasible, estimate how much it would increase the expected counts. A larger sample may help when the underlying category proportions are not extremely small.
For a sparse 2-by-2 table, Fisher’s exact test calculates probabilities under the null without relying on the large-sample chi-square approximation.
Combining Categories or Increasing the Sample
Combining categories changes the analysis. It is defensible when the categories represent substantively similar outcomes for the question at hand. It is not defensible merely because the observed table contains small counts or because combining categories makes a result statistically significant. Ideally, the combination is specified before examining the results. If categories are combined, report the revised definition and recognize that distinctions between the original categories are no longer tested.
Worked Example: Combining Rare Outcomes or Planning a Larger Sample
In a hypothetical study, two groups of 50 people each are classified as having no event, a mild event, or a severe event. The observed counts are:
| Group | No event | Mild event | Severe event | Total |
|---|---|---|---|---|
| A | 46 | 2 | 2 | 50 |
| B | 44 | 5 | 1 | 50 |
| Total | 90 | 7 | 3 | 100 |
Under the null hypothesis, the expected counts in each group are \(50(90)/100=45\) for no event, \(50(7)/100=3.5\) for mild event, and \(50(3)/100=1.5\) for severe event. The expected counts of 3.5 and 1.5 are sparse.
Suppose the scientifically relevant question is whether any event occurred, rather than whether it was mild or severe. Combining the two event categories then has a clear interpretation. The revised counts are 46 with no event and 4 with an event in Group A, and 44 with no event and 6 with an event in Group B. The column totals are 90 and 10, so the revised expected counts in each group are \(50(90)/100=45\) and \(50(10)/100=5\). All four expected counts are at least 5.
The revised analysis addresses whether the groups differ in the proportion with any event; it does not assess whether mild and severe events differ between groups. That loss of detail is acceptable only if the combined question is useful and scientifically justified.
Alternatively, suppose the three original outcome categories must remain distinct and the group sizes can be increased while keeping the same expected category proportions. In each group, the expected proportion with a severe event is \(3/100=0.03\). To make that expected count at least 5 requires about \(5/0.03=166.7\) people per group. With 167 per group, the expected counts are \(167(0.90)=150.3\), \(167(0.07)=11.69\), and \(167(0.03)=5.01\). This illustrates how a larger sample could address the sparse counts if the proportions remain similar; it does not guarantee that they will.
Fisher’s Exact Test for a Sparse 2-by-2 Table
Fisher’s exact test is a standard alternative for a 2-by-2 table when the chi-square approximation is questionable. Conditional on the observed row and column totals, it calculates the probability of each possible table under the null hypothesis of no association. With fixed margins, the count in one designated cell follows a hypergeometric distribution. The probability of a particular value \(x\) is
Here \(N\) is the grand total, \(K\) is the total in the selected column, and \(m\) is the total in the selected row. The possible values of \(x\) are those consistent with the fixed margins. A two-sided p-value sums probabilities of tables at least as incompatible with the null as the observed table. There are multiple conventions for defining “at least as incompatible” in a two-sided exact test, so the convention should be identified when reporting the result.
Worked Example: Applying Fisher’s Exact Test
Suppose a hypothetical randomized study assigns 10 participants to each of two groups and records whether each participant responds. The observed table is:
| Group | Response | No response | Total |
|---|---|---|---|
| A | 1 | 9 | 10 |
| B | 8 | 2 | 10 |
| Total | 9 | 11 | 20 |
Under independence, the expected response count in each group is \(10(9)/20=4.5\), and the expected no-response count is \(10(11)/20=5.5\). Two of the four expected counts are below 5, so Fisher’s exact test avoids relying on the usual chi-square approximation.
Condition on the observed margins, and let \(X\) be the number of responses in Group A. Under the null hypothesis, \(N=20\), \(K=9\), and \(m=10\), giving
For the observed value \(x=1\),
Using the two-sided convention that sums probabilities no greater than the observed table’s probability, the tables with \(x=0,1,8,\) or \(9\) are included. Their probability numerators are 11, 495, 495, and 11. Therefore,
Under this convention, the exact two-sided p-value is about 0.00548. If the null hypothesis of equal response proportions were true and the margins were fixed, tables at least this incompatible with the null would have probability about 0.55%. This is evidence of an association between group and response in this hypothetical study. The p-value alone does not establish the size or practical importance of the difference.
Common Pitfall and Practical Judgment
A common mistake is to treat an expected-count guideline as a switch: “Every expected count is at least 5, so the test is guaranteed valid,” or “One count is below 5, so no chi-square analysis is possible.” Neither statement is correct. The guidelines help assess a large-sample approximation; they are not universal guarantees or absolute prohibitions. The degree of sparsity, table size, sampling design, and available alternatives all matter.
Another mistake is to combine categories after seeing which combination gives the most favorable p-value. That makes the test depend on the observed result and can distort its error rate. Instead, combine categories only when the new grouping has a clear subject-matter rationale, explain how it changes the question, and report the choice transparently. If distinct categories must be retained, consider whether more observations are feasible or whether an exact method is suitable for the table and design.
Check Your Understanding
Use the definitions and decision principles above to answer these questions.
- A 2-by-3 table has six expected counts, two of which are below 5 and none of which is below 1. Does it meet the common guideline that at least 80% of expected counts be 5 or greater?
- Why can the chi-square approximation be unreliable when expected counts are very small?
- When might combining two outcome categories be scientifically defensible, and what information is lost by doing so?
- In Fisher’s exact test, what is held fixed when the hypergeometric probabilities are calculated?
- A researcher plans a larger sample because one expected count is small. What assumption about the category proportions is needed for a calculation of the required sample size to be informative?