Why a Significant Table Test Leads to More Questions
A chi-square test of independence or homogeneity evaluates an overall question: is there evidence that the distributions differ, or that the two categorical variables are associated? A significant result does not say which groups or cells account for the pattern. Follow-up comparisons can help locate that pattern, but searching many comparisons and reporting only the smallest p-value makes chance fluctuations look more convincing than they are.
The solution is to define the comparisons of interest and account for how many are tested. This does not change the observed table or the omnibus statistic. It changes how cautiously we evaluate the follow-up evidence.
For example, after comparing three groups on a binary outcome, one natural family is all three pairwise comparisons of their outcome proportions. Alternatively, the scientific question may call for a few pre-specified contrasts, or for cell-level follow-ups. The family should reflect the claims the analysis may make, not just the tests that happen to look interesting after the data are inspected.
Plan the Comparisons Before Testing Them
Start by stating which parameter or pattern each follow-up addresses. In a table comparing several groups on a binary outcome, a pairwise comparison asks whether two groups have the same probability of the outcome. For a table with \(k\) groups, testing every pair creates \(m=k(k-1)/2\) comparisons. A search over more groups or more outcomes can create a much larger family.
Cell-level follow-ups ask whether particular observed counts depart from what would be expected under independence or homogeneity. Residuals can indicate the direction and relative size of a cell’s departure, but testing many cells separately also creates multiplicity. Earlier material on residual patterns helps identify which cells merit attention; it does not make unadjusted cell-by-cell p-values automatically safe.
A planned comparison is specified before inspecting the results. An exploratory comparison is chosen after seeing the data. Either can be useful, but exploratory searches especially need a clear account of the full family considered. If many outcomes, subsets, or alternative contrasts were searched, counting only the reported tests understates the search.
Bonferroni and Holm Adjustments
The Bonferroni method controls the FWER by testing each of \(m\) hypotheses at level \(\alpha/m\). Equivalently, for a raw p-value \(p_i\), its Bonferroni-adjusted p-value is \(\min(mp_i,1)\). Compare that adjusted value with \(\alpha\). This method is valid even when the tests are dependent, though it can be conservative.
The justification uses the union bound. The event of at least one false rejection is the union of the false-rejection events for the individual tests. The probability of a union cannot exceed the sum of the event probabilities. If each true null is tested at level \(\alpha/m\), then
Holm’s step-down method also controls the FWER and is often less conservative. Sort the \(m\) raw p-values from smallest to largest, \(p_{(1)}\leq\cdots\leq p_{(m)}\). Compare the smallest with \(\alpha/m\). If it passes, reject that null and compare the next p-value with \(\alpha/(m-1)\); continue, reducing the denominator by one each time. Stop at the first comparison that does not pass, and do not reject that hypothesis or any remaining hypotheses.
Worked Example: Pairwise Follow-Ups After a Significant Test
Worked Example: Pairwise Follow-Ups After a Significant Test
Suppose a hypothetical survey records whether respondents in each of three independently sampled groups support a proposal. Each group has 120 respondents. The counts are 60 supporters in Group 1, 40 in Group 2, and 30 in Group 3. The remaining respondents do not support it. First consider the omnibus test, then compare all three pairs of groups.
Check the setting. The data are counts in mutually exclusive categories, and the hypothetical design samples respondents independently across groups. The expected counts under a common support proportion are all at least 43, and the expected non-support counts are all at least 76, so the usual large-sample chi-square approximation is reasonable for this illustration.
The overall support total is \(130\) out of \(360\), so the expected support count in each group under the null is \(120(130/360)=43.333\); the expected non-support count is \(76.667\). The observed-minus-expected support deviations are \(16.667\), \(-3.333\), and \(-13.333\). The non-support deviations are their negatives. Thus
The degrees of freedom are \((3-1)(2-1)=2\). The right-tail p-value is approximately \(0.000219\), so the omnibus test gives strong evidence that support proportions are not all equal. It does not by itself identify the differing pairs.
There are \(m=3(3-1)/2=3\) pairwise comparisons. For each pair, use a two-sided test of equal proportions. Under the pairwise null, the pooled proportion estimates the common proportion, and the standard error is the square root of that pooled proportion times its complement times the sum of the reciprocal sample sizes. The independent-group and large-sample conditions are reasonable here; the pooled expected counts for each pair are also ample.
For Groups 1 and 2, the sample proportions are \(60/120=0.5000\) and \(40/120=0.3333\). The pooled estimate is \(100/240=0.4167\), giving
The two-sided raw p-value is about \(0.0088\). For Groups 2 and 3, the pooled estimate is \(70/240=0.2917\):
The two-sided raw p-value is about \(0.156\). Finally, for Groups 1 and 3, the pooled estimate is \(90/240=0.3750\):
The raw two-sided p-value is approximately \(0.00006334\). With Bonferroni adjustment, multiply each raw p-value by three. The adjusted p-values are approximately \(0.0265\) for Groups 1 and 2, \(0.467\) for Groups 2 and 3, and \(0.000190\) for Groups 1 and 3. At family-wise level \(0.05\), the data support differences between Groups 1 and 2 and between Groups 1 and 3, but not between Groups 2 and 3. This does not establish why the proportions differ or imply that group membership caused the differences.
Worked Example: Applying Holm to the Same Comparison Family
Worked Example: Applying Holm to the Same Comparison Family
Use the three raw p-values from the previous example: approximately \(0.0088\), \(0.156\), and \(0.00006334\). To apply Holm’s procedure at \(\alpha=0.05\), sort them: \(0.00006334\), \(0.0088\), \(0.156\). The successive thresholds are \(0.05/3=0.0167\), \(0.05/2=0.0250\), and \(0.05\).
The smallest p-value, \(0.00006334\), is below \(0.0167\), so reject its null. The next, \(0.0088\), is below \(0.0250\), so reject that null as well. The last, \(0.156\), exceeds \(0.05\). Stop there: the evidence does not support rejecting the remaining hypothesis. Holm reaches the same pairwise conclusions as Bonferroni in this example, but the methods need not give identical decisions in every data set.
Worked Example: Adjusting Cell-Level Follow-Ups
Worked Example: Adjusting Cell-Level Follow-Ups
Return to the three-group support table. Suppose the analysis examines all six cells (three groups by two responses) using standardized residuals, with a two-sided approximate normal test for each cell. The null model gives Group 1’s expected support count as \(43.333\). Its row proportion is \(120/360=1/3\), and the support-column proportion is \(130/360=0.3611\). For the adjusted standardized residual, use the approximate standard error \(\sqrt{E(1-r)(1-c)}\), where \(E\) is the expected count, \(r\) the row proportion, and \(c\) the column proportion.
The unadjusted two-sided normal p-value is about \(0.000105\). If all six cells belong to the family, Bonferroni gives an adjusted p-value of about \(6(0.000105)=0.000630\), still below \(0.05\). This identifies Group 1’s support cell as unusually high relative to the independence model under this approximate cell-level procedure. The cell tests are dependent because the table margins constrain the counts; Bonferroni remains valid under dependence. The residual’s direction and the table’s broader pattern still matter when describing the result.
Common Pitfall: Searching First, Adjusting Later
A common mistake is to examine every cell or pair, find the smallest raw p-value, and report it as though it were the only test. The wrong reasoning is: “This comparison has \(p=0.04\), so it is significant at \(0.05\).” If that result was selected from, for example, ten comparisons, the chance of at least one false positive can be substantially larger than five percent.
The correction is to define the family and use an appropriate adjustment. For ten tests, Bonferroni compares each p-value with \(0.05/10=0.005\), or compares each Bonferroni-adjusted p-value with \(0.05\). Alternatively, Holm can be used on the sorted p-values. If the search included additional outcomes or subsets, those searches also matter when defining the family of claims.
Multiplicity control does not make a weak design strong, guarantee that every important pattern will be detected, or turn association into causation. It addresses one specific issue: the increased chance of one or more false rejections when several hypotheses are tested.
Check Your Understanding
Answer these questions using the definitions and procedures in this tutorial.
- A study compares four groups on a binary outcome and tests every pair. How many comparisons are in the family, and what per-test level does Bonferroni use when \(\alpha=0.05\)?
- Explain why a significant omnibus chi-square result does not establish that every pair of groups differs.
- Four follow-up tests have sorted p-values \(0.004\), \(0.018\), \(0.032\), and \(0.20\). Apply Holm’s procedure at \(\alpha=0.05\), showing each threshold and where the procedure stops.
- Why can searching many cells and reporting only the smallest unadjusted p-value exaggerate the evidence?
- In a cell-level analysis, why does dependence among the cell tests not invalidate Bonferroni adjustment?