Pairwise Comparisons After Kruskal-Wallis

Why Follow an Omnibus Test With Pairwise Comparisons?

The Kruskal-Wallis test assesses whether the rank distributions of several independent groups are alike. A significant result is evidence against the omnibus null hypothesis, but it does not say which groups account for the evidence. With four groups, for example, a single omnibus result does not distinguish a difference between groups 1 and 2 from one between groups 1 and 4.

Dunn’s test provides a set of pairwise comparisons using the ranks from the full, pooled data set. Each comparison measures how far apart two groups’ average pooled ranks are, relative to the standard error expected under their pairwise null hypothesis. Because there are several comparisons, their p-values need a multiplicity adjustment if the goal is to control the chance of making one or more false rejections across the set.

Definition: Dunn’s pairwise test compares two groups’ average ranks in the pooled sample. Its test statistic is the difference between those average ranks divided by a standard error that accounts for group sizes and ties. The resulting large-sample test is approximately standard normal under the pairwise null hypothesis.

The groups must be independent: each observation belongs to one group, and observations are not paired or repeated measurements. The outcome must be at least ordinal so that it can be ranked. The usual Dunn p-values rely on a large-sample normal approximation. As with the Kruskal-Wallis test, a rank difference is not automatically a difference in medians: interpreting it as a location difference requires the group distributions to have broadly similar shapes and spreads.

Calculating Dunn’s Test Statistic

Suppose there are \(k\) independent groups, with sizes \(n_1,\ldots,n_k\), and total sample size \(N\). Rank all \(N\) observations together, assigning average ranks to tied values. Let \(\bar{R}_i\) denote the average of the pooled ranks in group \(i\). For groups \(i\) and \(j\), Dunn’s statistic is

z_{ij} = \frac{\bar{R}_i-\bar{R}_j} {\sqrt{V\left(\frac{1}{n_i}+\frac{1}{n_j}\right)}}, \qquad V= \frac{N(N+1)}{12} - \frac{\sum_{\ell}(t_\ell^3-t_\ell)}{12(N-1)}.

Here \(t_\ell\) is the number of observations in the \(\ell\)th tied-value group; the sum is over all tied-value groups in the pooled sample. If there are no ties, the sum is zero and \(V=N(N+1)/12\). The usual two-sided p-value is \(2P(Z\geq |z_{ij}|)\), where \(Z\) has a standard normal distribution.

The tie term reduces the variance factor to reflect the fact that tied observations share average ranks rather than occupying distinct ranks. It is calculated from ties across all groups, because Dunn’s method uses the ranks pooled across the full data set.

Why This Standard Error Has This Form

Under a pairwise null hypothesis, the group labels do not systematically determine which ranks observations receive. Imagine assigning the pooled ranks at random to the group labels while keeping the group sizes fixed. The variance of the pooled ranks, adjusted for ties, leads to \(V\); comparing two group averages then multiplies that factor by \(1/n_i+1/n_j\).

To see the structure, let \(\sigma_R^2\) be the variance of the \(N\) pooled midranks, using divisor \(N\). For random assignment without replacement, the variance of the difference between the two group averages is

\operatorname{Var}(\bar{R}_i-\bar{R}_j) = \frac{N}{N-1}\sigma_R^2 \left(\frac{1}{n_i}+\frac{1}{n_j}\right).

For untied ranks, \(\sigma_R^2=(N^2-1)/12\), so \(N\sigma_R^2/(N-1)=N(N+1)/12\). With ties, the sum of squared deviations of the midranks is reduced by \(\sum_\ell(t_\ell^3-t_\ell)/12\). Therefore,

\frac{N}{N-1}\sigma_R^2 = \frac{N(N+1)}{12} - \frac{\sum_\ell(t_\ell^3-t_\ell)}{12(N-1)} = V.

Substituting this into the variance of the difference gives the squared standard error in Dunn’s statistic. The normal approximation is used to turn the standardized difference into a p-value; it is an approximation, not a claim that every finite-sample rank difference is exactly normal.

Worked Pairwise Comparisons and Multiplicity Adjustment

With \(k\) groups, there are \(m=k(k-1)/2\) distinct pairs. Testing each pair at level \(\alpha\) without adjustment raises the probability of at least one false rejection across the family. Bonferroni controls this familywise error rate by comparing each p-value with \(\alpha/m\). Holm’s method usually provides more power while also controlling the familywise error rate.

Procedure: For Holm adjustment, sort the \(m\) raw p-values from smallest to largest, \(p_{(1)}\leq\cdots\leq p_{(m)}\). Compare \(p_{(1)}\) with \(\alpha/m\), then \(p_{(2)}\) with \(\alpha/(m-1)\), continuing in order. Reject until the first comparison that fails; do not reject that comparison or any later one. Equivalently, the Holm-adjusted p-value at sorted position \(r\) is \(\max_{s\leq r}\{\min[1,(m-s+1)p_{(s)}]\}\).

Worked Example: One Pair From Untied Ranks

In a hypothetical analysis of three independent groups, each group has \(n=6\) observations. After pooling and ranking all 18 observations, the rank lists are: group A, 1, 2, 3, 7, 8, 9; group B, 4, 5, 6, 10, 15, 17; and group C, 11, 12, 13, 14, 16, 18. Suppose the Kruskal-Wallis test was significant. Calculate the unadjusted Dunn comparison between A and C.

There are no ties, so \(N=18\) and \(V=18(19)/12=28.5\). The mean ranks are \(\bar{R}_A=30/6=5\) and \(\bar{R}_C=84/6=14\). Thus,

SE_{AC} = \sqrt{28.5\left(\frac{1}{6}+\frac{1}{6}\right)} = \sqrt{9.5} = 3.082, \qquad z_{AC} = \frac{5-14}{3.082} = -2.920.

The two-sided normal p-value is approximately \(0.0035\). The negative sign indicates that group A’s average pooled rank is lower than group C’s; the two-sided p-value measures evidence of a difference in either direction. This is the unadjusted result for one pair, not yet a decision about a family of comparisons.

Worked Example: Ties and Holm Adjustment

Consider hypothetical observations from three independent groups: A has values 1, 2, 3, 4, 5; B has 3, 4, 5, 6, 7; and C has 8, 9, 10, 11, 12. The omnibus Kruskal-Wallis result is significant. Calculate Dunn comparisons and use Holm adjustment at \(\alpha=0.05\).

Pooling the 15 observations gives average ranks for tied values: 3 receives rank 3.5, 4 receives 5.5, and 5 receives 7.5. The group rank sums are 19.5, 35.5, and 65, so the mean ranks are \(3.9\), \(7.1\), and \(13.0\). There are three tied groups, each of size 2. Therefore,

V = \frac{15(16)}{12} - \frac{3(2^3-2)}{12(14)} = 20-\frac{18}{168} = 19.8929.

Each pair has standard error \(\sqrt{19.8929(1/5+1/5)}=2.821\). The absolute rank-mean differences are 3.2 for A versus B, 5.9 for B versus C, and 9.1 for A versus C. The resulting absolute statistics and approximate two-sided p-values are:

|z_{AB}|=\frac{3.2}{2.821}=1.134 \quad (p\approx0.257),\qquad |z_{BC}|=\frac{5.9}{2.821}=2.091 \quad (p\approx0.0365),\qquad |z_{AC}|=\frac{9.1}{2.821}=3.226 \quad (p\approx0.00126).

There are \(m=3\) comparisons. Sorted from smallest to largest, the p-values are \(0.00126,\ 0.0365,\ 0.257\). Holm first compares \(0.00126\) with \(0.05/3=0.0167\), so it rejects A versus C. Next, it compares \(0.0365\) with \(0.05/2=0.025\); this fails, so the procedure stops. At familywise level 0.05, only A and C show a statistically significant pairwise difference. Although the omnibus result was significant, it did not imply that every pair differs.

Worked Example: Why Adjustment Can Change a Conclusion

A hypothetical four-group analysis has eight observations in each group. The pooled ranks have mean ranks 9, 14, 19, and 24 for groups A through D, respectively; there are no ties. For illustration, calculate all six Dunn p-values and apply Holm adjustment.

Here \(N=32\), \(V=32(33)/12=88\), and every pair has standard error \(\sqrt{88(1/8+1/8)}=\sqrt{22}=4.690\). The absolute rank-mean differences are 5 for adjacent groups, 10 for groups A versus C and B versus D, and 15 for A versus D. Thus, the corresponding absolute \(z\)-statistics are \(1.066,\ 2.132,\) and \(3.198\), with approximate two-sided p-values \(0.286,\ 0.0330,\) and \(0.00138\), respectively. Across all pairs, the sorted p-values are \(0.00138,\ 0.0330,\ 0.0330,\ 0.286,\ 0.286,\ 0.286\).

Holm’s first threshold is \(0.05/6=0.00833\), so the A-versus-D comparison is rejected. The next threshold is \(0.05/5=0.0100\); the next p-value, \(0.0330\), fails it. Holm therefore stops: A versus C and B versus D are not rejected, despite their unadjusted p-values being below 0.05. The rank summaries describe the observed ordering, but the adjusted results are the appropriate basis for conclusions across all six comparisons.

Interpretation and Common Pitfalls

A significant Dunn comparison supports a difference in the groups’ average pooled ranks under the method’s assumptions. It does not, by itself, establish that their medians differ. If group distributions have substantially different spreads or shapes, rank differences can reflect more than a simple shift in location. Describe the outcome and the rank comparison accurately, and only make a median or location interpretation when the distributional conditions justify it.

A significant Kruskal-Wallis result is a common reason to proceed to post hoc comparisons, but the omnibus statistic does not identify a pair. Conversely, a non-significant omnibus result is not proof that all groups are identical. The key is to report what each test addresses and to account for the full family of pairwise comparisons being considered.

Pitfall: treating the omnibus result as a list of pairwise findings. A significant Kruskal-Wallis result says the omnibus null is inconsistent with the data; it does not name the differing groups. Use pairwise comparisons to examine the pairs.
Pitfall: testing every pair at 0.05 without adjustment. With many comparisons, this inflates the chance of at least one false rejection. Apply a stated multiplicity method, such as Holm.
Pitfall: using only the unadjusted p-value to decide. A raw value below 0.05 can fail Holm’s stricter sequential threshold, as in the four-group example.
Pitfall: calling every rank difference a median difference. Dunn compares pooled average ranks. A location or median interpretation needs suitably similar distribution shapes and spreads.

Key takeaway: Dunn’s test compares pooled average ranks pair by pair, using a standard error that includes a correction for ties. Adjust the set of pairwise p-values—Holm is one standard choice—and interpret rejections as evidence about ranks, not automatically about medians.

Check Your Understanding

Try these questions without looking back.

What does a significant Kruskal-Wallis result establish, and what does it leave unidentified?
Why does Dunn’s standard error use ranks pooled across all groups rather than ranks calculated separately within each pair?
In the tie correction, what does \(t_\ell\) count, and what happens to the correction when there are no ties?
Three groups produce six pairwise comparisons. Why can comparing all six raw p-values with 0.05 increase the chance of a false rejection?
A Dunn comparison is significant after Holm adjustment. What can you conclude about the groups’ ranks, and what additional caution applies before describing their medians?