Calibrating Kruskal–Wallis With Ties and Small Samples

Why Calibration Matters When Ranks Are Tied

The Kruskal–Wallis test compares the rank distributions of several independent groups. Its statistic is often calibrated using a chi-square distribution, but that reference distribution is an approximation. With small samples, the actual statistic can have a discrete distribution that differs noticeably from the chi-square curve. Ties also change the variability of the ranks, so the unadjusted statistic needs a correction before it is compared with the usual chi-square reference distribution.

A tie occurs when two or more observations have the same value. Assign each tied observation the average of the ranks it would have occupied. This is called a midrank. The tie correction adjusts the Kruskal–Wallis statistic for the reduction in rank variability caused by tied values. It does not, however, make the chi-square approximation exact. In a small sample, exact or permutation calibration can provide a more suitable reference distribution when its exchangeability conditions hold.

Model, Assumptions, and the Two Calibrations

The Kruskal–Wallis procedure applies to a quantitative or ordinal response measured on independent observations in \(k\) groups. Under its usual null hypothesis, the groups have the same response distribution. The alternative is that at least one group distribution differs. The method does not require normally distributed responses, but it does require independent observations; in a randomized study, each observation must belong to just one group. For a location or median interpretation, the group distributions should also have a comparable shape and spread. If their shapes differ, a rejection may reflect differences other than location.

The familiar large-sample calibration compares \(H_{\mathrm{adj}}\) with a chi-square distribution with \(df=k-1\). The resulting right-tail probability is an approximate p-value. The tie correction accounts for the tied ranks, but small sample size and discreteness can still make this approximation poor. There is no single sample-size cutoff that guarantees good calibration in every design.

A permutation calibration instead holds the observed response values fixed and reallocates group labels while preserving the group sizes. Under the null hypothesis of identical group distributions, the labels are exchangeable: each allocation consistent with the design is equally plausible under the null. For a randomized experiment, this condition follows from the random assignment mechanism under the corresponding sharp null. For observational groups, it requires a defensible exchangeability assumption; it does not follow merely from choosing to permute the data.

For an exact permutation test, calculate the statistic for every permitted allocation. The p-value is the proportion of these allocations with a statistic at least as large as the observed statistic. “At least as large” includes equality, which matters because rank statistics can take only a limited set of values in small samples. If all allocations are enumerated under the stated exchangeability condition, this is an exact conditional p-value. A random subset of allocations gives a Monte Carlo approximation instead, not an exact enumeration.

Why the Tie Correction Divides by \(C\)

The tie correction can be understood as an adjustment for the rank variability lost when a block of observations is assigned a shared midrank. With no ties, the pooled ranks are all distinct and the correction factor is 1. A tie block of size \(t\) contributes \(t^3-t\) to the amount removed from the untied rank variation. Summing across blocks gives the factor \(C\), which is between 0 and 1 when ties are present.

The uncorrected statistic \(H\) is scaled as though the pooled ranks had their untied variability. Ties reduce that variability by the factor \(C\). To put the statistic back on the usual scale, divide by that factor:

Thus, when \(C<1\), \(H_{\mathrm{adj}}\) is larger than \(H\). Under the usual large-sample reference, the adjusted statistic is compared with \(\chi^2_{k-1}\), not with a chi-square distribution having degrees of freedom chosen according to the number of ties. In a permutation calculation, the pooled tie pattern stays fixed across reallocations, so \(C\) is the same for every allocation. Dividing each statistic by this same positive constant preserves their ordering and therefore does not change the exact permutation tail proportion.

Exact Calibration in a Small Sample With Ties

The following example makes the discreteness visible. Suppose three hypothetical groups of size two contain the values A: \(1,4\); B: \(2,2\); and C: \(3,3\). Assume the six observations are independent, the group sizes are fixed, and under the null the group labels are exchangeable. The tied values receive midranks \(1, 2.5, 2.5, 4.5, 4.5, 6\). The observed rank sums are \(7,5,9\), respectively.

Small Samples Without Ties Can Also Show a Difference

Ties are not the only reason to consider exact calibration. Even without ties, a small-sample permutation distribution may be coarse. The next example uses the same group sizes but six distinct response values.

Choosing and Reporting a Calibration

A practical analysis should make the calibration choice explicit. First, calculate pooled midranks and inspect the tie pattern. Then calculate \(C\) and \(H_{\mathrm{adj}}\). If using the standard large-sample method, report that the p-value comes from a chi-square approximation with \(k-1\) degrees of freedom. If small sample size or a coarse rank distribution makes that approximation questionable, consider exact enumeration or a permutation procedure, provided exchangeability is justified by the design and null hypothesis.

For an exact or Monte Carlo permutation procedure, preserve the original group sizes and follow any restrictions imposed by the design. In each permitted allocation, recompute the statistic using the pooled ranks and the same tie adjustment. For an exact test, count statistics at least as extreme as the observed value and divide by the total number of permitted allocations. For Monte Carlo sampling, estimate that proportion from the sampled allocations and describe the result as an estimate, not an exact p-value.

The p-value is evidence about how unusual the observed statistic is under the stated null and calibration scheme. It is not the probability that the null hypothesis is true. A rejection indicates evidence that the group distributions are not all the same; by itself, it does not identify which groups differ or establish that their medians differ. Those conclusions require additional, appropriately calibrated analysis.

Common Mistakes

• Using ordinary ranks for tied observations. Giving tied values separate arbitrary ranks changes the rank sums. Use midranks, then calculate the tie correction from the sizes of the pooled tie blocks.
• Forgetting to divide by \(C\). The tie-adjusted statistic is \(H/C\), not \(H\times C\). Since \(C<1\) when ties occur, the adjusted statistic is larger.
• Assuming tie correction guarantees an accurate chi-square p-value. It corrects for ties, not for all small-sample discreteness. A permutation calibration can be useful when its exchangeability conditions hold.
• Permuting labels without justification. A permutation test is exact only when the permitted allocations are equally plausible under the null. In an observational study, that condition must be justified; it is not automatic.
• Counting only statistics strictly greater than the observed one. The right-tail p-value includes equality: count allocations with a statistic greater than or equal to the observed statistic.
• Calling every rejection a median difference. Kruskal–Wallis concerns group distributions. A median interpretation requires additional assumptions, such as comparable distributional shapes and spreads.