Why Use an Exact Rank-Sum Test?
The rank-sum test compares two independent samples using the order of their pooled observations. For small samples, a normal approximation can be inaccurate or unnecessary: under the null hypothesis, every allocation of the pooled ranks to the first sample is equally likely. Counting those allocations gives an exact reference distribution and an exact one-sided p-value.
The calculation is directional. If unusually large observations in sample 1 support the alternative, evidence is in the upper tail of its rank sum. If unusually small observations support the alternative, evidence is in the lower tail. The tail must be chosen before interpreting the observed statistic.
Null Model and Exact Tail Probabilities
For the standard exact calculation here, assume the two samples are independent random samples from the same continuous distribution under the null hypothesis. Continuity means ties have probability zero. Conditional on the pooled ordered observations, each selection of \(n_1\) of the \(N\) ranks for sample 1 is equally likely. There are \(\binom{N}{n_1}\) such selections.
The exact null distribution is obtained by counting how many rank selections have each possible sum. Write \(c(r)\) for the number of size-\(n_1\) subsets of \(\{1,\ldots,N\}\) whose ranks sum to \(r\). Then the probability of a particular rank sum is \(c(r)/\binom{N}{n_1}\). For an upper-tail alternative, the p-value includes the observed sum and all larger sums; for a lower-tail alternative, it includes the observed sum and all smaller sums.
The equality in the tail definition matters: outcomes as extreme as the observed result include the observed rank sum itself. An exact p-value is a probability under the null, not the probability that the null hypothesis is true.
Why Every Allocation Has Equal Probability
Under the null, all \(N\) observations come from the same continuous distribution and have no ties. By exchangeability, before observing which sample label is attached to which pooled order position, any \(n_1\) of the \(N\) positions are equally likely to carry the sample 1 label. There are \(\binom{N}{n_1}\) possible label allocations, so each has probability \(1/\binom{N}{n_1}\). The event \(R_1=r\) consists of exactly \(c(r)\) of those allocations. Adding their equal probabilities gives the formula above.
This argument does not require knowing the common distribution's shape. It does require independent samples and the common-distribution null. If ties occur, the no-tie rank allocation calculation must be adjusted; the simple enumeration below is for untied ranks.
Use ranks 1 through \(N\), and record the ranks belonging to sample 1.
Use the upper tail when large values in sample 1 support the alternative; use the lower tail when small values do.
List or count every size-\(n_1\) subset of the ranks and calculate its sum.
Divide the number of allocations in the inclusive tail by \(\binom{N}{n_1}\), then compare the p-value with the stated significance level.
Worked Example: An Upper-Tail Test With Three Observations per Sample
A hypothetical laboratory compares the measured recovery times for two independent procedures. Each procedure is applied to three randomly selected subjects; assume the samples are independent, the measurements have no ties, and under the null hypothesis the two procedures have the same continuous recovery-time distribution. The three sample 1 observations have pooled ranks 2, 5, and 6. Test the one-sided alternative that sample 1 tends to have larger recovery times. Find the exact p-value and the nonrandomized rejection region at \(\alpha=0.10\).
Method and direction. Larger recovery times have larger ranks, so the alternative calls for an upper-tail test. The observed rank sum is \(R_1=2+5+6=13\).
Enumerate the null distribution. There are \(\binom{6}{3}=20\) equally likely allocations. The possible sums and their counts are shown below. For example, sum 8 occurs for \(\{1,2,5\}\) and \(\{1,3,4\}\).
| Rank sum \(r\) | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |
|---|---|---|---|---|---|---|---|---|---|---|
| Count \(c(r)\) | 1 | 1 | 2 | 3 | 3 | 3 | 3 | 2 | 1 | 1 |
The counts sum to \(1+1+2+3+3+3+3+2+1+1=20\), as required. The upper-tail count at 13 is \(2+1+1=4\), from sums 13, 14, and 15. Therefore \(p_{\text{upper}}=4/20=0.2000\). Since \(0.2000>0.10\), the result is not significant at the 10% level.
For the rejection region, a rank sum of at least 14 has upper-tail probability \((1+1)/20=0.10\). Including 13 would give \(4/20=0.20\), which exceeds the level. Thus the nonrandomized rejection region is \(R_1\ge14\), with null rejection probability \(0.10\). The observed sum of 13 is outside it. Common mistake avoided: using the number of ranks above the observed sum but excluding 13 would omit outcomes as extreme as the observation.
Exact Rejection Regions Are Discrete
For a chosen significance level \(\alpha\), a nonrandomized upper-tail test rejects for sufficiently large \(R_1\), selecting a cutoff whose null upper-tail probability is at most \(\alpha\). The cutoff is usually a possible rank sum. Because the null distribution has only finitely many possible sums, its rejection probability may be strictly less than \(\alpha\). The lower-tail rule is defined analogously. A rule with rejection probability no greater than \(\alpha\) controls the test size; it need not attain \(\alpha\) exactly.
When comparing cutoffs, check the next less extreme possible cutoff as well. If including that rank sum would make the tail probability exceed \(\alpha\), it must not be included in a nonrandomized rejection region.
Worked Example: A Lower-Tail P-Value With Unequal Sample Sizes
In a hypothetical comparison of two independent calibration methods, sample 1 has two measurements and sample 2 has four. Assume independent sampling, a common continuous distribution under the null, and no ties. The pooled ranks assigned to sample 1 are 1 and 4, so \(R_1=1+4=5\). The alternative is that sample 1 tends to produce smaller measurements. Calculate the exact p-value and decide at \(\alpha=0.10\).
Method and direction. Smaller measurements correspond to smaller ranks, so use the lower tail. There are \(\binom{6}{2}=15\) equally likely pairs of ranks for sample 1. Counting all pairs gives the following distribution: sums 3 through 11 have counts \(1,1,2,2,3,2,2,1,1\), respectively. The counts total 15.
The observed sum is 5, so the inclusive lower tail contains the allocations with sums 3, 4, and 5. Its count is \(1+1+2=4\). Hence \(p_{\text{lower}}=4/15=0.2667\). Since \(0.2667>0.10\), there is insufficient evidence to reject the null at the stated level.
The lower-tail rejection region at \(\alpha=0.10\) is \(R_1\le4\): its probability is \((1+1)/15=0.1333\), which is already greater than 0.10. In fact, the only possible sum below 4 is 3, whose probability is \(1/15=0.0667\); thus the nonrandomized region is \(R_1\le3\), with size 0.0667. The observed sum of 5 is not in it. Common mistake avoided: choosing the upper tail merely because the statistic is called a rank sum would test the wrong direction.
Worked Example: Finding a Small-Sample Upper-Tail Critical Region
A hypothetical materials study compares three independently sampled specimens from process A with four from process B. Assume independent samples, no tied measurements, and identical continuous distributions under the null. The process A observations receive pooled ranks 5, 6, and 7. Test whether process A tends to produce larger measurements, using a nonrandomized test with \(\alpha=0.10\). Find the exact p-value and the rejection region.
Observed statistic and p-value. The observed rank sum is \(R_1=5+6+7=18\). There are \(\binom{7}{3}=35\) possible allocations. The largest possible rank sum, 18, occurs only for \(\{5,6,7\}\), so the inclusive upper-tail probability is \(1/35=0.0286\). This is the exact p-value. Since it is below 0.10, reject the null for this observation.
Critical region. The rank-sum counts for sums 16, 17, and 18 are 2, 1, and 1, respectively. Therefore \(P_0(R_1\ge17)=(1+1)/35=2/35=0.0571\), which is at most 0.10. Lowering the cutoff to 16 gives \(4/35=0.1143\), which exceeds 0.10. The nonrandomized rejection region is therefore \(R_1\ge17\), with actual size \(2/35=0.0571\), rather than exactly 0.10. The observed sum 18 is in this region.
This example illustrates the discreteness of the exact test: at the chosen level, no cutoff gives a rejection probability exactly equal to 0.10. Common mistake avoided: rounding \(2/35\) to 0.10 and then treating the test as having size 0.10 would misstate its actual null rejection probability.
Interpretation and Exam Traps
An exact one-sided rank-sum test reports how surprising the observed rank sum, or a more extreme one in the specified direction, would be under the common-distribution null. A small upper-tail p-value is evidence that sample 1 tends to have larger ranks; a small lower-tail p-value is evidence in the opposite direction. A p-value alone does not establish that the distributions differ only by a location shift. Interpreting the result specifically as a location comparison requires additional assumptions about the distributions' shapes.
- Check the alternative before calculating. “Sample 1 is larger” means an upper tail; “sample 1 is smaller” means a lower tail.
- Include the observed value. Use \(R_1\ge r_{\text{obs}}\) or \(R_1\le r_{\text{obs}}\), not a strict inequality.
- Use the correct number of allocations. The denominator is \(\binom{n_1+n_2}{n_1}\), because the rank allocation is determined by choosing which pooled ranks belong to sample 1.
- Respect discreteness. A nonrandomized test can have actual size below the nominal level; do not force a cutoff that makes the tail probability exceed \(\alpha\).
- Check the assumptions. The simple enumeration assumes independent samples and a common continuous distribution under the null. It is not the paired signed-rank procedure.
Check Your Understanding
For each problem, assume independent samples from a common continuous distribution under the null, with no ties, unless stated otherwise.
- There are two observations in sample 1 and three in sample 2. For an upper-tail alternative, the observed sample 1 ranks are 4 and 5. Find the exact p-value. (Answer: \(1/10=0.1000\))
- For two samples of size 3 each, the observed sample 1 rank sum is 12 and the alternative is upper-tailed. Using the rank-sum counts for ranks 1 through 6, find the exact p-value. (Answer: \(7/20=0.3500\))
- Sample 1 has size 2 and sample 2 has size 4. Its observed rank sum is 5, and the alternative is lower-tailed. Find the exact p-value. (Answer: \(4/15=0.2667\))
- Sample 1 has size 3 and sample 2 has size 4. At \(\alpha=0.05\), find the nonrandomized upper-tail rejection region. The possible sample 1 rank sums are 6 through 18. (Answer: reject for \(R_1\ge18\); the actual size is \(1/35=0.0286\))
- For sample sizes 2 and 3, the alternative is that sample 1 tends to be smaller. Its observed ranks are 2 and 3. Find the lower-tail p-value. (Answer: \(4/10=0.4000\))