Normal Approximation for Rank-Sum Tests

Why Approximate a Rank-Sum Test?

A rank-sum test compares two independent samples by pooling their observations and ranking them together. When the pooled observations have no ties, the rank-sum statistic has a discrete null distribution. For small samples, that distribution can be used directly to calculate an exact p-value. As the sample sizes increase, calculating or tabulating every possible rank sum becomes less convenient. A normal approximation provides a practical way to estimate the tail probability.

The approximation has three parts: find the null mean and standard deviation of the rank sum, adjust the observed boundary for its discrete values, and use the appropriate tail of the standard normal distribution. The direction of the test matters: the upper tail concerns unusually large rank sums for the first sample, while the lower tail concerns unusually small ones.

Model, Statistic, and Null Distribution

Suppose the first independent sample has size $n_1$, the second has size $n_2$, and $N=n_1+n_2$. Pool all $N$ observations and assign ranks from 1 through $N$, with rank 1 assigned to the smallest observation. Define $W$ as the sum of the ranks belonging to the first sample.

Definition: In a two-sample rank-sum test, $W$ is the sum of the pooled ranks assigned to the observations in the first sample. Under the null hypothesis that the two independent samples come from the same continuous distribution, and when there are no ties, the null mean and variance are $$ \mu_W=\frac{n_1(N+1)}{2}, \qquad \sigma_W^2=\frac{n_1n_2(N+1)}{12}. $$ The null standard deviation is $\sigma_W=\sqrt{\sigma_W^2}$.

The no-ties condition is satisfied with probability 1 when both populations are continuous. The formula also assumes that observations from the two samples are independent. If there are ties, the no-ties variance formula does not apply as written; the tied ranks affect the null variance. This tutorial uses no-tie data throughout. If sample sizes are small, the exact rank-sum distribution is generally preferable to a normal approximation.

Under the null, each assignment of $n_1$ of the $N$ pooled ranks to the first sample is equally likely. The rank-sum distribution is therefore discrete, not exactly normal. The normal approximation is most useful when the sample sizes are sufficiently large for that discrete distribution to be reasonably represented by a bell-shaped curve; there is no single sample-size cutoff that guarantees a good approximation in every setting.

Standardizing and Correcting for Discreteness

If $W$ were continuous, standardization would give $Z=(W-\mu_W)/\sigma_W$. But with no ties, $W$ takes integer values. A continuity correction places the normal boundary halfway between the integer values included and excluded from a tail event.

Continuity correction: For an observed integer rank sum $w$, approximate an upper-tail event $W\geq w$ using the normal boundary $w-0.5$. Approximate a lower-tail event $W\leq w$ using the boundary $w+0.5$.

For example, $W\geq w$ includes the integer values $w,w+1,\ldots$, so its continuous approximation begins halfway between $w-1$ and $w$, at $w-0.5$. Likewise, $W\leq w$ ends halfway between $w$ and $w+1$, at $w+0.5$. This midpoint adjustment usually better matches the probability of the discrete event than using $w$ itself.

p_{\text{upper}}\approx 1-\Phi\!\left(\frac{w-0.5-\mu_W}{\sigma_W}\right), \qquad p_{\text{lower}}\approx \Phi\!\left(\frac{w+0.5-\mu_W}{\sigma_W}\right),

Here $\Phi(z)$ is the cumulative distribution function of a standard normal random variable. These formulas are for one-sided tests in the direction indicated. For a two-sided test, “twice the tail” means twice the tail as far from the null mean as the observed rank sum—not a tail chosen by reusing the one-sided boundaries without reflecting them about the mean.

Two-sided approximation: Let $d=|w-\mu_W|$. For integer rank sums with no ties, use $$ z_{\mathrm{cc}}=\frac{\max(0,d-0.5)}{\sigma_W}, \qquad p_{\text{two-sided}}\approx \min\left(1,\;2\left[1-\Phi(z_{\mathrm{cc}})\right]\right). $$ If $w>\mu_W$, the observed upper-tail boundary is $w-0.5$, and the equally distant lower-tail boundary is $2\mu_W-w+0.5$. If $w<\mu_W$, the observed lower-tail boundary is $w+0.5$, and the equally distant upper-tail boundary is $2\mu_W-w-0.5$.

The reflected boundaries explain why the two-sided calculation uses the centered distance $d$. For an observation above the mean, for instance, the distance from the mean to $w-0.5$ is $w-\mu_W-0.5$. The matching lower boundary must be the same distance below the mean. Doubling the normal tail at $z_{\mathrm{cc}}$ accounts for both equally distant tails, and the result is capped at 1.

Worked Examples

Worked Example: Upper-Tail Approximation

Two independent samples have sizes $n_1=6$ and $n_2=7$. Under the null hypothesis, both samples come from the same continuous distribution, and the pooled observations have no ties. The first sample has rank sum $W=56$. Find the approximate p-value for the alternative that the first sample tends to have larger observations. No interest rate or mortality assumption is involved; the null distribution is the rank-sum distribution specified above.

Method and null parameters. The alternative calls for an upper-tail probability. The total sample size is $N=6+7=13$, so $$ \mu_W=\frac{6(13+1)}{2}=\frac{84}{2}=42, \qquad \sigma_W^2=\frac{6(7)(14)}{12}=\frac{588}{12}=49, \qquad \sigma_W=\sqrt{49}=7. $$

Apply the upper-tail correction. Since the observed sum is $w=56$, the corrected boundary is $56-0.5=55.5$. Its standardized value is $$ z=\frac{55.5-42}{7}=\frac{13.5}{7}=1.928571\ldots. $$

Find and check the p-value. The required probability is the standard normal upper tail beyond this $z$-value: $$ p\approx 1-\Phi(1.928571\ldots)\approx 0.02689. $$ The result is small and positive, as expected for a rank sum well above its null mean when the alternative is upper-tailed. The common mistake to avoid is using the uncorrected value $56$ as the normal boundary; the event is $W\geq56$, so its corrected boundary is $55.5$.

Worked Example: Two-Sided Approximation With an Upper Rank Sum

Two independent samples have sizes $n_1=9$ and $n_2=8$. Under the null, they come from the same continuous distribution, and there are no ties among the pooled observations. The first sample has rank sum $W=99$. Find the approximate two-sided p-value. The only distributional assumption used is the identical continuous null distribution for the independent samples.

Method and null parameters. Here $N=9+8=17$. Therefore, $$ \mu_W=\frac{9(17+1)}{2}=\frac{162}{2}=81, \qquad \sigma_W^2=\frac{9(8)(18)}{12}=\frac{1296}{12}=108, \qquad \sigma_W=\sqrt{108}\approx10.3923. $$

Measure the centered distance. The observed sum is above the mean by $99-81=18$. After the continuity correction, the distance is $18-0.5=17.5$. Thus $$ z_{\mathrm{cc}}=\frac{17.5}{\sqrt{108}}\approx1.68325. $$

Use both equally distant tails. The upper-tail boundary is $99-0.5=98.5$. The mean is 81, so the matching lower-tail boundary is $2(81)-99+0.5=63.5$, which is 17.5 below the mean. Hence $$ p\approx2\left[1-\Phi(1.68325)\right]\approx0.0923. $$ This probability is larger than the one-sided upper-tail probability because it includes equally extreme outcomes in both directions. A common error is to use $99+0.5$ as a lower-tail boundary. That boundary is not the reflection of the observed upper tail; the matching lower boundary is $63.5$.

Worked Example: Lower-Tail Approximation

A study compares two independent groups with $n_1=8$ and $n_2=10$. Under the null, observations in both groups come from the same continuous distribution, with no ties in the pooled sample. The first group has rank sum $W=87$. Find the approximate p-value for the lower-tail alternative that the first group tends to have smaller observations.

Method and null parameters. The pooled sample size is $N=8+10=18$. The null mean, variance, and standard deviation are $$ \mu_W=\frac{8(18+1)}{2}=\frac{152}{2}=76, \qquad \sigma_W^2=\frac{8(10)(19)}{12}=\frac{1520}{12}\approx126.6667, \qquad \sigma_W=\sqrt{126.6667}\approx11.2546. $$

Correct the lower boundary. The event is $W\leq87$, so its normal boundary is $87+0.5=87.5$. The standardized value is $$ z=\frac{87.5-76}{11.2546} =\frac{11.5}{11.2546} \approx1.0218. $$ This value is positive because $87.5$ is above the null mean. Therefore the lower-tail probability is $$ p\approx\Phi(1.0218)\approx0.8466. $$

The large lower-tail probability is consistent with the data: $87$ is above the null mean, not unusually small. This check is useful because a sign error in the standardization could incorrectly produce a small p-value. The common mistake is to use $87-0.5$ for the lower tail; the event $W\leq87$ instead requires $87+0.5$.

Choosing and Interpreting the Approximation

The p-value measures how unusual the observed rank sum, or one more extreme in the specified direction, would be under the null model. It is not the probability that the null hypothesis is true. A small upper-tail p-value is evidence against the null in the upper-tail direction; a small lower-tail p-value is evidence against it in the lower-tail direction. For a two-sided test, extremeness is measured in either direction from the null mean.

The normal approximation is a calculation method, not a change to the test’s hypotheses or assumptions. Before using it, check that the samples are independent, the pooled observations have no ties if applying the formulas here, and the null hypothesis gives the same continuous distribution to both groups. If the samples are small, the exact two-sided or one-sided rank-sum test is a more direct choice. If ties occur, do not use the no-ties variance formula unchanged.

Check the setup.
Confirm independent samples, the null model, sample sizes, and whether pooled observations are tied.

Calculate the null scale.
Set $N=n_1+n_2$, then compute $\mu_W=n_1(N+1)/2$ and $\sigma_W=\sqrt{n_1n_2(N+1)/12}$ for the no-ties case.

Choose the corrected boundary.
Use $w-0.5$ for an upper tail and $w+0.5$ for a lower tail. For a two-sided test, use the centered distance $|w-\mu_W|-0.5$, with the probability capped at 1.

Calculate and check the tail.
Standardize with $\sigma_W$, use the requested normal tail, and verify that the result is consistent with the observed rank sum and alternative.

A frequent exam trap is to double the wrong tail or to use two boundaries that are not equally distant from the null mean. For example, if $w>\mu_W$, the upper boundary is $w-0.5$, and the matching lower boundary is $2\mu_W-w+0.5$. Both are the same corrected distance from $\mu_W$. This symmetry is what makes doubling the corresponding normal tail appropriate.

Key takeaway: The rank-sum normal approximation requires the correct null mean and standard deviation, a continuity correction matched to the tail event, and a two-sided calculation centered on the null mean. Check the assumptions before applying the no-ties formulas.

Check Your Understanding

For each problem, assume the two samples are independent, the null distributions are identical and continuous, and there are no ties. Use a normal approximation with a continuity correction; report p-values to four decimal places.

The sample sizes are $n_1=6$ and $n_2=7$, and the first sample has rank sum $W=56$. Find the upper-tail p-value. (Answer: 0.0269)
The sample sizes are $n_1=9$ and $n_2=8$, and the first sample has rank sum $W=99$. Find the two-sided p-value. (Answer: 0.0923)
The sample sizes are $n_1=4$ and $n_2=5$, and the first sample has rank sum $W=28$. Find the upper-tail p-value. (Answer: 0.0331)
The sample sizes are $n_1=6$ and $n_2=7$, and the first sample has rank sum $W=28$. Find the lower-tail p-value. (Answer: 0.0269)

What You'll Learn