Ties and Approximate Rank-Sum P-Values

Why Ties Change the Rank-Sum Calculation

The Mann–Whitney rank-sum test compares two independent groups by ranking their pooled observations. When values are tied, each tied observation receives the average of the ranks it would have occupied. Those average ranks preserve the overall rank total, but ties change the spread of the ranks. As a result, the usual no-ties variance formula is not appropriate: the variance must be adjusted before a normal-approximation p-value is calculated.

This tutorial focuses on that adjustment and on how to use it. The procedure assumes independent observations and, under the null hypothesis, that group labels could be exchanged without changing the distribution of the pooled data. This is the equal-distributions null for the rank-sum test. If the two distributions have different shapes, a result should not automatically be described as a difference in medians.

Definition: Pool the $N=n_1+n_2$ observations from two independent groups and assign ranks from 1 to $N$. Give each observation in a tied group the average of the ranks that group occupies. Let $W_1$ be the sum of the ranks assigned to group 1. Its Mann–Whitney statistic is $U_1=W_1-n_1(n_1+1)/2$. Under the equal-distributions null, $E(U_1)=n_1n_2/2$.

For example, if three observations occupy ranks 3, 4, and 5, each receives rank $(3+4+5)/3=4$. The resulting ranks may be fractional, so $W_1$ and $U_1$ need not be integers. The two complementary statistics satisfy $U_1+U_2=n_1n_2$, where $U_2$ is calculated for group 2.

The Tie-Corrected Variance

Under the null, imagine holding the pooled observations and their ranks fixed, then assigning $n_1$ of the $N$ observations to group 1 at random. If there are no ties, the rank-sum variance is $n_1n_2(N+1)/12$. Tied ranks make the pooled ranks less spread out, reducing the variance. To adjust for this, let $t_j$ be the number of observations in the $j$th tied group, counting only groups of size two or more. Define $T=\sum_j(t_j^3-t_j)$.

Theorem: Conditional on the pooled observations, under random assignment of $n_1$ labels to group 1, the tie-corrected null variance of $W_1$, and therefore of $U_1$, is $$ \operatorname{Var}(U_1)=\operatorname{Var}(W_1)=\frac{n_1n_2}{12}\left[N+1-\frac{T}{N(N-1)}\right]. $$ The standard error is the square root of this variance. If there are no ties, $T=0$, and the formula reduces to $n_1n_2(N+1)/12$.

A short derivation explains the correction. If the pooled midranks are $r_1,\ldots,r_N$, their mean is $(N+1)/2$. For a random sample of $n_1$ ranks drawn without replacement from this fixed set, the variance of their sum is $n_1n_2/[N(N-1)]$ times the sum of squared deviations of the ranks from their mean. The sum of squared deviations of the midranks is $[N(N^2-1)-T]/12$: ties reduce the untied sum of squares by $T/12$. Substitution gives

\operatorname{Var}(W_1)=\frac{n_1n_2}{N(N-1)}\frac{N(N^2-1)-T}{12} =\frac{n_1n_2}{12}\left[N+1-\frac{T}{N(N-1)}\right].

Since $U_1=W_1-n_1(n_1+1)/2$ differs from $W_1$ by a constant, it has the same variance. The correction adjusts the variance for ties; it does not change the observed ranks or make a poor normal approximation reliable for a very small sample.

From the Statistic to an Approximate P-Value

After calculating the observed $U_1$, its null mean and tie-corrected standard error, standardize it to obtain a $z$-statistic. A continuity correction is often used because the rank-sum statistic has a discrete distribution, while the normal distribution is continuous. The correction moves the observed value half a unit toward the null mean. With ties, possible statistic increments can depend on the tie pattern, so the half-unit adjustment is a conventional approximation rather than an exact feature of every tied distribution. State whether it is used; the worked examples below use it.

Procedure: Write $\mu_U=n_1n_2/2$ and $SE=\sqrt{\operatorname{Var}(U_1)}$, using the tie-corrected variance. For a lower-tail alternative, use $z=(U_1+0.5-\mu_U)/SE$; for an upper-tail alternative, use $z=(U_1-0.5-\mu_U)/SE$. The approximate one-sided p-value is the corresponding standard normal tail probability. For a two-sided alternative, use $z$ adjusted toward the mean and calculate $2P(Z\leq -|z|)$, where $Z\sim N(0,1)$. This two-sided calculation is an approximation.

Worked Example: A Two-Sided Test With Several Ties

Suppose hypothetical measurements in group A are 2, 4, 4, 7, 9, and those in group B are 1, 4, 6, 7, 10. Test the equal-distributions null against a two-sided alternative using a tie-corrected normal approximation with a continuity correction.

There are $n_1=n_2=5$ observations and $N=10$. The pooled ranks are: 1 receives rank 1; 2 receives rank 2; the three 4s share ranks 3, 4, and 5, so each receives rank 4; 6 receives rank 6; the two 7s share ranks 7 and 8, so each receives rank 7.5; 9 receives rank 9; and 10 receives rank 10. Group A’s rank sum is

W_1=2+4+4+7.5+9=26.5,\qquad U_1=26.5-\frac{5(6)}{2}=11.5.

The tie groups have sizes 3 and 2, so $T=(3^3-3)+(2^3-2)=24+6=30$. The null mean and variance are

\mu_U=\frac{5(5)}{2}=12.5,\qquad \operatorname{Var}(U_1)=\frac{25}{12}\left[11-\frac{30}{10(9)}\right] =22.22, \qquad SE=\sqrt{22.22}=4.714.

Because the observed statistic is below its null mean, the continuity-corrected value is

z=\frac{11.5+0.5-12.5}{4.714}=-0.1061.

The two-sided approximate p-value is $2P(Z\leq-0.1061)\approx 0.9155$. These hypothetical data provide little evidence against equal distributions at common significance levels. The p-value is not the probability that the null hypothesis is true.

Worked Example: A One-Sided Approximation With Ties

Consider hypothetical observations with group A values 1, 2, 2, 3 and group B values 2, 4, 5, 6. Calculate the approximate p-value for the alternative that group A tends to have smaller values than group B.

The three observations equal to 2 share ranks 2, 3, and 4, so each receives rank 3. The other pooled ranks are 1 for the value 1, 5 for the value 3, and 6, 7, and 8 for values 4, 5, and 6. Thus group A has $W_1=1+3+3+5=12$, and

U_1=12-\frac{4(5)}{2}=2.

There is one tied group, of size 3, so $T=3^3-3=24$. Here $n_1=n_2=4$ and $N=8$. The null mean and tie-corrected standard error are

\mu_U=\frac{4(4)}{2}=8,\qquad \operatorname{Var}(U_1)=\frac{16}{12}\left[9-\frac{24}{8(7)}\right] =11.43,\qquad SE=\sqrt{11.43}=3.381.

The lower-tail continuity correction gives $z=(2+0.5-8)/3.381=-1.627$. Therefore the approximate lower-tail p-value is $P(Z\leq-1.627)\approx0.0518$. This calculation follows the specified smaller-values alternative; it is not the two-sided p-value. The small sample makes the normal approximation less dependable, so the result should be described as approximate.

How This Differs From an Exact Untied Calculation

An exact untied rank-sum calculation uses the discrete null distribution rather than approximating it with a normal curve. For a small sample, if the null makes all assignments of the group labels to the pooled observations equally likely, one can enumerate those assignments and count how many produce a statistic at least as extreme as the observed one. The next example makes the distinction concrete.

Worked Example: Exact and Approximate Results Without Ties

Suppose hypothetical group A values are 1, 2, 3 and group B values are 4, 5, 6. There are no ties, and the ranks are 1 through 6. Group A has $W_1=1+2+3=6$, so $U_1=6-3(4)/2=0$. Under the equal-distributions null, the three group A labels could occupy any of $\binom{6}{3}=20$ sets of ranks, each equally likely.

The observed $U_1=0$ is the smallest possible value. The complementary statistic is $U_2=3(3)-0=9$, the largest possible value. For a two-sided exact test, as extreme as the observed result means $U_1=0$ or $U_1=9$. Exactly one of the 20 allocations puts group A in ranks 1, 2, and 3, and exactly one puts it in ranks 4, 5, and 6. Therefore,

p_{\mathrm{exact}}=\frac{2}{20}=0.10.

For comparison, the untied normal variance is $n_1n_2(N+1)/12=3(3)(7)/12=5.25$, giving $SE=\sqrt{5.25}=2.291$ and $\mu_U=4.5$. With a continuity correction, $z=(0+0.5-4.5)/2.291=-1.746$, producing an approximate two-sided p-value of about 0.0808. The exact and approximate answers differ because one uses the discrete rank-sum distribution and the other uses a continuous normal approximation. Neither calculation has ties; tied data require the tie-adjusted variance for this normal-approximation procedure.

Assumptions, Limits, and Common Mistakes

For the usual rank-sum interpretation and null calculation, the groups should be independent, and observations within each group should be independent. Under the null, the two groups have the same distribution, so the group labels are exchangeable across the pooled observations. The approximate calculation also relies on the normal curve being a reasonable approximation to the null distribution. It tends to be more useful as sample sizes grow; with small samples, heavy ties, or an unusually extreme statistic, a normal approximation can be inaccurate. If exact inference is needed with ties, the null distribution can in principle be obtained by permuting labels among the fixed pooled observations and recalculating the tied-rank statistic. That is a separate exact enumeration, not the untied exact calculation in the example.

A frequent mistake is to apply the no-ties variance formula after assigning average ranks. For the first worked example, that would give variance $25(11)/12=22.92$, rather than the tie-corrected 22.22. The difference changes the standard error and therefore the p-value. Another mistake is to call a normal-approximation p-value exact merely because the calculation used the observed ranks. A p-value is exact only when its reference distribution and tail probability have been calculated exactly under the stated null; substituting a normal curve is an approximation.

Key takeaway: With tied observations, use pooled average ranks and include the tie correction $T=\sum_j(t_j^3-t_j)$ in the null variance before standardizing. Report a normal-based result as an approximate p-value, state the tail and continuity-correction choice, and do not confuse it with an exact untied calculation.

Check Your Understanding

Try these questions before reviewing the formulas.

Three observations occupy pooled ranks 5, 6, and 7. What rank should each receive, and what is the contribution of this tie group to $T$?
In a two-group rank-sum analysis with $N=12$, the tied group sizes are 2 and 4. Calculate $T$.
Why does a tie correction reduce the null variance compared with the no-ties formula?
If $U_1$ is below its null mean and the alternative is lower-tailed, which direction does the continuity correction move the observed statistic?
What distinguishes an exact untied p-value from a tie-corrected normal-approximation p-value?