Deriving the Rank-Sum Null Mean and Variance

Why Derive the Rank-Sum Moments?

The two-sample rank-sum statistic replaces measured values with their pooled ranks. Under the null hypothesis that two independent samples come from the same continuous distribution, the group labels are exchangeable across those ranks. That makes it possible to calculate the rank sum’s null mean and variance without knowing the distribution’s units or shape.

These moments describe the center and spread of the rank sum under random allocation. They are useful checks on rank calculations and are ingredients in later rank-sum procedures. The key detail is that allocation is without replacement: once one rank has been assigned to the first group, it cannot be assigned again.

Definition: Let the first independent sample have size $m$, and the second have size $n$, with $N=m+n$. Pool all $N$ observations and rank them from 1 through $N$, with rank 1 assigned to the smallest observation. If there are no ties, the first sample’s rank sum is $R_1$, the sum of its $m$ pooled ranks. Under the null hypothesis of identical continuous distributions, each allocation of $m$ ranks to the first sample is equally likely.

The no-ties condition matters because it ensures the pooled ranks are exactly $1,2,\ldots,N$. If observations tie, average ranks are used and the simple variance formula below does not apply unchanged; tie adjustments are needed. The focus here is the no-ties case.

Null Model and Formulas

Assume the observations in both samples are independent and come from the same continuous distribution $F$ under the null hypothesis. Continuity ensures that ties occur with probability zero. For example, a hypothetical illustration could take every observation to be independently distributed as $\operatorname{Uniform}(0,1)$; the particular continuous distribution does not affect the rank-allocation moments. Conditional on the pooled observations, the null makes every choice of $m$ ranks for the first sample equally likely.

Theorem: For two independent samples of sizes $m$ and $n$, with $N=m+n$ and no tied pooled observations, the first sample’s rank sum $R_1$ has null expectation and variance $$ E[R_1]=\frac{m(N+1)}{2} $$ $$ \operatorname{Var}(R_1)=\frac{mn(N+1)}{12}. $$ The formulas are for the rank sum, not the average rank. They rely on random allocation of the ranks without replacement.

Because the total of all ranks is fixed, the second sample’s rank sum is $R_2=N(N+1)/2-R_1$. It follows that $E[R_2]=n(N+1)/2$, and $R_2$ has the same variance as $R_1$. The two rank sums move in opposite directions: if one is larger, the other must be smaller.

Why the Expectation Formula Holds

For each rank $j$, define $I_j=1$ if rank $j$ is assigned to the first sample and $I_j=0$ otherwise. Then $R_1=\sum_{j=1}^{N}jI_j$. Since exactly $m$ of the $N$ ranks are assigned to the first sample, the chance that any particular rank is selected is $m/N$.

E[R_1] =\sum_{j=1}^{N}jE[I_j] =\frac{m}{N}\sum_{j=1}^{N}j =\frac{m}{N}\frac{N(N+1)}{2} =\frac{m(N+1)}{2}.

This is also intuitive: the average of the pooled ranks $1,\ldots,N$ is $(N+1)/2$, and the first sample receives $m$ ranks on average. Multiplying those quantities gives the expected rank sum.

Why the Variance Formula Holds

The rank indicators are not independent. For two distinct ranks $j$ and $k$, the probability that both are assigned to the first sample is $m(m-1)/(N(N-1))$, because one selected rank leaves $m-1$ selections among $N-1$ remaining ranks. Thus

\operatorname{Var}(I_j)=\frac{mn}{N^2}, \qquad \operatorname{Cov}(I_j,I_k)=-\frac{mn}{N^2(N-1)} \quad (j\ne k).

The negative covariance expresses the without-replacement constraint: selecting one rank slightly reduces the chance of selecting another. Using $R_1=\sum jI_j$, the variance is the sum of the individual variance terms and the pairwise covariance terms:

\begin{aligned} \operatorname{Var}(R_1) &=\frac{mn}{N^2}\sum_{j=1}^{N}j^2 -\frac{2mn}{N^2(N-1)} \sum_{1\le j<k\le N}jk\\ &=\frac{mn}{N^2} \left[ \sum_{j=1}^{N}j^2 -\frac{\left(\sum_{j=1}^{N}j\right)^2-\sum_{j=1}^{N}j^2}{N-1} \right]. \end{aligned}

The bracketed expression simplifies using $\sum j=N(N+1)/2$ and $\sum j^2=N(N+1)(2N+1)/6$. In particular,

\begin{aligned} &\sum_{j=1}^{N}j^2 -\frac{\left(\sum_{j=1}^{N}j\right)^2-\sum_{j=1}^{N}j^2}{N-1}\\ &\qquad=\frac{N\sum_{j=1}^{N}j^2-\left(\sum_{j=1}^{N}j\right)^2}{N-1} =\frac{N^2(N+1)}{12}. \end{aligned}

Substituting this result into the variance expression gives $\operatorname{Var}(R_1)=mn(N+1)/12$. The negative covariance terms are essential: omitting them incorrectly treats the selected ranks as independent.

Applying the Formulas

Worked Example: A Rank Sum With Four Observations

A hypothetical study compares measurements from two independent groups. Under the null model, every observation is independently distributed as $\operatorname{Uniform}(0,1)$ in both groups. The first group has $m=4$ observations and the second has $n=5$. There are no ties. Its pooled ranks are 1, 3, 6, and 9. Find the observed rank sum and the null mean and variance.

Method. Add the first group’s ranks for the observed statistic, then use the rank-sum moment formulas with $N=m+n$.

First, $N=4+5=9$. The observed rank sum is

$$ R_1=1+3+6+9=19. $$

The null expectation is

E[R_1]=\frac{4(9+1)}{2}=\frac{40}{2}=20.

The null variance is

\operatorname{Var}(R_1) =\frac{(4)(5)(9+1)}{12} =\frac{200}{12} =\frac{50}{3} \approx 16.6667.

As a check, the total rank sum is $9(10)/2=45$. The second group’s observed rank sum is $45-19=26$, and its expected rank sum is $5(10)/2=25$. The observed deviations from expectation are therefore $-1$ and $+1$, consistent with the fixed total. The variance is reported to four decimal places. The common mistake avoided here is using the number of ranks in the second group as the rank-sum sample size for $R_1$; $R_1$ contains exactly the first group’s four ranks.

Worked Example: Unequal Groups and the Complement Sum

In a hypothetical comparison, two independent samples have sizes $m=6$ and $n=4$. Under the null, observations in both samples are independent draws from the same continuous distribution; take $\operatorname{Uniform}(0,1)$ for illustration. There are no ties, and the first sample receives ranks 1, 4, 6, 7, 9, and 10. Find both observed rank sums and the null mean and variance for the first sample.

Method. Calculate the first sum directly, use the total of all ranks to obtain the complement, and apply the formulas with the first sample size $m=6$.

Here $N=6+4=10$. The observed first rank sum is

$$ R_1=1+4+6+7+9+10=37. $$

The total of ranks 1 through 10 is $10(11)/2=55$, so

$$ R_2=55-37=18. $$

For the first sample, the null expectation and variance are

E[R_1]=\frac{6(11)}{2}=33, \qquad \operatorname{Var}(R_1)=\frac{(6)(4)(11)}{12} =\frac{264}{12}=22.

The second group’s expected sum is $4(11)/2=22$. The expected sums total $33+22=55$, and the observed sums total $37+18=55$, as required. Both rank sums have variance 22 because one is the fixed total minus the other. The common mistake avoided is assuming the two group rank sums are independent; their sum is fixed, so they are dependent.

Worked Example: Checking a Small First Sample

A hypothetical two-group experiment has $m=3$ observations in the first group and $n=7$ in the second. Under the null, both samples are independent draws from the same continuous distribution, illustrated by $\operatorname{Uniform}(0,1)$. There are no ties. The first group’s ranks are 2, 5, and 9. Calculate its observed rank sum, null expectation, null variance, and deviation from the null expectation.

Method. Sum the observed ranks, calculate the total sample size, and then evaluate each null quantity separately.

The total sample size is $N=3+7=10$, and the observed rank sum is

$$ R_1=2+5+9=16. $$

The null expectation and variance are

E[R_1]=\frac{3(10+1)}{2}=\frac{33}{2}=16.5, $$ $$ \operatorname{Var}(R_1)=\frac{(3)(7)(11)}{12} =\frac{231}{12}=19.25.

The observed deviation from the null mean is $16-16.5=-0.5$. As an arithmetic check, the total rank sum is $10(11)/2=55$, so the other group’s rank sum is $55-16=39$; its expected value is $7(11)/2=38.5$, giving the opposite deviation $+0.5$. The variance is 19.25, not the square root of 19.25; the latter would be the standard deviation. The common mistake avoided is confusing these two measures of spread.

Common Exam Trap: Treating Ranks as Independent

A tempting but incorrect shortcut is to calculate the variance of one rank and multiply by $m$, as though the first sample’s ranks were $m$ independent selections from the pooled ranks. That calculation ignores the negative dependence created by allocation without replacement.

For example, with $N=9$ and $m=4$, the variance of the full set of ranks when viewed as a population is $(9^2-1)/12=80/12$. Multiplying by $m$ gives the incorrect value $4(80/12)=26.6667$. The correct rank-sum variance, with $n=5$, is

\frac{mn(N+1)}{12} =\frac{(4)(5)(10)}{12} =\frac{50}{3} \approx 16.6667.

The incorrect calculation is too large because it fails to account for the finite population: after one rank is selected, it is no longer available. A reliable check is to use the formula containing both group sizes, $mn(N+1)/12$, rather than a with-replacement calculation.

Key takeaway: Under the no-ties null model, the first sample receives a uniformly chosen set of $m$ ranks from $1,\ldots,N$. Its rank-sum expectation is $m(N+1)/2$, and its variance is $mn(N+1)/12$. The factor involving both $m$ and $n$ reflects allocation without replacement.

Check Your Understanding

For each problem, assume independent samples from the same continuous distribution under the null and no tied pooled observations.

A first sample has size $m=2$ and a second sample has size $n=3$. Find the null expectation and variance of the first sample’s rank sum. (Answer: $E[R_1]=6$, $\operatorname{Var}(R_1)=3$.)
For sample sizes $m=4$ and $n=6$, calculate the first sample’s null mean and variance. (Answer: $E[R_1]=22$, $\operatorname{Var}(R_1)=22$.)
There are $m=5$ observations in the first sample and $n=2$ in the second. Find the first sample’s null mean and variance. (Answer: $E[R_1]=20$, $\operatorname{Var}(R_1)=20/3$.)
For $m=3$ and $n=9$, the first sample has observed ranks 1, 4, and 8. Find its observed sum, null mean, variance, and the other sample’s observed sum. (Answer: $R_1=13$, $E[R_1]=19.5$, $\operatorname{Var}(R_1)=29.25$, $R_2=65$.)
With $m=6$ and $n=4$, a candidate uses $m(N^2-1)/12$ for the rank-sum variance. State the correct variance and identify the reason the candidate’s calculation fails. (Answer: The correct variance is $22$; the proposed calculation treats rank selections as independent and ignores allocation without replacement.)

What You'll Learn