Effect Sizes for Two-Sample Rank Tests

Why Measure the Size of a Rank-Based Effect?

A rank-sum test can assess whether two independent groups show evidence of a difference, but its p-value does not describe how large or practically important that difference is. Effect sizes help fill that gap. For two samples, one direct approach is to compare every value in one group with every value in the other. These comparisons lead to the common-language probability and rank-biserial correlation.

The measures describe relative ordering: how often an observation from one group exceeds an observation from the other. They do not measure a difference in the original units, and they do not by themselves establish that the groups differ in a particular feature such as their medians. The direction must be stated clearly, and the scientific meaning depends on the outcome being measured.

Here, “probability” is a sample-based estimate of a population comparison probability, not a claim that the observed data themselves are random after collection. The common-language estimate counts ties as half, so it can also be written as the number of A wins plus half the number of ties, divided by \(n_A n_B\). Rank-biserial correlation uses the same comparison counts, but expresses the balance of wins and losses on a scale centered at zero.

From Pairwise Comparisons to \(U\) and Rank-Biserial Correlation

For the calculations below, define \(U_A\) as the number of cross-group pairs in which A is larger, plus half the number of tied cross-group pairs. Thus, \(U_A\) ranges from 0 to \(n_A n_B\). This direction convention matters: switching A and B reverses the sign of rank-biserial correlation.

Here, \(U_B\) is the corresponding count for B over A, again giving half credit to ties. Each cross-group pair contributes a total of one across \(U_A\) and \(U_B\): A wins contribute one to \(U_A\), B wins contribute one to \(U_B\), and a tie contributes one-half to each. Therefore \(U_A+U_B=n_A n_B\). This identity gives the second expression for rank-biserial correlation.

The measures can also be calculated from pooled ranks. Rank all observations together from smallest to largest, using average ranks for tied values. Let \(R_A\) be the sum of the ranks for group A. Then

This rank-sum formula gives the same \(U_A\) as the pairwise method, including when pooled ties are assigned average ranks. For a modest data set, pairwise comparisons make the meaning especially transparent; for larger data sets, the rank sum is usually more efficient to calculate.

Worked Calculations

Why the Rank-Sum Formula Works

A short derivation connects the pooled ranks to the pairwise definition. Imagine first that all pooled observations have distinct values. For each observation in A, its rank is one plus the number of observations below it in the pooled ordering. Summing those ranks over A gives one for each A observation, plus contributions from other observations below A values. Of the latter contributions, the A observations account for \(0+1+\cdots+(n_A-1)=n_A(n_A-1)/2\). The remaining contributions count exactly the B observations below A observations: each such cross-group pair is an A win.

Consequently, \(R_A=n_A+n_A(n_A-1)/2+\text{number of A wins}\). The first two terms combine to \(n_A(n_A+1)/2\). Subtracting them gives the number of A wins, which proves the rank-sum formula when there are no ties. With ties assigned average ranks, each tied cross-group pair contributes half to the rank-based count; the same identity therefore gives the half-credit \(U_A\) used above.

Interpreting Magnitude and Direction

The common-language estimate is often easiest to explain in words: “An observation selected from A exceeds one selected from B in an estimated \(q\) of cross-group comparisons, counting ties as half.” Rank-biserial correlation is a signed balance: \(1\) means every cross-group pair favors A; \(-1\) means every pair favors B; and \(0\) means the strict-win proportions are equal. A value of zero does not necessarily mean the two distributions are identical; different distribution shapes can produce a balanced count of wins and losses.

These measures summarize relative ordering, not the size of the gaps. If every A value exceeds its paired comparison by a tiny amount, the common-language estimate can still be 1. The measures also do not have universal cutoffs that determine whether an effect is “small,” “moderate,” or “large.” Practical importance depends on the outcome, the consequences of a difference, and the study context. When reporting a result, state which group is being compared as larger, give the measure and its numerical value, and explain what that ordering means in the outcome’s context.

The comparison probability does not, without further assumptions, mean that one group has a higher median. It is an across-distribution probability statement: a random draw from one target population is more likely to exceed a random draw from the other. The groups may differ in spread or shape as well as location, so do not translate a rank-based effect automatically into a simple shift in typical values.

Conditions, Reporting, and Common Pitfalls

For a meaningful two-sample interpretation, the observations should come from two independent groups, and each recorded value should be comparable on the same ordered outcome scale. If the goal is to generalize beyond the observed samples, the sampling or study design must support that generalization. The effect-size arithmetic itself does not require normally distributed outcomes. If the samples are paired or measurements are repeated on the same individuals, the independent-sample pair counting described here is not appropriate.

A report should identify the direction convention, the sample sizes, the effect estimate, and the outcome context. For example: “Among the hypothetical measurements, an A observation exceeded a B observation in 66.7% of cross-group comparisons; the rank-biserial correlation for A over B was 0.333.” If a test is also reported, present its p-value separately and do not treat it as a measure of effect magnitude.

• Wrong: “The p-value is the probability that there is no difference.” Correction: A p-value is calculated under a specified null model; it is not the probability that the null hypothesis is true.
• Wrong: “A positive rank-biserial correlation proves A has the higher median.” Correction: It indicates a balance of pairwise comparisons favoring A; it does not by itself establish a median difference.
• Wrong: “A common-language estimate of 0.70 means the A values are 70% larger.” Correction: It means 70% of cross-group comparisons favor A, counting ties as half, not that values differ by 70% in size.
• Wrong: “The effect is unimportant because the test was not significant.” Correction: Statistical significance and practical magnitude are different questions. Consider the effect estimate, study size, uncertainty, and context together.