From Data to a Rank-Based Analysis
Choosing a test is only one part of an analysis. A sound account also shows what the data look like, why the study design calls for that procedure, what assumptions make its p-value meaningful, and what the result says in context. This tutorial works through two hypothetical settings: skewed measurements from independent groups and measurements taken before and after an intervention on the same individuals.
Rank-based tests replace the original measurements with their order. That can make them useful when measurements are strongly skewed or include an unusually large value. The tests do not, however, automatically test a difference in medians. Their interpretation depends on the null hypothesis and on whether the groups’ distributions have comparable shapes.
Analysis 1: Skewed Measurements in Independent Groups
Suppose a hypothetical laboratory compares the time, in minutes, to complete a task using two independent methods. The data are:
| Method A | Method B |
|---|---|
| 3, 20, 25, 30, 100 | 1, 2, 4, 5, 6 |
The measurements for Method A are right-skewed: most are between 3 and 30 minutes, while one is 100 minutes. The mean for A is \(178/5=35.6\) minutes, but its median is 25 minutes. Method B has mean \(18/5=3.6\) minutes and median 4 minutes. The medians and ranges (3 to 100 minutes for A; 1 to 6 minutes for B) give a more resistant summary of the typical values and spread than the means alone. The groups consist of different task runs, not matched measurements.
Here the observations are independent between and within groups, and the measurements can be meaningfully ordered. There are no tied values. The unusually large value motivates attention to skew, but it does not itself invalidate a rank test. The exact calculation below uses the null model that all assignments of five pooled ranks to either group are equally likely; that model is appropriate when the two population distributions are the same.
Worked Example: Ranking Skewed Independent Samples
Pool the ten times and assign ranks from smallest to largest. The pooled order is \(1,2,3,4,5,6,20,25,30,100\). Method A therefore has ranks \(3,7,8,9,10\), with rank sum \(W_A=3+7+8+9+10=37\).
For group A of size \(n_A=5\), its Mann–Whitney statistic in the direction of A exceeding B is
There are \(5(5)=25\) cross-group pairs. Counting directly confirms the statistic: the A value 3 exceeds two B values, and each of the other four A values exceeds all five B values. Thus \(U_A=2+4(5)=22\). It estimates a strong tendency for an A observation to exceed a B observation, but the test’s p-value is needed to assess how surprising the ranks would be under the null.
For an exact two-sided p-value, there are \(\binom{10}{5}=252\) equally likely assignments of the pooled ranks to A. The observed statistic is 22, near its maximum of 25. By symmetry, assignments at least this extreme in either direction correspond to \(U\geq22\) or \(U\leq3\). The counts for \(U=0,1,2,3\) are \(1,1,2,3\), respectively, giving seven assignments in the lower tail and seven in the upper tail. Therefore,
At significance level \(\alpha=0.05\), this result is not statistically significant. The sample shows a higher distribution of observed times for Method A, but these data do not provide sufficient evidence at the 5% level that the population distributions differ. This conclusion does not establish that the methods are equivalent, and the p-value is not the probability that the null hypothesis is true.
Analysis 2: Before-and-After Measurements
Now consider a hypothetical set of task times, in minutes, measured on the same six people before and after a training session. Define each change as after minus before, so a negative difference means a shorter time after training.
| Person | Before | After | Difference (after − before) |
|---|---|---|---|
| 1 | 10 | 9 | −1 |
| 2 | 12 | 14 | 2 |
| 3 | 15 | 19 | 4 |
| 4 | 20 | 25 | 5 |
| 5 | 25 | 31 | 6 |
| 6 | 30 | 37 | 7 |
The paired differences range from −1 to 7 minutes, with five positive changes and one negative change. The median difference is \((4+5)/2=4.5\) minutes, and the mean difference is \(23/6=3.833\) minutes. These summaries describe the observed changes; the test assesses evidence about a population-level center of change under its assumptions.
The pairing matters: each after measurement must be matched to its own before measurement, and the analysis is performed on the six differences, not on two unrelated samples. Here the six pairs represent different people, so independence between pairs is plausible by the design as described. The differences are all nonzero and their absolute values are distinct, so no zero differences or tied absolute ranks complicate the calculation. Symmetry of population differences cannot be proved from six observations; it is a modeling assumption to consider alongside the exploratory display and subject-matter knowledge.
Worked Example: Applying the Signed-Rank Test to Paired Changes
Rank the absolute differences from smallest to largest. The absolute differences are \(1,2,4,5,6,7\), so their ranks are \(1,2,3,4,5,6\). Only the difference of −1 has a negative sign, and it has rank 1. The sum of positive ranks is \(W_+=2+3+4+5+6=20\); the sum of negative ranks is \(W_-=1\).
Under the null hypothesis and the stated symmetry and independence assumptions, each of the \(2^6=64\) possible assignments of signs to the six ranks is equally likely. For a two-sided test, outcomes at least as extreme as the observed one have \(W_+\leq1\) or \(W_+\geq20\). There are two assignments with \(W_+\leq1\): all ranks are negative, or only rank 1 is positive. By symmetry there are two assignments with \(W_+\geq20\). Thus,
At \(\alpha=0.05\), the result is not statistically significant. The observed changes are mostly positive, and the sample median increase is 4.5 minutes, but there is not sufficient evidence at the 5% level to conclude that the population center of paired changes differs from zero under the signed-rank model. In particular, a positive difference here means a longer time after training; the direction must follow the difference definition.
Why Sample Size and Direction Matter
An exact p-value is determined not only by how consistent the observed signs or ranks appear, but also by how many distinct outcomes the null model allows. With very small samples, even an extreme pattern can fail to cross a conventional significance threshold.
Worked Example: Five Consistent Paired Changes
In a separate hypothetical before-and-after analysis, five people have changes of \(1,2,4,7,12\) minutes, each defined as after minus before. All changes are positive. Their absolute ranks are \(1,2,3,4,5\), so \(W_+=1+2+3+4+5=15\), its maximum. If independent paired differences are continuous and symmetric about zero under the null, there are \(2^5=32\) equally likely sign assignments. Only the all-positive and all-negative assignments are as extreme in a two-sided test, so
Despite all five changes having the same sign, the exact p-value exceeds 0.05. The data describe five increases, but they do not reach statistical significance at that level under the specified test and assumptions. This illustrates why a conclusion should follow the calculated p-value rather than a visual impression alone.
A Short Justification for the Exact Calculations
Both exact calculations enumerate outcomes that are equally likely under their null models. For the independent-groups example, if the population distributions are identical and the observations are independent, assigning the pooled observations to A or B does not change their joint distribution. Each choice of five of the ten ranks for A is therefore equally likely, which gives \(\binom{10}{5}\) possible assignments.
For the paired signed-rank examples, symmetry about zero implies that, conditional on a nonzero absolute difference, either sign is equally likely under the null. With independent pairs, the sign assignments across \(n\) pairs are equally likely, giving \(2^n\) possible assignments. The exact p-value is the proportion of those assignments producing a statistic at least as extreme as the observed one under the chosen alternative. These justifications also show why dependence between groups or pairs, ties or zeros handled incorrectly, or a failure of the stated null model can undermine an exact calculation.
Common Pitfalls and Reporting
- Calling every rank-sum result a test of medians. The basic null concerns equality of distributions. A median or location-shift interpretation needs an additional common-shape condition.
- Ignoring the design. Before-and-after values from the same person are paired; treating them as independent discards the matching. Conversely, unrelated groups should not be analyzed as paired.
- Forgetting how differences are defined. If change is after minus before, a positive difference means an increase. State the subtraction order and units.
- Reading a p-value as a probability about the null. A p-value describes how often data at least this extreme would occur under the null model and assumptions; it does not give the probability that the null is true.
- Equating non-significance with no effect. A result above \(\alpha\) is not proof of equality or equivalence. Report the observed summaries and the limits of what the test establishes.
A concise report names the groups or paired outcome, the direction and units of measurement, the procedure, the statistic and p-value, and the contextual conclusion. Include the assumptions that matter: independent groups for the rank-sum analysis, or independent pairs and symmetric differences for the signed-rank analysis. Exploratory summaries are not decoration; they help reveal skew, unusual values, and whether an interpretation based on a location shift is sensible.
Check Your Understanding
Use the worked analyses to check your reasoning.
- Why is the rank-sum procedure appropriate for two sets of task times measured on different runs rather than on matched runs?
- In the independent-groups example, what does \(U_A=22\) count, and what does it not establish by itself?
- Why does a rank-sum test not always amount to a test of equal population medians?
- If a paired difference is defined as before minus after, how would the interpretation of a positive difference change?
- What assumptions justify enumerating \(2^n\) sign assignments for an exact signed-rank p-value?
- Why does a p-value of 0.06250 not prove that the population effect is zero?