Selecting a Test from Study Design

Start With the Study Design

A skewed outcome does not, by itself, tell you which nonparametric test to use. The key questions are how many conditions are being compared and whether the observations are independent, paired, or organized into blocks. A test that handles ranks appropriately can still give a misleading answer if it treats related observations as independent, or independent groups as paired.

The procedures in this tutorial address comparisons of one or more groups using signs or ranks. First identify the observational units and how they were assigned to, or measured under, the conditions. Then select a procedure whose design and assumptions match those facts. The table summarizes the main choices.

Design	Typical number of conditions	Procedure
One set of paired differences; focus on direction or median	Two related conditions	Sign test
One set of paired differences; use magnitudes and assume symmetry	Two related conditions	Signed-rank test
Separate, independent groups	Two groups	Rank-sum (Mann–Whitney) test
Separate, independent groups	Three or more groups	Kruskal–Wallis test
Measurements grouped into independent blocks, with each condition measured in each block	Usually three or more conditions	Friedman test

Definition: Independent groups contain different observational units, with no pairing or block structure connecting units across groups. Paired observations are linked measurements, commonly two measurements on each unit or matched units; their analysis uses the within-pair differences. In a blocked design, each block contains measurements under every condition being compared, and different blocks are independent of one another.

Two Related Conditions: Sign or Signed-Rank?

For paired data, calculate one difference per pair using a consistent order, such as “after minus before.” The differences—not the two columns considered as independent samples—are the basic observations for these procedures. The choice between the sign test and the signed-rank test depends on what information is appropriate to use and what assumptions are reasonable.

Sign test: Uses only whether each nonzero paired difference is positive or negative. Under the null hypothesis of no directional tendency, positive and negative signs are equally likely. With independent pairs and continuous differences, this corresponds to a population median difference of zero. Zero differences are omitted from the sign count.

The sign test is useful when the direction of change is meaningful but the sizes of the differences are not trustworthy or are too asymmetric for the signed-rank procedure. It does not use the ranking of difference magnitudes, so it may have less power when those magnitudes contain reliable information.

Signed-rank test: Ranks the absolute nonzero paired differences and uses their signs. It requires independent pairs and a distribution of differences that is symmetric about its hypothesized center, usually zero. Symmetry makes the center and median coincide; the procedure is not justified merely because the original measurements are numerical.

The signed-rank test uses more information than the sign test, but that advantage comes with the symmetry assumption. If the distribution of differences is strongly asymmetric, or if only the direction of change is defensible, the sign test is the safer choice. If the differences are plausibly symmetric and their magnitudes are meaningful, signed-rank is a natural choice.

Worked Example: Choosing a Sign Test for Paired Changes

In a hypothetical pilot exercise, seven volunteers record a measurement before and after a session. The paired differences, defined as after minus before, are \(2, 1, 3, -1, 2, 4, 1\) units. Suppose the differences are not plausibly symmetric, so their magnitudes should not be used to rank changes. Which procedure fits, and what does its sign calculation show?

The observations are paired: each volunteer contributes one before-and-after pair. The differences are independent across volunteers, and the sign test is appropriate because it does not require symmetry. There are seven nonzero differences: six positive and one negative.

Under the null hypothesis of equally likely positive and negative signs, the number \(B\) of positive differences has a binomial distribution with \(n=7\) and probability \(0.5\). For a two-sided exact test, the observed count of six is as far from \(3.5\) as a count of one, so

P\text{-value}=P(B\geq 6)+P(B\leq 1) =2\frac{\binom{7}{0}+\binom{7}{1}}{2^7} =2\frac{1+7}{128}=0.125.

At a \(0.05\) significance level, this p-value is not small enough to reject the null hypothesis. The data do not provide statistically significant evidence of a directional tendency in the paired changes. This does not establish that the median change is zero.

Worked Example: When Signed-Rank Is the Better Paired Choice

Suppose four independent participants have paired differences \(1, 2, 4, -3\) units, and subject-matter knowledge makes a roughly symmetric distribution of differences plausible. The magnitudes are meaningful. Which procedure fits, and how are its signed ranks formed?

The study has two related conditions and one difference per participant. Because the differences are plausibly symmetric and their magnitudes can be used, the signed-rank test is suitable. Rank the absolute differences \(1, 2, 4, 3\) from smallest to largest. Their ranks are \(1, 2, 4, 3\), respectively. The positive differences have ranks \(1,2,4\), and the negative difference has rank \(3\).

The positive and negative rank sums are therefore

W_+=1+2+4=7,\qquad W_-=3,\qquad W_++W_-=10.

The test compares the observed balance of positive and negative ranks with what would be expected under the null hypothesis. The calculation shows why signed-rank uses more than direction: the difference of \(4\) units contributes more rank weight than the difference of \(1\) unit. The design and symmetry assumption—not the fact that there are only four observations—are the reasons for choosing this procedure.

Independent Groups: Rank-Sum or Kruskal–Wallis?

For independent groups, the units in one group must not be paired with units in another group. Under the rank-sum test, observations from both groups are pooled and ranked together. Under the Kruskal–Wallis test, observations from three or more independent groups are pooled and ranked in the same way, then the rank distributions are compared across groups.

Both procedures require independent observations and an outcome that can be ordered. Their basic null hypotheses concern equality of the group distributions. A conclusion specifically about medians or a common location shift requires more: the group distributions should have similar shapes and spreads, differing mainly in location. If those shapes differ, a rank-based rejection does not by itself show that the medians differ.

Worked Example: Selecting Rank-Sum for Two Independent Groups

A hypothetical team compares a skewed response in two separately recruited groups. Four unrelated units are measured in each group, with results \(3,5,7,9\) for Group A and \(2,4,6,8\) for Group B. No unit appears in both groups. Which procedure fits, and what rank-sum calculation results?

There are two independent groups, so the rank-sum (Mann–Whitney) test is the relevant choice, rather than a paired procedure. All eight observations are distinct, so there are no tied ranks. In pooled order, the values have ranks \(1\) through \(8\); Group A receives ranks \(2,4,6,8\). Its rank sum is

$$ R_A=2+4+6+8=20. $$

For Group A, the corresponding Mann–Whitney statistic is

U_A=R_A-\frac{n_A(n_A+1)}{2} =20-\frac{4(5)}{2}=10.

This calculation identifies the test statistic; a p-value would still require its null distribution or an appropriate approximation. The sample values alone do not change the design classification: the groups remain independent because different units contribute to them.

Worked Example: Choosing Kruskal–Wallis for Three Independent Groups

Consider three independent groups in a hypothetical comparison, with three observations per group: Group A has \(1,4,7\), Group B has \(2,5,8\), and Group C has \(3,6,9\). The values are distinct. Choose a procedure and calculate its rank-based statistic.

There are three separate independent groups, so use Kruskal–Wallis. The pooled ranks are the values themselves, from \(1\) to \(9\). The rank sums are \(R_A=1+4+7=12\), \(R_B=2+5+8=15\), and \(R_C=3+6+9=18\). With total sample size \(N=9\), the statistic is

H=\frac{12}{N(N+1)}\sum_{j=1}^{3}\frac{R_j^2}{n_j}-3(N+1) =\frac{12}{9(10)}\left(\frac{12^2+15^2+18^2}{3}\right)-30 =0.800.

The value of \(H\) alone is not a p-value; it must be compared with an appropriate null distribution or calibration. The example illustrates the selection rule: because the three groups contain different, unrelated units, Kruskal–Wallis fits. Friedman would not fit this design merely because there are three groups.

Blocked or Repeated Measurements: Friedman

When every block contributes an observation under every condition, ranks should be compared within blocks rather than across all observations as if they were independent. The Friedman test ranks the conditions separately inside each block, then assesses whether some conditions tend to receive higher ranks than others. It is designed for blocked or repeated-measures data, typically with three or more conditions.

The blocks must be independent, and the measurements within a block must correspond across conditions. For example, each participant might receive every treatment, or each matched set might include one unit under each treatment. This structure is not the same as independent groups: measurements from the same block are related by design.

Worked Example: Selecting Friedman for Repeated Measurements

Four hypothetical participants each try three conditions, labelled A, B, and C. Their measurements are: participant 1, \(A=8, B=5, C=7\); participant 2, \(A=6, B=9, C=4\); participant 3, \(A=10, B=8, C=9\); participant 4, \(A=5, B=7, C=6\). Assume participants are independent and there are no within-participant ties. Which procedure fits?

Each participant is a block, and each block contains all three conditions, so the Friedman test fits. Rank the conditions from lowest to highest separately within each participant. The rank triples \((A,B,C)\) are \((3,1,2)\), \((2,3,1)\), \((3,1,2)\), and \((1,3,2)\). Thus the condition rank sums are \(R_A=9\), \(R_B=8\), and \(R_C=7\). With \(n=4\) blocks and \(k=3\) conditions,

Q=\frac{12}{nk(k+1)}\sum_{j=1}^{k}R_j^2-3n(k+1) =\frac{12}{4(3)(4)}(9^2+8^2+7^2)-3(4)(4) =0.500.

As with the other rank statistics, this value needs an appropriate reference distribution to obtain a p-value. The design determines the choice: ranking measurements within each participant preserves the pairing that would be lost by treating all twelve values as independent.

A Short Justification for the Sign-Test Calculation

The sign test’s binomial calculation follows directly from its null model. Let \(B\) count positive differences among \(m\) independent nonzero paired differences. Under the null hypothesis of equally likely signs, each difference has probability \(0.5\) of being positive. Independence means the count is a sum of \(m\) independent Bernoulli outcomes, so

B\sim\operatorname{Binomial}(m,0.5),\qquad P(B=b)=\binom{m}{b}(0.5)^b(0.5)^{m-b} =\frac{\binom{m}{b}}{2^m}.

This is why an exact sign-test p-value can be calculated by adding the binomial probabilities at outcomes at least as extreme as the observed count. The argument depends on independent pairs and equal sign probabilities under the null; it does not depend on the sizes of the differences.

A Practical Selection Sequence

Count the conditions being compared.
Separate a two-condition comparison from one involving three or more conditions.

Identify dependence.
Ask whether the same units or matched units occur across conditions, or whether the data are grouped into blocks.

Choose the design-matched procedure.
Use sign or signed-rank for two paired conditions, rank-sum for two independent groups, Kruskal–Wallis for three or more independent groups, and Friedman for blocked or repeated measurements.

Check the procedure’s additional assumptions.
For paired differences, decide whether symmetry and meaningful magnitudes justify signed-rank, or whether sign is more appropriate. For group comparisons, check independence and be cautious about location interpretations when distribution shapes differ.

Common Mistakes and Why the Choice Matters

Choosing only by the number of groups. Three conditions do not automatically mean Kruskal–Wallis. If every participant receives all three conditions, the observations are blocked and Friedman is the design-matched choice.
Treating paired measurements as independent. A before-and-after measurement on the same person is not two independent groups. Analyze the within-person differences with a paired procedure.
Choosing signed-rank just because the outcome is skewed. The relevant assumption is symmetry of the paired differences, not symmetry of each original measurement separately. If the differences are asymmetric, consider the sign test rather than assuming signed-rank is justified.
Calling every rank-test result a median comparison. The basic group null concerns distributions. A median or location-shift interpretation requires additional similarity in distribution shapes.
Reading a large p-value as proof of equality. A result that does not reject the null is not proof that the conditions are identical; the data may simply provide insufficient evidence to distinguish them.

Key takeaway: Match the test to the relationship among observations first, then check assumptions and the intended interpretation. Two paired conditions call for a sign-based or signed-rank procedure; two independent groups call for rank-sum; three or more independent groups call for Kruskal–Wallis; and blocked or repeated measurements call for Friedman.

Check Your Understanding

For each situation, identify the design and the most suitable procedure from this tutorial.

A clinic measures the same patients before and after a program. The paired differences are strongly asymmetric, and only their directions are considered reliable. Which procedure fits?
Two separate groups of patients contribute one measurement each, with no matching between groups. Which test is appropriate?
Four independent treatment groups are compared, with different participants in each group. Which test fits?
Each of six blocks receives measurements under all three conditions, and blocks are independent. Which procedure should be considered?
Why does a significant rank-sum or Kruskal–Wallis result not automatically establish that the group medians differ?