Start With the Question and the Design
Several procedures can compare more than two groups without relying on the ordinary equal-variance ANOVA model. They are not interchangeable versions of the same test. An F-statistic permutation test, Kruskal–Wallis, Friedman, and a trimmed-mean procedure may use different statistics, rely on different assumptions, and test different null hypotheses. Choosing among them starts with two questions: how were the observations collected, and what feature of the groups is the scientific target?
First distinguish independent groups from blocked data. In an independent-groups design, an observational unit contributes to one group. In a blocked design, observations are organized into sets—such as matched units or repeated measurements—so comparisons should respect those sets. Next state the null hypothesis in terms of the target: equality of full distributions, no treatment effect under a randomization scheme, equality of rank patterns within blocks, or equality of trimmed means.
What Each Procedure Is Designed to Answer
For independent groups, a one-way F-statistic permutation test uses the familiar ratio of between-group to within-group variation, but obtains its reference distribution by reallocating group labels according to a specified exchangeability or randomization scheme. Under a sharp null in a randomized experiment, the outcomes would be unchanged by treatment assignment. If the assignment mechanism is known, the test can compare the observed statistic with statistics from all assignments permitted by that mechanism. Exactness refers to that null and assignment scheme; it does not mean the test is exact for every weaker claim, such as equal population means with unequal distributions.
Kruskal–Wallis is a rank-based test for independent groups. It ranks the pooled observations and assesses whether the groups tend to occupy different parts of that pooled ranking. Its standard null is equality of the group distributions. If the distributions have a common shape and spread and differ only by location, a result is often interpreted as evidence of a location difference; without such a condition, it should not be described simply as a test of equal medians. Ties require appropriate handling, and small-sample calibration may call for a permutation reference distribution rather than relying on a large-sample approximation.
Friedman is designed for blocked or repeated data with one observation for each treatment in each block. It ranks the treatments separately within each block and combines those ranks. It asks whether the treatment rank patterns are alike within blocks. For a randomized complete-block experiment, a permutation test should reassign treatment labels within each block, not freely across all observations. This preserves the design and gives a randomization test of the sharp null under the assignment scheme. Friedman’s usual chi-square approximation can be less reliable when there are few blocks.
A trimmed-mean procedure such as Yuen’s test targets equality of group trimmed means. A trimmed mean is calculated after removing a specified proportion of the lowest and highest observations from each group. Yuen’s procedure uses winsorized variability to estimate uncertainty in the trimmed means. It is an option for independent groups when extreme values or heavy tails make ordinary means and variances sensitive, provided the inferential assumptions of the procedure are reasonable. It is not a distribution-free test and does not automatically test equality of medians.
| Procedure | Typical design | Primary target or null | Important qualification |
|---|---|---|---|
| F-statistic permutation | Independent groups, or a design with a specified restricted assignment scheme | No group effect under the stated exchangeability or sharp randomization null | Permutations must match the null and design; equal means alone need not justify unrestricted permutations |
| Kruskal–Wallis | Independent groups | Equality of group distributions, assessed through pooled ranks | A location or median interpretation needs additional common-shape assumptions |
| Friedman | Blocked or repeated observations, one value per treatment per block | No systematic treatment rank differences within blocks | Keep each block intact in any permutation calibration |
| Trimmed-mean procedure | Independent groups | Equality of specified trimmed means | Robust to some extreme observations, but not assumption-free |
Worked Example: An Exact F-Statistic Permutation Test
Worked Example: An Exact F-Statistic Permutation Test
Suppose a hypothetical randomized experiment assigns six independent units to three treatments, two units per treatment. The observed outcomes are 1 and 2 in group A, 3 and 4 in group B, and 5 and 6 in group C. The outcome is quantitative, units are independent, and the equal-sized assignment is the one used in the experiment. We test the sharp null that treatment assignment changes none of the six outcomes, using an F statistic and all assignments allowed by that design.
The group means are 1.5, 3.5, and 5.5, and the grand mean is 3.5. Thus the between-group sum of squares is
Within each group, the two observations differ by 1. Their squared deviations from their group mean sum to 0.5, so the three within-group sums of squares add to 1.5. With \(k=3\) groups and \(N=6\) observations, the observed statistic is
There are \(\binom{6}{2}\binom{4}{2}=90\) assignments to the three labeled groups. For any allocation, the pooled total sum of squares is fixed at 17.5, so \(SS_{\text{between}}=17.5-SS_{\text{within}}\). The denominator degrees of freedom and group sizes are also fixed. Consequently, \(F\) decreases as \(SS_{\text{within}}\) increases. For a pair of observations separated by \(d\), that pair’s within-group sum of squares is \(d^2/2\). The smallest possible total is therefore 0.5 for each of three adjacent pairs, or \(SS_{\text{within}}\geq 1.5\); the observed adjacent-pair grouping attains this minimum. Only the six assignments that give these same three adjacent pairs to the labeled groups are at least as extreme as observed. The exact upper-tail p-value is \(6/90=0.0667\). At a 0.05 significance level, this test does not reject the sharp null. This conclusion concerns the specified randomization null, not a general claim that the three population means are equal.
Worked Example: Independent Groups and a Rank Question
Worked Example: Independent Groups and a Rank Question
Imagine a hypothetical study comparing an ordinal symptom score in three independent groups, with four participants per group. The observed scores are A: 1, 4, 7, 10; B: 2, 5, 8, 11; and C: 3, 6, 9, 12. Each participant contributes one score, and there are no ties. The scientific question is whether the score distributions differ, not whether the arithmetic means differ. Kruskal–Wallis is therefore a natural statistic to consider.
The pooled ranks are 1 through 12. The rank sums are \(R_A=22\), \(R_B=26\), and \(R_C=30\). With \(N=12\) observations and \(n_j=4\) in each group, the Kruskal–Wallis statistic is
The groups’ scores are interleaved in rank, and this statistic is not large. Because each group is small, a large-sample chi-square calibration should not be treated as automatically reliable; an exact or suitably implemented permutation calibration of the rank statistic is a sensible option if the labels are exchangeable under the null. That calibration must reflect the independent-group design. The result should be interpreted as evidence about distributional rank patterns, not as a direct test of equality of means. A median or location interpretation would require the groups to have comparable distributional shapes and spreads.
Worked Example: Respecting Blocks
Worked Example: Respecting Blocks
A hypothetical field experiment compares three treatments in four blocks. Each block contains one plot assigned to each treatment; plots within a block are similar, and treatment labels were randomized separately within blocks. The observed measurements are shown below. Since observations in a block are connected by design, treating all twelve measurements as independent groups would discard important structure. Friedman, with a restricted within-block permutation calibration, matches the design.
| Block | Treatment A | Treatment B | Treatment C |
|---|---|---|---|
| 1 | 8 | 6 | 7 |
| 2 | 5 | 7 | 6 |
| 3 | 9 | 8 | 10 |
| 4 | 4 | 6 | 5 |
Rank each row from 1 (smallest) to 3 (largest). The row ranks are, in order, \((3,1,2)\), \((1,3,2)\), \((2,1,3)\), and \((1,3,2)\). The treatment rank sums are \(R_A=7\), \(R_B=8\), and \(R_C=9\). With \(b=4\) blocks and \(k=3\) treatments, the Friedman statistic is
The ranks vary little across treatments in this example. With only four blocks, the usual large-sample reference distribution is a concern. Under the stated randomization, an exact test can enumerate the \(6^4\) possible within-block label assignments and compare their Friedman statistics with the observed one. Permuting observations across blocks would not be justified: it would break the randomized-block structure. The conclusion concerns a treatment pattern within these blocks, not a comparison that ignores them.
Worked Example: When the Target Is a Trimmed Mean
Worked Example: When the Target Is a Trimmed Mean
Consider hypothetical independent measurements from two treatment groups: A has 0, 1, 2, 3, 100, and B has \(-100\), 2, 3, 4, 5. Suppose the extreme values are plausible observations but the prespecified scientific target is a 20% trimmed mean, rather than the ordinary mean or the full distribution. With five observations per group, trimming 20% removes one value from each end. The trimmed means are
A Yuen procedure is aligned with this target: it tests whether the population 20% trimmed means are equal, using winsorized variability to estimate uncertainty. The observations must be independent across and within groups, and the method’s approximation should be reasonable for the available sample sizes and distributions. Here, the example makes the target explicit and shows why the ordinary mean could be strongly affected by the extremes. It does not establish a p-value or a significant difference: that would require calculating the appropriate Yuen statistic and its reference distribution. Nor would a trimmed-mean result be a test of equality of medians or of entire distributions.
A Practical Selection Sequence
Use the following sequence before calculating a statistic. It keeps the null hypothesis, design, and assumptions connected rather than selecting a test solely because the data appear skewed or contain outliers.
Decide whether the question concerns a sharp treatment-effect null, full distributional differences, within-block rank patterns, or equality of trimmed means.
Use independent-group methods only when the groups consist of independent observational units. If units are matched or grouped into blocks, preserve that structure.
Consider F-statistic permutation for an appropriate exchangeability or randomization null, Kruskal–Wallis for independent-group rank comparisons, Friedman for blocked ranks, or a trimmed-mean procedure for a trimmed-mean target. Then check that the reference distribution matches the assumptions and design.
Report which feature of the groups the procedure addresses. Do not translate a rank or trimmed-mean result into a claim about means, medians, or causation unless the design and assumptions support it.
Why the Choice Matters
The procedures differ in more than sensitivity to nonnormal data. A rank test changes the information used by the statistic; a trimmed-mean test changes the location target; a permutation method changes how the null distribution is obtained. These choices affect both validity and interpretation. The same data could support different analyses if the scientific questions differ, but the question and method must be stated together.
A permutation test is not automatically valid merely because labels can be shuffled in software. Its allowed permutations must follow the null exchangeability claim or the experiment’s assignment mechanism. Likewise, ranks do not make a test assumption-free: the design must still be appropriate, and a location interpretation of Kruskal–Wallis requires additional distributional conditions. A robust procedure can reduce sensitivity to extreme observations, but it tests its specified robust target rather than repairing every possible design or modeling problem.
Common Mistakes
- Choosing only by appearance. “The data are skewed, so use a rank test” skips the scientific target. First decide whether the question concerns distributions, ranks, or a trimmed mean.
- Calling every permutation test exact for equal means. Exactness requires the stated exchangeability or sharp randomization null. Equal means alone do not generally make outcomes freely exchangeable when distributions or variances differ.
- Permuting blocked observations across blocks. This ignores the design. Restricted permutations must keep each block’s observations together and permute only as the assignment scheme permits.
- Interpreting Kruskal–Wallis as automatically a median test. Its general null concerns distributions; a simple location or median interpretation needs common-shape conditions.
- Calling a trimmed-mean test a test of medians. A trimmed mean is a distinct population quantity. Name the trimming proportion and target when reporting the result.
Check Your Understanding
For each situation, identify a suitable procedure and state one key condition or interpretation limit.
- A randomized experiment has independent groups and a sharp null that no unit’s outcome would change under treatment assignment. What makes an F-statistic permutation test exact under the design?
- Why is a Kruskal–Wallis result not automatically evidence that group medians differ?
- In a randomized complete-block study, which labels may be permuted for an exact randomization test, and why?
- A researcher prespecifies equality of 20% trimmed means as the target and has independent groups with extreme observations. Which procedure is aligned with that target?
- Explain why choosing a rank-based statistic and choosing a permutation reference distribution are separate decisions.