Two-Group Permutation Tests for Mean Differences

Why Shuffle Group Labels?

Suppose two independent groups have different sample means. Is the observed difference larger than we might see from chance variation if the group labels carried no information about the measured outcome? A permutation test addresses this question by keeping the observed outcomes fixed and repeatedly reallocating them to groups of the same sizes.

The method gives a reference distribution for a chosen statistic under a null model. For comparing two means, the statistic is the difference between the group means. A label shuffle changes which outcomes contribute to each mean, while leaving the pooled set of outcomes and the two group sizes unchanged.

Definition: For two independent groups, define the observed difference in sample means as $D_{\text{obs}}=\bar{x}_1-\bar{x}_2$. A two-group permutation distribution is the collection of values of this statistic obtained by reallocating the pooled observed outcomes to two groups of their original sizes, according to the null model's allowed assignments.

The order in the difference matters: $D_{\text{obs}}>0$ means the first group has the larger sample mean. A permutation test does not resample new outcomes from a population. It reassigns the observed outcomes to group labels.

When Is Label Shuffling Valid?

A permutation distribution is a valid null reference only when the labels are interchangeable under the null being tested. One common setting is that the two groups are independent samples from the same outcome distribution under the null. Conditional on the pooled outcomes and the fixed group sizes, each allocation of labels is then equally likely. Another setting is a completely randomized experiment: under the sharp null that treatment changes no individual's outcome, the random assignment mechanism determines which label allocations are allowed.

These conditions are stronger than merely saying that the two population means are equal. If the groups have different outcome distributions, such as equal means but different spreads, unrestricted shuffling need not represent the null distribution correctly. The permutation procedure should match the way the data were collected and the null hypothesis of interest.

The basic procedure here is for two independent groups. It is not a procedure for paired measurements: paired observations have a different assignment structure, so shuffling individual labels between groups would break the pairs.

Condition for the basic test: Keep the group sizes fixed, and use label allocations that are equally plausible under the null. For independent samples, a standard sufficient condition is that observations in both groups are independent draws from the same distribution under the null. In a randomized experiment, use the experiment's assignment mechanism and the sharp null of no individual treatment effect.

Building and Reading the Distribution

For each allowed allocation, calculate the mean of the outcomes assigned to group 1 and the mean assigned to group 2, then subtract in the same order. The set of resulting differences is the permutation distribution. The observed difference is compared with that distribution in the direction specified by the alternative hypothesis.

For $H_a:\mu_1-\mu_2>0$, count permutation differences at least as large as $D_{\text{obs}}$.
For $H_a:\mu_1-\mu_2<0$, count permutation differences at most as small as $D_{\text{obs}}$.
For a two-sided alternative, count differences whose absolute values are at least $|D_{\text{obs}}|$.

When all allowed allocations are considered, the relevant tail proportion is the permutation p-value. It measures how often the null model produces a statistic at least as far from zero in the direction or directions specified by the alternative. A small p-value is evidence against the null model; it is not the probability that the null hypothesis is true.

Justification: Under the exchangeability condition, the observed assignment is one of the null-allowed assignments, and each such assignment has equal probability conditional on the pooled outcomes. Therefore the rank of the observed statistic among the statistics from those assignments is governed by the null allocation mechanism. The proportion in the specified tail or tails gives the probability, under that mechanism, of a statistic at least as extreme as the observed one. If the null does not make those assignments equally plausible, this justification does not apply.

Worked Examples

Worked Example: A Two-Sided Test with Two Outcomes per Group

Consider hypothetical measurements in two independent groups. Group 1 has outcomes 6 and 8 units; group 2 has outcomes 2 and 4 units. Suppose the null model makes the group labels exchangeable, and test $H_0$ against the two-sided alternative that the group means differ.

The observed statistic is $$ D_{\text{obs}}=\bar{x}_1-\bar{x}_2=\frac{6+8}{2}-\frac{2+4}{2}=7-3=4\text{ units}. $$

Pool the four values, 2, 4, 6, and 8. There are six ways to assign two of them to group 1; the remaining two go to group 2. The resulting differences are:

Group 1 receives 2 and 4: $3-7=-4$ units.
Group 1 receives 2 and 6: $4-6=-2$ units.
Group 1 receives 2 and 8: $5-5=0$ units.
Group 1 receives 4 and 6: $5-5=0$ units.
Group 1 receives 4 and 8: $6-4=2$ units.
Group 1 receives 6 and 8: $7-3=4$ units.

For a two-sided test, a result is at least as extreme as the observed one when its absolute value is at least 4. Only $-4$ and $4$ meet that condition, so the permutation p-value is $2/6=0.3333$. Under this null model, one third of the allowed allocations give a difference at least as far from zero as the observed 4 units. This small example does not provide strong evidence against the null.

Worked Example: Locating a Difference in a Larger Distribution

In another hypothetical comparison, group 1 has measurements 4, 5, and 6 units, while group 2 has 1, 2, and 3 units. Assume the observations are independent and that the two groups are exchangeable under the null. The observed difference is $$ D_{\text{obs}}=\frac{4+5+6}{3}-\frac{1+2+3}{3}=5-2=3\text{ units}. $$

The pooled outcomes sum to 21. If a particular allocation places values with sum $S$ in group 1, the other group has sum $21-S$. Since both groups contain three observations, the permuted difference is $$ D^*=\frac{S}{3}-\frac{21-S}{3}=\frac{2S-21}{3}. $$

The table shows the possible group 1 sums and how many of the 20 allocations produce each sum. There are 20 allocations because choosing three of six pooled observations determines group 1, with the rest assigned to group 2.

Group 1 sum $S$	Number of allocations	Permuted difference $D^*$, in units
6	1	-3
7	1	-2.333
8	2	-1.667
9	3	-1
10	3	-0.3333
11	3	0.3333
12	3	1
13	2	1.667
14	1	2.333
15	1	3

For the two-sided alternative, the condition is $|D^*|\geq |3|$. Only the allocations with differences $-3$ and $3$ qualify: 2 of the 20 allocations. Thus the permutation p-value is $2/20=0.1000$. This calculation uses the absolute value of the same difference statistic, so it counts extreme outcomes in either direction.

Worked Example: Unequal Group Sizes and a One-Sided Alternative

Suppose hypothetical group 1 measurements are 8 and 10 units, and group 2 measurements are 3, 5, and 6 units. Assume independent observations and a null model under which all allocations preserving the two-versus-three group sizes are equally plausible. Test for a larger mean in group 1.

The observed difference is $$ D_{\text{obs}}=\frac{8+10}{2}-\frac{3+5+6}{3}=9-\frac{14}{3}=\frac{13}{3}=4.333\text{ units}. $$

The pooled sum is 32. If group 1 receives a pair with sum $S$, then group 2's sum is $32-S$, so $$ D^*=\frac{S}{2}-\frac{32-S}{3}=\frac{5S-64}{6}. $$

The ten possible pairs from the pooled values have sums $8, 9, 11, 11, 13, 13, 14, 15, 16,$ and $18$. The observed pair, 8 and 10, has sum 18 and gives $D^*=4.333$ units. For the alternative that group 1 has the larger mean, count allocations with $D^*\geq4.333$. Only the pair with sum 18 qualifies: the next largest sum, 16, gives $D^*=(80-64)/6=2.667$ units. The one-sided p-value is therefore $1/10=0.1000$. In this hypothetical comparison, only one tenth of the null-allowed allocations produce a difference at least as large in the positive direction as the observed difference.

What the Test Does—and Does Not—Say

The statistic is measured in the outcome's original units, so the observed difference and each permuted difference have an immediate interpretation. The p-value, by contrast, is a probability under the null model and the assignment scheme. It is not a measure of how large or practically important the observed difference is. Report the observed group means and their difference along with the p-value so the size of the contrast is clear.

The central limitation is the meaning of the null model. A test based on freely permuting labels compares the observed allocation with allocations justified by exchangeability (or by a specified randomized assignment design). If the scientific claim is only that two population means are equal, while the populations may otherwise have different distributions, the basic unrestricted permutation argument may not be valid. In that case, the test's p-value should not be presented as though equality of means alone had guaranteed the shuffle distribution.

Common Mistake

Wrong reasoning: “The two population means are equal under the null, so any outcome can be shuffled between the groups and the resulting p-value is valid.”

Correction: Equal means alone do not make observations exchangeable. The unrestricted two-group test requires a null model that justifies the allowed label reallocations, such as a common distribution for independent samples or an appropriate randomized assignment under a sharp null. State that condition before interpreting the permutation p-value.

Key takeaway: A two-group permutation test compares the observed difference in sample means with differences obtained by reallocating the pooled outcomes while preserving group sizes. Its validity depends on the null model making those reallocations appropriate.

Check Your Understanding

Answer each question by referring to the statistic, allocation rule, and null model.

Group 1 has a mean of 12 units and group 2 has a mean of 9 units. What is $D_{\text{obs}}$ when the statistic is defined as group 1 minus group 2?
Why must a label shuffle retain the original number of observations in each group?
For a two-sided test with an observed difference of $-2$ units, what condition determines whether a permuted difference is at least as extreme?
What null-model condition can justify unrestricted shuffling for two independent samples?
Why does equality of population means alone not necessarily justify a basic two-group permutation test?