Permutation Tests for Several Group Means

Why Permute Labels When Comparing Several Means?

Suppose a researcher compares an outcome measured in three or more groups. The question is whether the population means are all equal, or whether at least one group mean differs. A one-way ANOVA F statistic summarizes how far apart the observed group means are relative to the variation within groups. A permutation test asks whether an F statistic at least this large would be unusual if the group labels carried no information about the outcomes.

Instead of relying on a theoretical F distribution, the permutation approach builds a reference distribution by repeatedly shuffling the group labels while keeping the observed outcomes and each group’s size fixed. The observed statistic is compared with that reference distribution. The test is therefore based on how the data could be rearranged under a specified null model.

Hypotheses and the F Statistic

Definition: For \(k\) groups with population means \(\mu_1,\ldots,\mu_k\), the null hypothesis is \(H_0:\mu_1=\mu_2=\cdots=\mu_k\). The alternative is \(H_a:\) not all of the population means are equal; equivalently, at least one mean differs from another. The alternative does not say which groups differ.

Let \(x_{ij}\) be observation \(j\) in group \(i\), let \(n_i\) be the size of group \(i\), and let \(N=\sum_{i=1}^k n_i\) be the total sample size. The mean of group \(i\) is \(\bar{x}_i\), and the grand mean \(\bar{x}\) is the mean of all \(N\) observations combined. The between-group sum of squares measures how far the group means are from the grand mean, weighted by group size. The within-group sum of squares measures variation among observations around their own group mean.

SS_B=\sum_{i=1}^k n_i(\bar{x}_i-\bar{x})^2,\qquad SS_W=\sum_{i=1}^k\sum_{j=1}^{n_i}(x_{ij}-\bar{x}_i)^2

The corresponding mean squares divide each sum of squares by its degrees of freedom. The one-way ANOVA F statistic is the ratio of these mean squares. A large value means the group means are spread apart relative to the variability of observations within groups.

MS_B=\frac{SS_B}{k-1},\qquad MS_W=\frac{SS_W}{N-k},\qquad F_{\mathrm{obs}}=\frac{MS_B}{MS_W}

Worked Example: Calculating the Observed F Statistic

Consider hypothetical measurements, in points, from three independently sampled groups: A has 8 and 9; B has 10 and 11; C has 12 and 13. Calculate the observed one-way ANOVA F statistic.

The group means are \(\bar{x}_A=8.5\), \(\bar{x}_B=10.5\), and \(\bar{x}_C=12.5\). All six observations sum to 63, so the grand mean is \(\bar{x}=63/6=10.5\). Each group has size 2.

The between-group sum of squares is

$$ SS_B=2[(8.5-10.5)^2+(10.5-10.5)^2+(12.5-10.5)^2] =2(4+0+4)=16 $$

In each group, the two observations lie 0.5 points from their group mean. Thus each group contributes \(0.5^2+0.5^2=0.5\) to \(SS_W\), giving \(SS_W=0.5+0.5+0.5=1.5\). Here \(k-1=2\) and \(N-k=3\), so

MS_B=\frac{16}{2}=8,\qquad MS_W=\frac{1.5}{3}=0.5,\qquad F_{\mathrm{obs}}=\frac{8}{0.5}=16

The observed statistic is 16. It is unitless because it is a ratio of quantities measured in squared points. Whether 16 is unusually large for a permutation test will depend on the permutation distribution, not on this calculation alone.

Building the Permutation Distribution

For each permutation, pool the \(N\) observed outcomes and reassign the group labels, keeping exactly \(n_i\) observations in group \(i\). Recalculate all group means, the grand mean, \(SS_B\), \(SS_W\), and the F statistic for that reassignment. Repeating this process produces the permutation distribution of F under the specified null model.

The group sizes must remain fixed because they are part of the observed design. If the group sizes are 5, 7, and 8, for example, every reassignment must again have groups of sizes 5, 7, and 8. Do not shuffle the outcomes within each existing group: that would leave the group memberships unchanged and would not create the null reference distribution.

Permutation-test procedure: Calculate \(F_{\mathrm{obs}}\), repeatedly reassign the pooled outcomes to groups of the original sizes, and calculate \(F^*\) for every reassignment. The p-value is the proportion of the permutation distribution with \(F^*\geq F_{\mathrm{obs}}\). The comparison is upper-tail because large F values indicate group means that are far apart relative to within-group variation.

If all possible distinct assignments are enumerated, the resulting distribution is exact for the stated permutation null. Often there are too many assignments to enumerate, so a random sample of \(B\) permutations is used instead. If \(b\) of the sampled statistics are at least as large as \(F_{\mathrm{obs}}\), a common Monte Carlo estimate is \((b+1)/(B+1)\). Counting ties with the observed statistic in the upper tail is important: an equally extreme result is not less compatible with the tail event just because it ties rather than exceeds the observation.

Why the F Statistic Splits Variation

The F statistic uses a standard decomposition of total variation. This identity explains why it measures separation among group means relative to variation within groups. Write \(SS_T=\sum_{i=1}^k\sum_{j=1}^{n_i}(x_{ij}-\bar{x})^2\) for the total sum of squares around the grand mean. For an observation in group \(i\), insert and subtract its group mean:

x_{ij}-\bar{x}=(x_{ij}-\bar{x}_i)+(\bar{x}_i-\bar{x})

Squaring and summing over observations gives a within-group term, a between-group term, and a cross-product term. The cross-product is zero: for each group, \(\sum_{j=1}^{n_i}(x_{ij}-\bar{x}_i)=0\), since deviations from a group mean sum to zero. Therefore,

\begin{aligned} SS_T &=SS_W+SS_B+ 2\sum_{i=1}^k(\bar{x}_i-\bar{x}) \sum_{j=1}^{n_i}(x_{ij}-\bar{x}_i)\\ &=SS_W+SS_B \end{aligned}

Thus the total variability is partitioned into within-group and between-group components. In a permutation test the total outcomes stay fixed, while their group assignments change. The F statistic then records how much of the variation appears between groups compared with how much remains within them.

Worked Example: Calculating F for One Reassignment

Use the pooled hypothetical outcomes 7, 10, 8, 11, 8, and 12. Suppose one shuffled assignment gives group A the values 7 and 10, group B the values 8 and 11, and group C the values 8 and 12. Calculate the F statistic for this reassignment.

The group means are 8.5, 9.5, and 10. The grand mean is \(56/6=9.333\) to three decimal places. The between-group sum of squares is

SS_B=2[(8.5-9.333)^2+(9.5-9.333)^2+(10-9.333)^2] \approx 2.333

For groups A and B, the values are 1.5 points below and above their respective means, so each contributes \(1.5^2+1.5^2=4.5\) to \(SS_W\). Group C’s values are 2 points below and above its mean, contributing \(2^2+2^2=8\). Therefore \(SS_W=4.5+4.5+8=17\). With \(k-1=2\) and \(N-k=3\),

F^*=\frac{2.333/2}{17/3} \approx\frac{1.1665}{5.667} \approx 0.2059

This is one reassigned statistic, not the permutation p-value. It contributes to the p-value only if it is at least as large as the observed F statistic. In this example it would not be as large as an observed value of 16.

Worked Example: Estimating a Monte Carlo P-Value

Worked Example: Interpreting a Simulated Tail Count

Suppose a hypothetical study compares three groups and produces \(F_{\mathrm{obs}}=4.72\). A simulation generates \(B=999\) random reassigned data sets, preserving the group sizes each time. In 31 of them, the recalculated statistic is at least 4.72. Estimate the permutation p-value.

Using the plus-one estimate,

\hat{p}_{\mathrm{perm}}=\frac{b+1}{B+1} =\frac{31+1}{999+1} =\frac{32}{1000}=0.032

The estimated p-value is 0.032. Under the permutation null and the procedure used, about 3.2% of random reassigned data sets produce an F statistic at least as large as 4.72. This is evidence against the null at a significance level such as \(\alpha=0.05\); it is not the probability that the null hypothesis is true. The conclusion is that the data provide evidence that not all three population means are equal under the stated model. The test does not identify which means differ.

Assumptions and Scope of the Conclusion

A permutation calculation is only as meaningful as the reassignment scheme. For the usual independent-groups test, labels must be exchangeable under the null being used: under that null, the outcomes could be reassigned among the group labels without changing their joint distribution. One common setting is that all groups are independent samples from the same population distribution under the null. In a randomized experiment, reassignment can instead reflect the actual random treatment-assignment scheme, under a sharp null that treatment has no effect on any unit.

This qualification matters because equal means alone do not guarantee exchangeability. Populations can have the same mean but different spreads or shapes. In that case, freely permuting labels need not give an exact test of the mean-only null. A permutation test based on the F statistic is often described as testing whether several means differ, but its exact validity depends on the null model and design that justify the permutations. The assumptions should be considered before interpreting its p-value.

The method is useful because it offers a data-based reference distribution without requiring the usual theoretical F reference distribution. It also preserves the observed outcomes, including their skewness or outliers, rather than simulating new values from a normal model. But it does not remove the need for a defensible design, an appropriate statistic, or a carefully stated null.

Common Mistakes

Using a lower-tail comparison. A small F statistic does not indicate stronger evidence that means differ. The permutation p-value counts \(F^*\geq F_{\mathrm{obs}}\), because larger F values reflect greater between-group separation relative to within-group variation.
Changing group sizes during shuffling. Each reassignment must preserve the original number of observations in every group. Otherwise the simulated assignments do not match the observed design.
Claiming the test identifies the differing groups. A small overall p-value supports the conclusion that not all means are equal; it does not specify which pair differs.
Assuming equal means automatically makes labels interchangeable. Equal means do not imply equal distributions. If the null does not justify exchanging labels, the resulting permutation p-value may not be an exact test of the mean-only hypothesis.
Interpreting the p-value as a probability about the null. The p-value describes the frequency of statistics at least as extreme as the observed one under the permutation null, not the probability that \(H_0\) is true.

Key takeaway: For several groups, the one-way ANOVA F statistic compares between-group variation with within-group variation. A permutation test evaluates that statistic against values obtained by shuffling labels while preserving group sizes. Interpret the resulting upper-tail p-value only under a null model and design that justify those permutations.

Check Your Understanding

Try these questions without looking back.

Write the null and alternative hypotheses for comparing the means of four populations.
Why does a permutation test for several group means compare the observed F statistic with the upper tail of its permutation distribution?
A study has group sizes 8, 10, and 12. What group sizes must every reassignment retain, and why?
In 499 simulated permutations, 14 statistics are at least as large as the observed statistic. Calculate the plus-one estimate of the permutation p-value.
Explain why a permutation test that freely shuffles labels may not be exact for a null hypothesis asserting only equal means.
A permutation p-value is 0.018. State a correct conclusion, and name one conclusion that this overall test does not establish.