Why Rearrange Data to Represent a Null Hypothesis?
A test asks whether an observed statistic would be surprising if a specified null hypothesis were true. A permutation test answers this by rearranging the observed data in ways that represent the null, then recalculating the statistic for each rearrangement. The resulting values form a null distribution: a reference distribution describing the statistic when the hypothesized effect or pattern is absent.
The rearrangements do not create new measurements. They change which observed values are paired with which labels, positions, or other features, according to a rule justified by the null hypothesis and the study design. The central question is therefore not simply, “Can these data be shuffled?” It is, “Under the null hypothesis, are these rearrangements equally plausible?”
The null hypothesis must be stated in terms of the pattern being tested. For example, it might specify no systematic ordering of outcomes across positions, or no effect of a label on an outcome. A permutation test does not automatically test every possible meaning of “no effect.” The null hypothesis must justify the particular rearrangements used.
How the Null Distribution Is Built
Let the observed data be \(D\), and let \(T(D)\) be a statistic chosen to measure the pattern of interest. Examples of statistics include a difference, a slope, or a measure of association. The observed statistic is \(T_{\text{obs}}=T(D)\). Let \(\mathcal{P}\) be the set of rearrangements allowed under the null hypothesis. For each \(\pi\in\mathcal{P}\), rearrange the data to obtain \(D_\pi\), then calculate \(T(D_\pi)\).
The statistic should reflect the alternative of interest. If unusually large values would count against the null, compare the observed statistic with the upper tail of the null distribution. If unusually small values are relevant, use the lower tail. For a two-sided question, define extremeness in both directions—for example, by considering values whose distance from the null reference value is at least as large as the observed distance. The choice must match the hypothesis, not be selected after seeing which direction gives a smaller p-value.
A valid permutation test depends on exchangeability under the null: the observations that the procedure rearranges must have a joint distribution unchanged by the permitted rearrangements. In plain terms, if the null is true, the data must not favor one permitted assignment or ordering over another. Independent observations with the same distribution are one setting in which this symmetry can hold. A study design may impose additional restrictions, such as requiring rearrangements to remain within blocks.
A broad statement such as “there is no effect” may be insufficient by itself. For instance, it may say that average outcomes do not differ while allowing other features of their distributions to differ. If the proposed rearrangement requires the full distributions to be interchangeable, that weaker null may not justify the test. The null model, the design, and the rearrangement rule must fit together.
Worked Examples
Worked Example: Testing for a Pattern Across Ordered Positions
Suppose a hypothetical instrument records values \(2,4,6,8\) at four ordered positions. The question is whether there is evidence of a systematic increase from the first position to the last. Use the statistic “last value minus first value,” and consider a two-sided alternative. Assume under the null that the four values are exchangeable across positions.
The observed statistic is \[ T_{\text{obs}}=8-2=6. \] A two-sided result at least as extreme has \(|T|\geq 6\). Across all \(4!=24\) assignments of the four distinct values to the positions, the statistic depends only on which values occupy the first and last positions.
To have \(|T|\geq 6\), the endpoints must be 2 and 8 in either order: their difference is \(8-2=6\), while any other pair differs by less than 6. For each endpoint order, the two remaining values can occupy the middle positions in \(2!=2\) ways. Thus, \(2\times2=4\) arrangements have endpoints 2 and 8, and the reverse endpoint order contributes another 4. The two-sided permutation p-value is therefore \[ \frac{8}{24}=0.3333. \]
Under the stated exchangeability assumption, about one-third of the possible arrangements produce an endpoint difference at least as large in absolute value as the observed difference. This example illustrates the logic of a permutation test; the conclusion depends on the null model making all 24 arrangements equally plausible.
Worked Example: Why a Bootstrap Distribution Is Different
Consider the same four hypothetical values, \(2,4,6,8\), whose sample mean is \[ \bar{x}=\frac{2+4+6+8}{4}=\frac{20}{4}=5. \] A bootstrap sample of size four is drawn with replacement from these observed values. For example, the resample \(2,2,6,8\) has mean \[ \bar{x}^{*}=\frac{2+2+6+8}{4}=\frac{18}{4}=4.5. \] Repeating this process gives a bootstrap distribution of sample means, used to describe sampling variability in estimating the population mean.
Now permute the original four values across four positions and calculate their mean. Every permutation still has sum \(20\), so every permuted mean is \(20/4=5\). That unchanging mean cannot serve as a bootstrap distribution for the mean: permutation without replacement preserves the observed values, whereas bootstrap sampling with replacement can change the resampled values and their mean.
The procedures answer different questions. A bootstrap typically approximates how an estimator varies across repeated samples from a population resembling the observed data. A permutation test instead asks how a statistic would vary under a null hypothesis that makes particular rearrangements valid. When data include labels or pairings, a bootstrap may resample intact observations and retain an observed pattern; a permutation scheme may rearrange the relevant labels or pairings to represent no such pattern. The correct method depends on the inferential question.
Worked Example: Respecting a Blocked Structure
Suppose a hypothetical timing study records a value at an early and a late position in each of two clinics. The observed values are 2 and 4 at Clinic A, and 8 and 10 at Clinic B. The question is whether early positions tend to have smaller values than late positions, while allowing clinics to have different typical values. Under the null, assume the early and late labels are exchangeable within each clinic, but not across clinics.
Use the statistic “sum of the values assigned to the early positions.” The observed statistic is \(2+8=10\). Within Clinic A, the early value can be 2 or 4; within Clinic B, it can be 8 or 10. There are \(2\times2=4\) permitted rearrangements, giving early-position sums \[ 2+8=10,\quad 2+10=12,\quad 4+8=12,\quad 4+10=14. \] The observed sum is the smallest permitted value. For a lower-tailed test, one of the four equally plausible permitted arrangements is at least as small as the observed sum, so the permutation proportion is \(1/4=0.25\).
Permuting all four values freely across both clinics would allow values from one clinic to be assigned to the other. That would ignore the stated structure and would not represent the null model in this example. The allowed rearrangements must preserve the features—here, clinic membership—that the null does not remove.
Why the Rearrangement Rule Matters
The justification for a permutation distribution can be seen by considering the null hypothesis directly. If the null makes every permitted rearrangement equally plausible, then under that null the observed arrangement is just one member of the set \(\mathcal{P}\). Applying the same statistic to every member gives the statistic values that could occur under the null. The fraction of values at least as extreme as the observed one is then a measure of how far into the null distribution the observation falls.
This reasoning is only as sound as the symmetry assumption. If rearrangements treat observations as exchangeable when the study design does not, the resulting distribution may not represent the null. Conversely, when there are genuine restrictions, preserving them is part of the test—not a cosmetic adjustment. A permutation test is not justified merely because software can shuffle the data.
The permutation p-value is interpreted conditionally on the observed data and the chosen null-compatible scheme. A small value means that the observed statistic is unusual among the rearrangements allowed by that null model, providing evidence against it. It does not establish that the alternative is true, quantify the size of an effect, or show that the study design supports a causal conclusion.
Common Mistakes and a Practical Checklist
- “The null is true, so any shuffle is valid.” A null statement alone does not make every rearrangement equally plausible. Specify which observations may be exchanged and why the null and design support that symmetry.
- “Permutation and bootstrap distributions are interchangeable.” They are built differently and answer different questions. Bootstrap resampling with replacement approximates sampling variation; a permutation test rearranges data under a null model.
- “A small p-value is the probability that the null is true.” It is a probability of obtaining a statistic at least as extreme under the specified null model, not a probability assigned to the null itself.
- “Any statistic will do.” The statistic must capture the pattern relevant to the alternative, and the rule for defining extremeness must be specified accordingly.
Before carrying out a permutation test, state the null hypothesis, name the statistic, identify the permitted rearrangements, and explain why those rearrangements are valid under the null. Then compare the observed statistic with the resulting null distribution and interpret the comparison in context. The particular rearrangement schemes and applications vary, but this reasoning is the foundation of each one.
Check Your Understanding
Use the definitions and examples above to answer the following questions.
- In a permutation test, what feature of the null hypothesis justifies rearranging the observed data?
- A statistic is observed to be unusually large. What must be decided before calculating an upper-tail permutation p-value?
- Why does permuting the values \(2,4,6,8\) across four positions leave their mean unchanged?
- How does a bootstrap distribution differ from a permutation null distribution in the question it is designed to answer?
- In a study with two clinics, why might rearrangements need to occur within clinics rather than across all observations?