Why Permute at the Donor Level?
A composition test compares the relative abundances of cell types across conditions. The data may contain thousands of cells, but condition is usually assigned to, or observed for, a much smaller number of donors. The donor—not the cell—is therefore the unit whose condition label can be permuted. Permuting cell labels as if each cell were an independent experimental unit creates a much larger set of assignments than the study supports and can make evidence appear stronger than it is.
A permutation test builds a reference distribution by recalculating a chosen statistic under condition-label assignments allowed by the study design. Its validity depends on those assignments being exchangeable under the null hypothesis. In a randomized experiment, the design itself can justify the permutations under a sharp null of no treatment effect for any donor. In an observational comparison, validity instead requires that donor outcomes be exchangeable across condition labels under the null, possibly within covariate strata. A permutation procedure cannot repair confounding or an invalid exchangeability assumption.
Choose the Permutation Scheme Before Calculating
For independent groups, permute whole donor labels while preserving the observed number of donors in each group. For a paired design, keep each donor’s two observations together and swap the condition labels within that donor. Equivalently, for a paired difference \(d_i\), the permitted transformation is to replace \(d_i\) by \(-d_i\). This sign-flip scheme is justified by randomized within-pair assignment under a sharp null; in nonrandomized paired data, it requires the paired differences to be exchangeable in sign under the null.
The assignment scheme must also respect the study’s restrictions. If randomization occurred within blocks, permute labels within blocks. If treatment was assigned at the donor level but samples from a donor were measured repeatedly, do not separate those measurements. If condition is completely confounded with batch, no relabeling can distinguish a condition effect from a batch effect. Covariate-adjusted statistics may be useful, but the permutation scheme must still be valid for the design and covariate structure.
A statistic must be selected before examining which one produces the smallest p-value. A prespecified contrast can target a particular cell type’s proportion or a log-ratio coordinate. A global statistic can summarize several prespecified coordinates, such as the largest absolute mean coordinate difference. For each allowed relabeling, recalculate the statistic using the same definition. If the statistic includes estimated scales or other fitted quantities, recalculate those quantities as part of each permutation unless the test’s construction specifically holds them fixed.
Exact p-Values and Their Justification
Let \(\mathcal{A}\) be the set of assignments allowed by the design, and let \(T_{\mathrm{obs}}\) be the observed statistic. For a two-sided test based on a nonnegative extremeness statistic, the exact permutation p-value is
The observed assignment is included in \(\mathcal{A}\). Under the sharp null and a valid randomization scheme, every allowed assignment is equally likely. Thus the rank of the observed statistic among the permuted statistics is uniform when there are no ties; counting ties with “at least as extreme” makes the test conservative when ties occur. For an observational study, the same calculation is only as reliable as the assumed exchangeability of donors under the null.
If only \(B\) random permutations are drawn rather than all assignments enumerated, a common estimate is \((b+1)/(B+1)\), where \(b\) is the number of sampled statistics at least as extreme as observed. The added one prevents reporting a zero p-value just because no sampled permutation was more extreme. Monte Carlo error also means the reported value is an approximation, not an exact enumeration result.
Worked Example: Exact Paired Test of One Cell-Type Proportion
Suppose four hypothetical donors each have a control and treated sample. The entries below are donor-level proportions for cell types A, B, and C; every row sums to 1. The treatment was randomized within donor. Test the prespecified two-sided change in the proportion of type A using the absolute mean paired difference.
| Donor | Control (A, B, C) | Treated (A, B, C) | Difference in A |
|---|---|---|---|
| 1 | (0.20, 0.50, 0.30) | (0.30, 0.40, 0.30) | 0.10 |
| 2 | (0.30, 0.40, 0.30) | (0.40, 0.30, 0.30) | 0.10 |
| 3 | (0.40, 0.30, 0.30) | (0.50, 0.20, 0.30) | 0.10 |
| 4 | (0.30, 0.40, 0.30) | (0.20, 0.50, 0.30) | -0.10 |
The observed mean difference is \((0.10+0.10+0.10-0.10)/4=0.05\), or 5 percentage points. The test statistic is therefore \(T_{\mathrm{obs}}=|0.05|=0.05\). There are \(2^4=16\) within-donor swap patterns. Under any pattern, the four differences are signed versions of \(0.10,0.10,0.10,0.10\). The absolute mean is at least 0.05 exactly when the absolute signed sum is at least 0.20. Eight patterns have signed sum \(0.20\) or \(-0.20\), and two have sum \(0.40\) or \(-0.40\); the other six have sum 0.
Thus \(p_{\mathrm{perm}}=(8+2)/16=0.625\). This small hypothetical dataset does not provide evidence against the sharp null at conventional significance levels. The result is not a 62.5% probability that the null is true. The exact calculation is justified here because the treatment assignment was randomized within each donor, and each permitted swap preserves the pairing.
Prespecified and Global Composition Statistics
A prespecified contrast answers a focused question. For example, a contrast in a log-ratio coordinate compares the relative abundance of one cell group with another, rather than treating a component’s proportion as independent of the rest of the composition. Earlier material on log-ratio coordinates provides ways to express such contrasts. The contrast, its directionality, and the test statistic should be chosen before examining the condition differences.
A global statistic is useful when the scientific question is whether the composition changes in any of several specified directions. For \(q\) chosen coordinates, one option is the maximum absolute mean difference, \(T=\max_{1\leq j\leq q}|\bar d_j|\). Each permutation applies the same donor-level label changes to the full coordinate vector. This preserves the dependence among coordinates, which matters because components of a composition are constrained to sum to one. A global test indicates evidence of some change in the selected coordinates; by itself it does not identify which coordinate drives the result.
Worked Example: Unpaired Donor Permutation
Consider four hypothetical independent donors, two assigned to each condition. The prespecified statistic is the treated-minus-control mean proportion of type A. The other entries complete each composition, so each donor’s proportions sum to 1.
| Condition | Donor composition (A, B, C) |
|---|---|
| Control | (0.20, 0.50, 0.30) |
| Control | (0.30, 0.40, 0.30) |
| Treated | (0.40, 0.30, 0.30) |
| Treated | (0.50, 0.20, 0.30) |
The control mean for A is \((0.20+0.30)/2=0.25\), and the treated mean is \((0.40+0.50)/2=0.45\). The observed difference is \(T_{\mathrm{obs}}=0.45-0.25=0.20\), or 20 percentage points. Under the null, the two treated labels can be assigned to any two of the four donors, giving \(\binom{4}{2}=6\) assignments. The absolute differences for treated pairs \((0.20,0.30)\), \((0.20,0.40)\), \((0.20,0.50)\), \((0.30,0.40)\), \((0.30,0.50)\), and \((0.40,0.50)\) are, respectively, \(0.20, 0.10, 0, 0, 0.10,\) and \(0.20\).
Two of the six assignments are at least as extreme as observed, so \(p_{\mathrm{perm}}=2/6=0.333\). This is an exact enumeration only if the assignment mechanism permits all six allocations with equal probability, or if donor outcomes are exchangeable across labels under the null. With observational groups, the latter is a substantive assumption, not a consequence of the arithmetic.
Worked Example: A Global Test Across Two Coordinates
Suppose three hypothetical paired donors have differences in two prespecified, comparably scaled log-ratio coordinates as follows. The coordinates are dimensionless; the entries are treated-minus-control differences.
| Donor | Coordinate 1 difference | Coordinate 2 difference |
|---|---|---|
| 1 | 1 | 0 |
| 2 | 0 | 1 |
| 3 | 1 | 1 |
Use the global statistic \(T=\max(|\bar d_1|,|\bar d_2|)\). The observed coordinate means are both \(2/3\), so \(T_{\mathrm{obs}}=2/3\). There are \(2^3=8\) within-donor sign patterns. A permuted coordinate sum is \(s_1+s_3\) for coordinate 1 and \(s_2+s_3\) for coordinate 2, where each \(s_i\) is either \(1\) or \(-1\). For two patterns, \(s_1=s_2=-s_3\), both sums are zero and \(T=0\). For each of the other six patterns, at least one sum has absolute value 2, giving \(T\geq 2/3\) after division by 3.
Therefore \(p_{\mathrm{perm}}=6/8=0.75\). This is a global test for changes in the two selected coordinates. It does not test every possible change in the full composition, and it does not identify a coordinate-specific effect.
Diagnostics, Alternatives, and Limitations
Before trusting a result, check that the permutations represent assignments the design could actually have produced. Confirm that donor identifiers and pairs are intact, group sizes or block sizes are preserved, and all analysis steps used to calculate the statistic are repeated consistently. Inspect the donor-level values and paired differences for data errors or a result dominated by one donor. For paired sign flips in nonrandomized data, strongly asymmetric differences are a warning that sign exchangeability may be implausible. For observational comparisons, assess whether condition is entangled with donor covariates or batch; unrestricted permutations are not justified merely because the group sizes match.
Permutation tests offer a design-based or exchangeability-based reference distribution without requiring a particular parametric distribution for the statistic. Their main alternatives include regression and other donor-level models, which can incorporate covariates and estimate effects under their own modeling assumptions. Those models may be preferable when the study has important covariates or a complex repeated-measures structure. A permutation test is not automatically robust to these features: it needs a valid restricted scheme or an appropriate statistic and null construction.
The test also does not account automatically for uncertainty from observing only a finite number of cells per donor. It compares the donor-level summaries supplied to it; donors with very different cell counts may have proportion estimates of different precision. Nor does one global p-value provide effect estimates or intervals. Finally, a test of one prespecified statistic does not license unadjusted claims about many separately inspected cell types or contrasts.
Common Mistakes
- Permuting cells instead of donors. This treats within-donor cells as independent condition assignments. Permute donor labels, retaining each donor’s observations together.
- Breaking pairs. Swapping labels across all samples ignores the within-donor assignment. In a paired design, swap within each donor or flip that donor’s paired difference.
- Using every possible label shuffle by default. Unrestricted shuffling is valid only when the design or null justifies exchangeability across those donors. Respect blocks and other assignment restrictions.
- Interpreting a global rejection as a component-specific conclusion. The global statistic tests the selected collection jointly; follow-up claims require an appropriate plan for component-wise inference.
Check Your Understanding
Try each question before reading the brief solutions.
- A study has six donors, each measured before and after treatment. What labels can be permuted for a paired test?
- Why is permuting 2,000 cells from each of two conditions not a valid substitute for permuting donor assignments when there are only eight donors?
- In an unpaired study with three control and four treated donors, how many assignments are available if all seven donors are exchangeable?
- What does a small p-value from a maximum-statistic global test establish, and what does it not establish?
- An observational comparison has all treated donors processed in one batch and all control donors in another. Does unrestricted donor-label permutation isolate a treatment effect?
Solutions. (1) Swap the before and after labels independently within each donor, giving \(2^6=64\) assignments. (2) Cells from the same donor are not independent assignment units; cell-level shuffling breaks donor structure and overstates the number of independent assignments. (3) There are \(\binom{7}{3}=35\) ways to choose the three control labels. (4) It provides evidence against the joint null for the selected coordinates and statistic, but does not identify which coordinate changed or test all possible compositional changes. (5) No. Condition and batch are confounded, so unrestricted permutations do not distinguish their effects.