Introduction
Clinical trials frequently rely on statistical tests whose validity depends on distributional assumptions. For example, a two-sample t-test uses a model for the sampling distribution of a difference in means, while many regression methods rely on assumptions about errors and their variance.
A permutation test takes a different approach.
Instead of specifying a theoretical reference distribution and asking how extreme the observed treatment effect is relative to that distribution, a permutation test constructs a reference distribution directly from the observed data by repeatedly rearranging the treatment labels in a way that is consistent with the null hypothesis.
This makes permutation methods especially attractive when the sample size is small, the endpoint distribution is unusual, or the investigator wants a test that is closely tied to the randomization structure of the clinical trial.
The Basic Permutation Principle
Consider a randomized two-arm clinical trial with:
- \(n_T\) patients assigned to treatment
- \(n_C\) patients assigned to control
- a total sample size of \(N=n_T+n_C\)
Suppose the observed endpoint values are:
and each patient has a treatment indicator:
The observed treatment effect might be measured using a difference in means:
A permutation test repeatedly rearranges the treatment labels while keeping the observed endpoint values fixed.
For every rearrangement, the same statistic is calculated.
The collection of these statistics forms the permutation distribution.
Why Does Permuting Labels Make Sense?
The central assumption is called exchangeability.
Under a sharp null hypothesis that treatment has no effect, the treatment labels can be rearranged without changing the joint distribution of the observed outcomes.
Conceptually, if treatment truly has no effect, then the observed endpoint values should not contain information about which patients received treatment.
Sharp Null vs. Average Treatment Effect
This distinction is particularly important in randomized clinical trials.
A permutation test based on unrestricted reassignment of treatment labels is naturally associated with a sharp null hypothesis.
For example:
This says that treatment has no effect on any individual patient's potential outcome.
A more conventional clinical-trial hypothesis might instead concern an average treatment effect:
These hypotheses are not identical.
Consequently, the precise interpretation of a permutation test depends on the statistic, randomization scheme, and null hypothesis being tested.
A Simple Clinical Example
Suppose a randomized clinical trial compares change from baseline in a continuous biomarker.
There are 10 patients:
| Patient | Treatment | Change |
|---|---|---|
| 1 | Treatment | 8 |
| 2 | Treatment | 6 |
| 3 | Treatment | 9 |
| 4 | Treatment | 7 |
| 5 | Treatment | 5 |
| 6 | Control | 2 |
| 7 | Control | 3 |
| 8 | Control | 1 |
| 9 | Control | 4 |
| 10 | Control | 2 |
The treatment mean is:
The control mean is:
Therefore the observed treatment difference is:
The question is whether a difference of 4.6 is unusually large under the null hypothesis of no treatment effect.
What Happens During a Permutation?
The endpoint values are left unchanged:
But the treatment labels are rearranged so that five observations are assigned to treatment and five to control.
For example, one possible permutation might assign:
- Treatment: 8, 6, 2, 3, 1
- Control: 9, 7, 5, 4, 2
The corresponding permuted difference is:
which is calculated and stored.
This process is repeated for every possible treatment assignment in an exact permutation test.
How Many Permutations Are There?
If there are \(N\) patients and exactly \(n_T\) are assigned to treatment, the number of distinct treatment assignments is:
For the example with 10 patients and five treatment assignments:
Therefore, an exact permutation test requires only 252 distinct treatment assignments.
This is small enough to enumerate completely.
The Permutation Distribution
After calculating the treatment-effect statistic for every possible assignment, we obtain a distribution such as:
| Permutation | Permuted Treatment Mean | Permuted Control Mean | Difference |
|---|---|---|---|
| 1 | 4.8 | 4.6 | 0.2 |
| 2 | 5.4 | 4.0 | 1.4 |
| 3 | 3.8 | 5.6 | -1.8 |
| ... | ... | ... | ... |
| 252 | 7.0 | 2.4 | 4.6 |
The observed statistic is then compared with this reference distribution.
One-Sided Permutation p-Value
Suppose the alternative hypothesis is that treatment produces a larger endpoint than control.
The one-sided permutation p-value is:
where \(M\) is the number of permutations.
For the example:
The smaller the proportion of permutation statistics at least as extreme as the observed statistic, the stronger the evidence against the null.
The Plus-One Convention for Monte Carlo Tests
When the number of possible permutations is too large to enumerate, a random sample of permutations can be generated.
Suppose \(B\) random permutations are generated and \(b\) produce a statistic at least as extreme as the observed statistic.
A commonly used Monte Carlo p-value is:
The plus-one adjustment prevents a Monte Carlo p-value of exactly zero and provides a conservative finite-simulation estimate.
Exact Permutation Test vs. Randomization Test
In a randomized clinical trial, permutation tests are closely connected to randomization tests.
A randomization test uses the treatment assignments allowed by the actual randomization scheme to construct the null distribution.
For example, suppose the trial used blocked randomization.
It would generally be inappropriate to ignore the blocks and randomly assign treatment labels across the entire dataset.
Instead, the permutations should respect the randomization mechanism.
Blocked Randomization
Suppose patients were randomized in blocks of four with two treatment and two control assignments in each block.
A valid permutation procedure should preserve that structure.
Within each block, the treatment labels can be rearranged among the patients in that block, but treatment labels should not be freely exchanged between blocks.
The resulting permutation distribution therefore reflects the randomization that could actually have occurred.
Stratified Randomization
The same principle applies to stratified randomization.
Suppose patients are randomized separately within:
- Region
- Disease stage
- Baseline risk category
The permutation procedure should generally preserve the treatment allocation structure within these strata.
Otherwise, the reference distribution may no longer represent the design that generated the data.
Permutation Tests for Continuous Endpoints
Continuous endpoints are a natural application.
Possible statistics include:
- Difference in means
- Difference in medians
- Difference in trimmed means
- Standardized mean difference
- Regression coefficient
- Rank-based statistics
The permutation framework is flexible because the test statistic can be chosen separately from the mechanism used to generate the null distribution.
Difference in Means
For two independent groups, define:
For a two-sided test, unusually large positive or negative values are considered evidence against the null.
A common two-sided permutation p-value is therefore:
The absolute value makes the test sensitive to departures in either direction.
Permutation Tests for Binary Endpoints
Permutation tests can also be applied to binary clinical endpoints.
Suppose the primary endpoint is response:
A possible statistic is the difference in response rates:
The treatment labels are permuted while preserving the observed response values.
For a randomized trial, this can produce an exact finite-sample reference distribution when the number of treatment assignments is manageable.
Connection With Fisher's Exact Test
An important special case is the analysis of a \(2\times2\) table.
Suppose the data are:
| Response | No Response | Total | |
|---|---|---|---|
| Treatment | 8 | 2 | 10 |
| Control | 3 | 7 | 10 |
| Total | 11 | 9 | 20 |
Under a fixed-margin null hypothesis, the distribution of the treatment-group response count is hypergeometric.
Fisher's exact test can therefore be understood as an exact conditional randomization/permutation-style test for a \(2\times2\) table.
Permutation Tests for Paired Clinical Endpoints
Permutation methods can also be used for paired data.
Suppose each patient has a pre-treatment and post-treatment measurement.
Define the paired difference:
Under a null hypothesis of no systematic treatment-related change, the signs of the paired differences can sometimes be permuted.
For each patient:
This is often called a sign-permutation test.
With \(n\) independent pairs, there are potentially:
possible sign assignments.
Permutation Tests and Wilcoxon Tests
Permutation tests are closely related to rank-based procedures such as the Wilcoxon signed-rank test and the Mann–Whitney test, but they should not be treated as interchangeable in every setting.
A permutation test is defined by:
- The statistic being permuted
- The set of allowable permutations
- The null hypothesis
- The method used to calculate the p-value
Changing the statistic can change the scientific question being tested.
Permutation Tests and the t-Test
Suppose the endpoint is approximately normally distributed and the standard two-sample t-test assumptions are reasonable.
A permutation test and t-test may produce similar conclusions.
However, the permutation approach constructs its reference distribution from the observed data rather than relying on the theoretical t distribution.
The difference becomes more interesting when:
- Sample sizes are small
- Distributions are strongly skewed
- Outliers are influential
- The endpoint is difficult to model parametrically
- The randomization mechanism provides a natural reference distribution
A Critical Limitation: Exchangeability
Permutation testing should never be described as completely assumption-free.
For simple unrestricted label permutation, the observations must be exchangeable under the null hypothesis.
Exchangeability can fail because of:
- Repeated measurements
- Clustering
- Center effects
- Stratification
- Covariate imbalance
- Time trends
- Matched designs
- Family or household dependence
The permutation scheme must therefore be adapted to the structure of the study.
Permutation Within Clusters
Suppose a multicenter trial contains substantial center effects.
Patients from the same center may not be exchangeable with patients from a different center.
A naive global permutation can therefore break the dependence structure.
One approach is to permute treatment labels within appropriate clusters or strata.
Permutation Tests for Regression Models
Permutation tests can also be applied to regression coefficients.
Suppose a model estimates a treatment coefficient:
The observed statistic might be:
Treatment labels can then be permuted and the regression model refitted for each permutation.
This produces:
The observed coefficient is compared with this empirical null distribution.
Covariate-Adjusted Permutation Tests
Covariate adjustment requires more care.
Simply permuting treatment labels and ignoring important design variables may produce an invalid reference distribution.
One approach is to use a statistic derived from a covariate-adjusted model while preserving the randomization restrictions.
For example:
where \(X_i\) is a prespecified baseline covariate.
The permutation procedure then refits the model under each allowable treatment assignment and records the resulting value of the treatment statistic.
Permutation Tests for Survival Endpoints
Time-to-event endpoints present additional complications.
Suppose the endpoint is overall survival or progression-free survival.
Each patient has:
- A survival or event time
- An event indicator
- A treatment assignment
A suitable test statistic could be based on a log-rank statistic or another survival-analysis statistic.
The treatment labels can then be permuted according to the trial's randomization structure.
For each permutation, the survival statistic is recalculated.
and:
form the permutation reference distribution.
Censoring Requires Special Care
Censored survival data are not equivalent to ordinary continuous observations.
Therefore, permutation procedures for survival endpoints must preserve the relevant censoring and randomization structure.
A naive permutation method that ignores informative censoring, stratification, or other design features may not have the intended type I error.
Exact vs. Monte Carlo Permutation Tests
| Feature | Exact Permutation | Monte Carlo Permutation |
|---|---|---|
| Number of permutations | All allowable permutations | Random subset |
| Simulation error | None | Yes |
| Computational cost | Can become very large | Controlled by \(B\) |
| Reproducibility | Deterministic | Requires a random seed |
| Suitable for huge permutation spaces | Often no | Yes |
When Exact Enumeration Becomes Impossible
Consider a trial with 100 patients and 50 assigned to treatment.
The number of possible assignments is:
This number is astronomically large.
It is therefore impossible to enumerate every permutation in practice.
Instead, a Monte Carlo procedure might generate:
or:
randomly selected allowable permutations.
Monte Carlo Standard Error
The estimated permutation p-value itself is subject to Monte Carlo variability.
If the true permutation probability is approximately \(p\), the standard error of the estimated proportion is approximately:
For example, if \(p\approx0.05\) and \(B=10{,}000\):
Thus, increasing the number of permutations improves the numerical precision of the Monte Carlo p-value.
Why Very Small p-Values Require Many Permutations
Suppose the desired p-value is approximately 0.001.
With only 1,000 random permutations, there may be very few extreme permutations.
With 100,000 permutations, the simulation has substantially more resolution.
This is one reason computationally intensive permutation tests often use tens or hundreds of thousands of permutations.
Permutation p-Values Are Discrete
Exact permutation tests produce a finite set of possible p-values.
If there are 252 possible permutations, the basic one-sided p-value can only change in increments of:
Therefore, an exact permutation p-value is not necessarily a continuous quantity.
This is another reason why an exact p-value may not equal a familiar value such as exactly 0.0500.
Permutation Test Workflow
A Complete Worked Example in R
Consider the 10-patient example above.
The endpoint is change from baseline, and the treatment-effect statistic is the difference in means.
y <- c(8, 6, 9, 7, 5, 2, 3, 1, 4, 2) z <- c( 1, 1, 1, 1, 1, 0, 0, 0, 0, 0 ) obs_stat <- mean(y[z == 1]) - mean(y[z == 0]) obs_stat
The observed difference is:
# 4.6
Generate All Exact Permutations
For five treatment assignments among ten patients, there are only 252 possible treatment assignments.
treatment_sets <- combn( seq_along(y), sum(z == 1) ) n_perm <- ncol(treatment_sets) n_perm
This returns:
# 252
Calculate the Permutation Distribution
perm_stats <- numeric(n_perm)
for (j in seq_len(n_perm)) {
idx_t <- treatment_sets[, j]
idx_c <- setdiff(seq_along(y), idx_t)
perm_stats[j] <-
mean(y[idx_t]) -
mean(y[idx_c])
}
head(perm_stats)
Each value in perm_stats represents the treatment
difference that could have been observed under one permissible treatment
assignment.
Calculate an Exact One-Sided p-Value
p_perm <- mean(perm_stats >= obs_stat) p_perm
The p-value is the proportion of allowable assignments producing a treatment difference at least as large as the observed difference.
Calculate a Two-Sided p-Value
p_perm_two_sided <-
mean(abs(perm_stats) >=
abs(obs_stat))
p_perm_two_sided
The absolute value makes the procedure sensitive to departures in either direction.
The Plus-One Exact/Monte Carlo Convention
If a Monte Carlo sample of permutations is used, a common implementation is:
B <- 10000
set.seed(2026)
perm_stats_mc <- numeric(B)
for (b in seq_len(B)) {
idx_t <- sample(
seq_along(y),
size = sum(z == 1),
replace = FALSE
)
idx_c <- setdiff(
seq_along(y),
idx_t
)
perm_stats_mc[b] <-
mean(y[idx_t]) -
mean(y[idx_c])
}
b_extreme <- sum(
perm_stats_mc >= obs_stat
)
p_mc <- (b_extreme + 1) /
(B + 1)
p_mc
A Reusable R Function
It is often useful to create a general function rather than rewriting the permutation algorithm for every endpoint.
permutation_test <- function(
y,
z,
B = NULL,
alternative = c("two.sided",
"greater",
"less")
) {
alternative <- match.arg(alternative)
n <- length(y)
statistic <- function(y, z) {
mean(y[z == 1]) -
mean(y[z == 0])
}
observed <- statistic(y, z)
n_t <- sum(z == 1)
if (is.null(B)) {
sets <- combn(
seq_len(n),
n_t
)
permuted <- apply(
sets,
2,
function(idx_t) {
z_star <- integer(n)
z_star[idx_t] <- 1
statistic(y, z_star)
}
)
if (alternative == "greater") {
p_value <-
mean(permuted >= observed)
} else if (alternative == "less") {
p_value <-
mean(permuted <= observed)
} else {
p_value <-
mean(
abs(permuted) >=
abs(observed)
)
}
} else {
permuted <- numeric(B)
for (b in seq_len(B)) {
z_star <-
sample(
z,
size = n,
replace = FALSE
)
permuted[b] <-
statistic(y, z_star)
}
if (alternative == "greater") {
b_extreme <-
sum(permuted >= observed)
} else if (alternative == "less") {
b_extreme <-
sum(permuted <= observed)
} else {
b_extreme <-
sum(
abs(permuted) >=
abs(observed)
)
}
p_value <-
(b_extreme + 1) /
(B + 1)
}
list(
statistic = observed,
p_value = p_value,
permutation_statistics = permuted
)
}
Using the Function
result <- permutation_test( y = y, z = z, alternative = "greater" ) result$statistic result$p_value
The result contains the observed treatment statistic, the permutation p-value, and the permutation distribution.
Visualizing the Permutation Distribution
A simple histogram can be used to display the reference distribution.
hist( result$permutation_statistics, breaks = 30, main = "Permutation Distribution", xlab = "Permuted treatment effect" ) abline( v = result$statistic, lwd = 3 )
The observed treatment effect appears as a vertical reference line.
If the observed statistic lies far into the tail of the permutation distribution, the permutation p-value will be small.
Permutation Tests and Confidence Intervals
A permutation test directly provides a hypothesis test, but investigators often also want an effect estimate and confidence interval.
For example, the observed treatment effect might be:
The permutation p-value answers a hypothesis-testing question, while a confidence interval addresses uncertainty around the estimated treatment effect.
Permutation Tests Are Not Automatically Bootstrap Methods
Permutation tests and bootstrap methods are both resampling techniques, but they solve different problems.
| Feature | Permutation | Bootstrap |
|---|---|---|
| Primary purpose | Hypothesis testing | Estimation of sampling variability |
| Basic operation | Rearrange labels/signs | Resample observations |
| Reference distribution | Null/randomization distribution | Estimated sampling distribution |
| Typical question | How unusual is the observed statistic under \(H_0\)? | How variable is the estimated statistic? |
Both methods can be useful in clinical research, but they should not be described as interchangeable.
Permutation Tests and Parametric Tests
Permutation tests can be thought of as an alternative way to obtain a reference distribution.
| Method | Reference Distribution |
|---|---|
| t-test | Theoretical t distribution |
| Chi-square test | Theoretical chi-square distribution |
| F-test | Theoretical F distribution |
| Permutation test | Empirical permutation distribution |
| Randomization test | Distribution induced by allowable treatment assignments |
The advantage of permutation testing is that the reference distribution can be closely connected to the observed data and experimental design.
Advantages of Permutation Tests
- Can work well with small samples.
- Do not require a normal reference distribution for the test statistic.
- Can accommodate many different test statistics.
- Can be closely tied to randomized treatment assignment.
- Can provide exact finite-sample inference in appropriate settings.
- Can be adapted to complex statistics.
- Can provide an intuitive empirical null distribution.
Limitations of Permutation Tests
- They are not assumption-free.
- Exchangeability must be justified.
- Complex randomization schemes require careful implementation.
- Exact enumeration can become computationally impossible.
- Monte Carlo tests introduce simulation error.
- Missing data can complicate the permutation scheme.
- Covariate adjustment may require specialized methods.
- Interpretation depends on the null hypothesis and statistic.
Common Mistake: Permuting the Outcome Instead of the Assignment
In a randomized treatment comparison, the usual permutation operation is to rearrange the treatment assignments while keeping the observed outcomes fixed.
The purpose is to recreate the assignments that could have occurred under the null.
Randomly generating entirely new outcome values would answer a different question.
Common Mistake: Ignoring the Randomization Scheme
Suppose treatment was randomized using blocks.
If an analyst randomly permutes treatment labels across the entire dataset, the resulting distribution may contain assignments that were impossible under the actual trial design.
This can invalidate the intended randomization inference.
Common Mistake: Treating a Small p-Value as Proof of a Clinical Effect
A permutation p-value measures the extremeness of the observed statistic under the specified null framework.
It does not measure:
- The probability that the treatment works
- The probability that the null hypothesis is true
- The clinical importance of the treatment effect
- The probability that the result will replicate
Clinical interpretation still requires consideration of effect size, precision, safety, endpoint relevance, and study design.
Common Mistake: Assuming More Permutations Change the Statistical Answer
Increasing the number of Monte Carlo permutations primarily reduces simulation error.
It does not change the underlying null hypothesis or the scientific meaning of the statistic.
For example, increasing:
can produce a more precise estimate of the same permutation probability, but it does not make an invalid permutation scheme valid.
Common Mistake: Using Permutation Tests Because the Data Are "Non-Normal"
Non-normality alone does not automatically imply that permutation testing is the correct solution.
The more fundamental question is: what permutations are justified by the study design and null hypothesis?
For a randomized trial, the randomization mechanism may provide a compelling basis for permutation inference.
For observational data, however, unrestricted treatment-label permutation is generally not justified because treatment assignment was not randomized.
Permutation Tests in Observational Studies
Permutation methods can sometimes be used in observational research, but exchangeability must be justified carefully.
In an observational study, treatment groups may differ systematically in:
- Age
- Disease severity
- Baseline risk
- Comorbidities
- Prior treatment
Simply shuffling treatment labels can destroy these relationships and produce an inappropriate null distribution.
Permutation Tests in Adaptive Trials
Adaptive clinical trials introduce another consideration.
If treatment assignment probabilities change during the trial, the permutation distribution must reflect the adaptive randomization mechanism.
The relevant reference distribution is not necessarily the set of all assignments with equal probability.
Instead, assignments should be weighted or generated according to the actual design when randomization-based inference is intended.
Permutation Tests and Interim Analyses
Repeatedly calculating an ordinary permutation p-value at multiple unplanned interim analyses can create the same type of multiplicity problem encountered with repeated conventional hypothesis tests.
If interim decisions are planned, the overall testing procedure should account for the sequential design.
A permutation method does not automatically eliminate the need for multiplicity or type I error control.
Permutation Tests and Multiple Endpoints
Clinical trials often have multiple endpoints.
Suppose investigators evaluate:
- Overall survival
- Progression-free survival
- Objective response rate
- A biomarker endpoint
Running a separate permutation test for every endpoint does not automatically control the family-wise type I error rate.
Permutation methods can, however, be incorporated into multiplicity-adjusted procedures by constructing joint permutation distributions for appropriate sets of statistics.
Permutation-Based Multiple Testing
Suppose there are \(K\) test statistics:
For each permutation, calculate all \(K\) statistics.
The resulting joint distribution can be used to evaluate the maximum statistic:
This can provide a basis for simultaneous inference under suitable assumptions and permutation schemes.
Permutation Tests and Missing Data
Missing observations present another important challenge.
Suppose treatment assignment is known for every patient, but the primary endpoint is missing for some patients.
A naive permutation test that simply drops missing observations may alter the effective randomization structure.
The appropriate approach depends on:
- The estimand
- The missing-data mechanism
- The analysis population
- The randomization scheme
- The prespecified analysis method
Permutation inference should therefore be integrated with the missing-data strategy rather than treated as a completely separate computational step.
Permutation Tests and Protocol Prespecification
If permutation testing is to be used for a confirmatory clinical endpoint, the statistical analysis plan should specify the procedure in advance.
At minimum, document:
- Primary endpoint
- Estimand
- Null hypothesis
- Test statistic
- Permutation mechanism
- Randomization restrictions
- Number of permutations
- Exact vs. Monte Carlo implementation
- One-sided or two-sided testing
- p-value convention
- Random-number seed for reproducible simulations
- Handling of missing observations
- Handling of ties
- Multiplicity adjustments
Reproducibility of Monte Carlo Permutation Tests
Monte Carlo procedures use random numbers.
Therefore, a statistical programmer should set and document a random-number seed.
set.seed(20260913)
The exact seed is less important than having a documented, reproducible procedure.
Ties in Permutation Distributions
Discrete clinical endpoints can produce many tied permutation statistics.
For example, binary response data may generate only a relatively small number of distinct treatment-effect values even though many treatment assignments are possible.
The p-value calculation must therefore specify how equality is handled.
For a one-sided test, a common rule is:
rather than strictly:
The choice should be prespecified and used consistently.
One-Sided vs. Two-Sided Permutation Tests
A one-sided alternative might be:
while a two-sided alternative is:
For a two-sided permutation test, a common approach is to compare absolute statistics:
The definition should be established before examining the treatment effect.
Interpreting a Permutation p-Value
Suppose the permutation p-value is:
A reasonable interpretation is:
Under the specified null hypothesis and permutation scheme, approximately 1.8% of allowable treatment assignments produce a test statistic at least as extreme as the observed statistic.
This provides evidence against the null hypothesis.
It does not mean there is a 98.2% probability that the treatment is effective.
Permutation Test vs. Bootstrap: A Clinical Example
Suppose a treatment reduces a biomarker by an estimated 12 units.
A permutation test might ask:
How unusual would a treatment difference of at least 12 units be if treatment assignment had no effect?
A bootstrap procedure might instead ask:
How variable is the estimated 12-unit treatment difference across samples resembling the observed study?
The two procedures therefore answer different inferential questions.
When a Permutation Test Is Particularly Attractive
- A randomized trial provides a clear treatment-assignment mechanism.
- The sample size is relatively small.
- The endpoint has a difficult or unusual distribution.
- A conventional parametric reference distribution is questionable.
- A scientifically meaningful custom test statistic is available.
- An exact test is computationally feasible.
- The randomization structure can be faithfully reproduced.
When a Permutation Test May Not Be Appropriate
- The observations are not exchangeable under the proposed permutations.
- The treatment assignment mechanism is ignored.
- The data contain substantial clustering that is not preserved.
- The analysis involves complex censoring that the permutation scheme does not handle appropriately.
- The procedure is being used simply because the endpoint is non-normal.
- The observational nature of the data makes unrestricted label permutation unjustified.
- The permutation test does not correspond to the estimand of interest.
A Practical Clinical-Trial Decision Framework
What Should Be Reported in a Statistical Analysis Plan?
A permutation test should be described sufficiently precisely that an independent statistical programmer could reproduce it.
A strong SAP description should include:
- The clinical estimand
- The null and alternative hypotheses
- The endpoint definition
- The treatment-effect statistic
- The randomization/permutation mechanism
- Any blocking or stratification restrictions
- The number of permutations
- Whether the test is exact or Monte Carlo
- The p-value calculation
- The direction of the alternative
- Rules for ties
- Handling of missing data
- Multiplicity adjustment
- Random-number seed, if applicable
Worked Example Summary
| Component | Example |
|---|---|
| Study | Randomized two-arm clinical trial |
| Endpoint | Continuous change from baseline |
| Treatment sample size | 5 |
| Control sample size | 5 |
| Observed treatment mean | 7.0 |
| Observed control mean | 2.4 |
| Observed difference | 4.6 |
| Number of exact assignments | 252 |
| Reference distribution | All allowable treatment assignments |
| One-sided statistic | \(\bar Y_T-\bar Y_C\) |
| Two-sided statistic | \(|\bar Y_T-\bar Y_C|\) |
The Most Important Concept
The most important point is that a permutation test is not simply a technique for avoiding normality assumptions.
Its fundamental logic is:
The quality of the inference therefore depends on whether the rearrangements are scientifically and statistically justified.
In a randomized clinical trial, the treatment assignment mechanism provides a particularly strong foundation for this approach.
The test statistic can then be compared with the distribution generated by valid alternative assignments.
References
Fisher, R.A. (1935).
The Design of Experiments.
Oliver and Boyd.
Good, P. (2005).
Permutation, Parametric and Bootstrap Tests of Hypotheses.
3rd ed. Springer.
Lehmann, E.L. & Romano, J.P. (2005).
Testing Statistical Hypotheses.
3rd ed. Springer.
Pesarin, F. & Salmaso, L. (2010).
Permutation Tests for Complex Data: Theory, Applications and Software.
Wiley.
Higgins, J.J. (2004).
Introduction to Modern Nonparametric Statistics.
Brooks/Cole.
Edgington, E.S. & Onghena, P. (2007).
Randomization Tests.
4th ed. Chapman & Hall/CRC.
Winkler, A.M., Ridgway, G.R., Webster, M.A., Smith, S.M. &
Nichols, T.E. (2014).
Permutation inference for the general linear model.
NeuroImage, 92, 381–397.