Tutorials › Biostatistics › Permutation Tests for Clinical Endpoints

Nonparametric & Resampling Methods

Permutation Tests for Clinical Endpoints

A practical guide to permutation testing for clinical endpoints, including the permutation principle, exchangeability, exact and Monte Carlo tests, treatment-group comparisons, p-values, covariate adjustment, survival endpoints, and implementation in R.

Advanced 18 min read

What You'll Learn

  • What a permutation test is and why it can be useful in clinical research
  • How exchangeability creates the reference distribution
  • How exact and Monte Carlo permutation p-values are calculated
  • How to perform a permutation test for a continuous treatment endpoint
  • How permutation tests extend to binary, paired, and survival endpoints
  • How to implement permutation tests and interpret them in R

Introduction

Clinical trials frequently rely on statistical tests whose validity depends on distributional assumptions. For example, a two-sample t-test uses a model for the sampling distribution of a difference in means, while many regression methods rely on assumptions about errors and their variance.

A permutation test takes a different approach.

Instead of specifying a theoretical reference distribution and asking how extreme the observed treatment effect is relative to that distribution, a permutation test constructs a reference distribution directly from the observed data by repeatedly rearranging the treatment labels in a way that is consistent with the null hypothesis.

This makes permutation methods especially attractive when the sample size is small, the endpoint distribution is unusual, or the investigator wants a test that is closely tied to the randomization structure of the clinical trial.

Key idea: A permutation test asks: if treatment assignment had no relationship to the endpoint, how often would random rearrangements of the treatment labels produce a result at least as extreme as the one actually observed?

The Basic Permutation Principle

Consider a randomized two-arm clinical trial with:

  • \(n_T\) patients assigned to treatment
  • \(n_C\) patients assigned to control
  • a total sample size of \(N=n_T+n_C\)

Suppose the observed endpoint values are:

\[ Y_1,Y_2,\ldots,Y_N \]

and each patient has a treatment indicator:

\[ Z_i= \begin{cases} 1,&\text{if patient }i\text{ receives treatment}\\ 0,&\text{if patient }i\text{ receives control} \end{cases} \]

The observed treatment effect might be measured using a difference in means:

\[ T_{\mathrm{obs}} = \bar Y_T-\bar Y_C \]

A permutation test repeatedly rearranges the treatment labels while keeping the observed endpoint values fixed.

For every rearrangement, the same statistic is calculated.

The collection of these statistics forms the permutation distribution.

Why Does Permuting Labels Make Sense?

The central assumption is called exchangeability.

Under a sharp null hypothesis that treatment has no effect, the treatment labels can be rearranged without changing the joint distribution of the observed outcomes.

Conceptually, if treatment truly has no effect, then the observed endpoint values should not contain information about which patients received treatment.

Exchangeability is the foundation. Permutation tests are not assumption-free. They replace one set of assumptions with another, and the validity of the test depends critically on whether the proposed permutations are justified by the study design and null hypothesis.

Sharp Null vs. Average Treatment Effect

This distinction is particularly important in randomized clinical trials.

A permutation test based on unrestricted reassignment of treatment labels is naturally associated with a sharp null hypothesis.

For example:

\[ H_0:Y_i(1)=Y_i(0) \quad\text{for every patient }i \]

This says that treatment has no effect on any individual patient's potential outcome.

A more conventional clinical-trial hypothesis might instead concern an average treatment effect:

\[ H_0:E[Y(1)-Y(0)]=0 \]

These hypotheses are not identical.

Consequently, the precise interpretation of a permutation test depends on the statistic, randomization scheme, and null hypothesis being tested.

A Simple Clinical Example

Suppose a randomized clinical trial compares change from baseline in a continuous biomarker.

There are 10 patients:

Patient Treatment Change
1 Treatment 8
2 Treatment 6
3 Treatment 9
4 Treatment 7
5 Treatment 5
6 Control 2
7 Control 3
8 Control 1
9 Control 4
10 Control 2

The treatment mean is:

\[ \bar Y_T = \frac{8+6+9+7+5}{5} = 7 \]

The control mean is:

\[ \bar Y_C = \frac{2+3+1+4+2}{5} = 2.4 \]

Therefore the observed treatment difference is:

\[ T_{\mathrm{obs}} = 7-2.4 = 4.6 \]

The question is whether a difference of 4.6 is unusually large under the null hypothesis of no treatment effect.

What Happens During a Permutation?

The endpoint values are left unchanged:

\[ 8,\ 6,\ 9,\ 7,\ 5,\ 2,\ 3,\ 1,\ 4,\ 2 \]

But the treatment labels are rearranged so that five observations are assigned to treatment and five to control.

For example, one possible permutation might assign:

  • Treatment: 8, 6, 2, 3, 1
  • Control: 9, 7, 5, 4, 2

The corresponding permuted difference is:

\[ T^{*} = \frac{8+6+2+3+1}{5} - \frac{9+7+5+4+2}{5} \]

which is calculated and stored.

This process is repeated for every possible treatment assignment in an exact permutation test.

How Many Permutations Are There?

If there are \(N\) patients and exactly \(n_T\) are assigned to treatment, the number of distinct treatment assignments is:

\[ {N\choose n_T} = \frac{N!}{n_T!(N-n_T)!} \]

For the example with 10 patients and five treatment assignments:

\[ {10\choose5} = 252 \]

Therefore, an exact permutation test requires only 252 distinct treatment assignments.

This is small enough to enumerate completely.

Exact permutation test: When the number of possible assignments is manageable, every valid permutation can be evaluated. No random sampling of permutations is necessary.

The Permutation Distribution

After calculating the treatment-effect statistic for every possible assignment, we obtain a distribution such as:

Permutation Permuted Treatment Mean Permuted Control Mean Difference
1 4.8 4.6 0.2
2 5.4 4.0 1.4
3 3.8 5.6 -1.8
... ... ... ...
252 7.0 2.4 4.6

The observed statistic is then compared with this reference distribution.

One-Sided Permutation p-Value

Suppose the alternative hypothesis is that treatment produces a larger endpoint than control.

The one-sided permutation p-value is:

\[ p_{\mathrm{perm}} = \frac{ \#\{T^{*}\ge T_{\mathrm{obs}}\} }{ M } \]

where \(M\) is the number of permutations.

For the example:

\[ p_{\mathrm{perm}} = \frac{ \text{number of permutations with }T^{*}\ge4.6 }{ 252 } \]

The smaller the proportion of permutation statistics at least as extreme as the observed statistic, the stronger the evidence against the null.

The Plus-One Convention for Monte Carlo Tests

When the number of possible permutations is too large to enumerate, a random sample of permutations can be generated.

Suppose \(B\) random permutations are generated and \(b\) produce a statistic at least as extreme as the observed statistic.

A commonly used Monte Carlo p-value is:

\[ \hat p = \frac{b+1}{B+1} \]

The plus-one adjustment prevents a Monte Carlo p-value of exactly zero and provides a conservative finite-simulation estimate.

Exact vs. Monte Carlo: An exact test evaluates every permissible rearrangement. A Monte Carlo permutation test evaluates only a random subset and therefore introduces simulation error.

Exact Permutation Test vs. Randomization Test

In a randomized clinical trial, permutation tests are closely connected to randomization tests.

A randomization test uses the treatment assignments allowed by the actual randomization scheme to construct the null distribution.

For example, suppose the trial used blocked randomization.

It would generally be inappropriate to ignore the blocks and randomly assign treatment labels across the entire dataset.

Instead, the permutations should respect the randomization mechanism.

Clinical-trial principle: Whenever possible, the permutation scheme should mirror the actual treatment assignment mechanism used in the trial.

Blocked Randomization

Suppose patients were randomized in blocks of four with two treatment and two control assignments in each block.

A valid permutation procedure should preserve that structure.

Within each block, the treatment labels can be rearranged among the patients in that block, but treatment labels should not be freely exchanged between blocks.

The resulting permutation distribution therefore reflects the randomization that could actually have occurred.

Stratified Randomization

The same principle applies to stratified randomization.

Suppose patients are randomized separately within:

  • Region
  • Disease stage
  • Baseline risk category

The permutation procedure should generally preserve the treatment allocation structure within these strata.

Otherwise, the reference distribution may no longer represent the design that generated the data.

Permutation Tests for Continuous Endpoints

Continuous endpoints are a natural application.

Possible statistics include:

  • Difference in means
  • Difference in medians
  • Difference in trimmed means
  • Standardized mean difference
  • Regression coefficient
  • Rank-based statistics

The permutation framework is flexible because the test statistic can be chosen separately from the mechanism used to generate the null distribution.

Difference in Means

For two independent groups, define:

\[ T = \bar Y_T-\bar Y_C \]

For a two-sided test, unusually large positive or negative values are considered evidence against the null.

A common two-sided permutation p-value is therefore:

\[ p_{\mathrm{perm}} = \frac{ \#\{|T^{*}|\ge|T_{\mathrm{obs}}|\} }{ M } \]

The absolute value makes the test sensitive to departures in either direction.

Permutation Tests for Binary Endpoints

Permutation tests can also be applied to binary clinical endpoints.

Suppose the primary endpoint is response:

\[ Y_i= \begin{cases} 1,&\text{response}\\ 0,&\text{no response} \end{cases} \]

A possible statistic is the difference in response rates:

\[ T = \hat p_T-\hat p_C \]

The treatment labels are permuted while preserving the observed response values.

For a randomized trial, this can produce an exact finite-sample reference distribution when the number of treatment assignments is manageable.

Connection With Fisher's Exact Test

An important special case is the analysis of a \(2\times2\) table.

Suppose the data are:

Response No Response Total
Treatment 8 2 10
Control 3 7 10
Total 11 9 20

Under a fixed-margin null hypothesis, the distribution of the treatment-group response count is hypergeometric.

Fisher's exact test can therefore be understood as an exact conditional randomization/permutation-style test for a \(2\times2\) table.

Useful connection: Permutation methods are not an isolated statistical technique. Fisher's exact test, randomization tests, and many modern resampling procedures share the same basic idea of constructing a reference distribution from permissible rearrangements.

Permutation Tests for Paired Clinical Endpoints

Permutation methods can also be used for paired data.

Suppose each patient has a pre-treatment and post-treatment measurement.

Define the paired difference:

\[ D_i=Y_{i,\mathrm{post}}-Y_{i,\mathrm{pre}} \]

Under a null hypothesis of no systematic treatment-related change, the signs of the paired differences can sometimes be permuted.

For each patient:

\[ D_i^{*}\in\{D_i,-D_i\} \]

This is often called a sign-permutation test.

With \(n\) independent pairs, there are potentially:

\[ 2^n \]

possible sign assignments.

Permutation Tests and Wilcoxon Tests

Permutation tests are closely related to rank-based procedures such as the Wilcoxon signed-rank test and the Mann–Whitney test, but they should not be treated as interchangeable in every setting.

A permutation test is defined by:

  • The statistic being permuted
  • The set of allowable permutations
  • The null hypothesis
  • The method used to calculate the p-value

Changing the statistic can change the scientific question being tested.

Permutation Tests and the t-Test

Suppose the endpoint is approximately normally distributed and the standard two-sample t-test assumptions are reasonable.

A permutation test and t-test may produce similar conclusions.

However, the permutation approach constructs its reference distribution from the observed data rather than relying on the theoretical t distribution.

The difference becomes more interesting when:

  • Sample sizes are small
  • Distributions are strongly skewed
  • Outliers are influential
  • The endpoint is difficult to model parametrically
  • The randomization mechanism provides a natural reference distribution

A Critical Limitation: Exchangeability

Permutation testing should never be described as completely assumption-free.

For simple unrestricted label permutation, the observations must be exchangeable under the null hypothesis.

Exchangeability can fail because of:

  • Repeated measurements
  • Clustering
  • Center effects
  • Stratification
  • Covariate imbalance
  • Time trends
  • Matched designs
  • Family or household dependence

The permutation scheme must therefore be adapted to the structure of the study.

Permutation Within Clusters

Suppose a multicenter trial contains substantial center effects.

Patients from the same center may not be exchangeable with patients from a different center.

A naive global permutation can therefore break the dependence structure.

One approach is to permute treatment labels within appropriate clusters or strata.

General rule: Do not permute observations in a way that destroys the dependence or randomization structure that was present in the original study.

Permutation Tests for Regression Models

Permutation tests can also be applied to regression coefficients.

Suppose a model estimates a treatment coefficient:

\[ Y_i = \beta_0+\beta_1Z_i+\epsilon_i \]

The observed statistic might be:

\[ T_{\mathrm{obs}}=\hat\beta_1 \]

Treatment labels can then be permuted and the regression model refitted for each permutation.

This produces:

\[ T_1^{*},T_2^{*},\ldots,T_M^{*} \]

The observed coefficient is compared with this empirical null distribution.

Covariate-Adjusted Permutation Tests

Covariate adjustment requires more care.

Simply permuting treatment labels and ignoring important design variables may produce an invalid reference distribution.

One approach is to use a statistic derived from a covariate-adjusted model while preserving the randomization restrictions.

For example:

\[ Y_i = \beta_0+\beta_1Z_i+\beta_2X_i+\epsilon_i \]

where \(X_i\) is a prespecified baseline covariate.

The permutation procedure then refits the model under each allowable treatment assignment and records the resulting value of the treatment statistic.

Important: A permutation test does not automatically solve covariate-adjustment problems. The permutation mechanism and analysis statistic must be designed together.

Permutation Tests for Survival Endpoints

Time-to-event endpoints present additional complications.

Suppose the endpoint is overall survival or progression-free survival.

Each patient has:

  • A survival or event time
  • An event indicator
  • A treatment assignment

A suitable test statistic could be based on a log-rank statistic or another survival-analysis statistic.

The treatment labels can then be permuted according to the trial's randomization structure.

For each permutation, the survival statistic is recalculated.

\[ T_{\mathrm{obs}} = T(\text{observed treatment assignments}) \]

and:

\[ T_1^{*},\ldots,T_M^{*} \]

form the permutation reference distribution.

Censoring Requires Special Care

Censored survival data are not equivalent to ordinary continuous observations.

Therefore, permutation procedures for survival endpoints must preserve the relevant censoring and randomization structure.

A naive permutation method that ignores informative censoring, stratification, or other design features may not have the intended type I error.

Practical point: For confirmatory survival analyses, permutation methods should be justified carefully rather than applied simply because the endpoint is non-normal.

Exact vs. Monte Carlo Permutation Tests

Feature Exact Permutation Monte Carlo Permutation
Number of permutations All allowable permutations Random subset
Simulation error None Yes
Computational cost Can become very large Controlled by \(B\)
Reproducibility Deterministic Requires a random seed
Suitable for huge permutation spaces Often no Yes

When Exact Enumeration Becomes Impossible

Consider a trial with 100 patients and 50 assigned to treatment.

The number of possible assignments is:

\[ {100\choose50} \]

This number is astronomically large.

It is therefore impossible to enumerate every permutation in practice.

Instead, a Monte Carlo procedure might generate:

\[ B=10{,}000 \]

or:

\[ B=100{,}000 \]

randomly selected allowable permutations.

Monte Carlo Standard Error

The estimated permutation p-value itself is subject to Monte Carlo variability.

If the true permutation probability is approximately \(p\), the standard error of the estimated proportion is approximately:

\[ SE(\hat p) \approx \sqrt{ \frac{p(1-p)}{B} } \]

For example, if \(p\approx0.05\) and \(B=10{,}000\):

\[ SE(\hat p) \approx \sqrt{ \frac{0.05(0.95)}{10{,}000} } \approx0.00218 \]

Thus, increasing the number of permutations improves the numerical precision of the Monte Carlo p-value.

Why Very Small p-Values Require Many Permutations

Suppose the desired p-value is approximately 0.001.

With only 1,000 random permutations, there may be very few extreme permutations.

With 100,000 permutations, the simulation has substantially more resolution.

This is one reason computationally intensive permutation tests often use tens or hundreds of thousands of permutations.

Rule of thumb: The number of permutations should be large enough that Monte Carlo error is small relative to the precision needed for the statistical conclusion.

Permutation p-Values Are Discrete

Exact permutation tests produce a finite set of possible p-values.

If there are 252 possible permutations, the basic one-sided p-value can only change in increments of:

\[ \frac{1}{252} \approx0.00397 \]

Therefore, an exact permutation p-value is not necessarily a continuous quantity.

This is another reason why an exact p-value may not equal a familiar value such as exactly 0.0500.

Permutation Test Workflow

1
Define the clinical endpoint and treatment-effect statistic.
2
State the null hypothesis being tested.
3
Determine which treatment assignments are exchangeable under the null.
4
Calculate the observed test statistic.
5
Generate every allowable permutation or a large Monte Carlo sample.
6
Recalculate the statistic for every permutation.
7
Compare the permuted statistics with the observed statistic.
8
Calculate the permutation p-value.
9
Quantify Monte Carlo error when simulation is used.
10
Interpret the result in the context of the clinical endpoint and trial design.

A Complete Worked Example in R

Consider the 10-patient example above.

The endpoint is change from baseline, and the treatment-effect statistic is the difference in means.

y <- c(8, 6, 9, 7, 5, 2, 3, 1, 4, 2)

z <- c(
  1, 1, 1, 1, 1,
  0, 0, 0, 0, 0
)

obs_stat <- mean(y[z == 1]) -
  mean(y[z == 0])

obs_stat

The observed difference is:

# 4.6

Generate All Exact Permutations

For five treatment assignments among ten patients, there are only 252 possible treatment assignments.

treatment_sets <- combn(
  seq_along(y),
  sum(z == 1)
)

n_perm <- ncol(treatment_sets)

n_perm

This returns:

# 252

Calculate the Permutation Distribution

perm_stats <- numeric(n_perm)

for (j in seq_len(n_perm)) {

  idx_t <- treatment_sets[, j]

  idx_c <- setdiff(seq_along(y), idx_t)

  perm_stats[j] <-
    mean(y[idx_t]) -
    mean(y[idx_c])
}

head(perm_stats)

Each value in perm_stats represents the treatment difference that could have been observed under one permissible treatment assignment.

Calculate an Exact One-Sided p-Value

p_perm <-
  mean(perm_stats >= obs_stat)

p_perm

The p-value is the proportion of allowable assignments producing a treatment difference at least as large as the observed difference.

Calculate a Two-Sided p-Value

p_perm_two_sided <-
  mean(abs(perm_stats) >=
       abs(obs_stat))

p_perm_two_sided

The absolute value makes the procedure sensitive to departures in either direction.

The Plus-One Exact/Monte Carlo Convention

If a Monte Carlo sample of permutations is used, a common implementation is:

B <- 10000

set.seed(2026)

perm_stats_mc <- numeric(B)

for (b in seq_len(B)) {

  idx_t <- sample(
    seq_along(y),
    size = sum(z == 1),
    replace = FALSE
  )

  idx_c <- setdiff(
    seq_along(y),
    idx_t
  )

  perm_stats_mc[b] <-
    mean(y[idx_t]) -
    mean(y[idx_c])
}

b_extreme <- sum(
  perm_stats_mc >= obs_stat
)

p_mc <- (b_extreme + 1) /
  (B + 1)

p_mc

A Reusable R Function

It is often useful to create a general function rather than rewriting the permutation algorithm for every endpoint.

permutation_test <- function(
  y,
  z,
  B = NULL,
  alternative = c("two.sided",
                  "greater",
                  "less")
) {

  alternative <- match.arg(alternative)

  n <- length(y)

  statistic <- function(y, z) {
    mean(y[z == 1]) -
      mean(y[z == 0])
  }

  observed <- statistic(y, z)

  n_t <- sum(z == 1)

  if (is.null(B)) {

    sets <- combn(
      seq_len(n),
      n_t
    )

    permuted <- apply(
      sets,
      2,
      function(idx_t) {

        z_star <- integer(n)
        z_star[idx_t] <- 1

        statistic(y, z_star)
      }
    )

    if (alternative == "greater") {

      p_value <-
        mean(permuted >= observed)

    } else if (alternative == "less") {

      p_value <-
        mean(permuted <= observed)

    } else {

      p_value <-
        mean(
          abs(permuted) >=
          abs(observed)
        )
    }

  } else {

    permuted <- numeric(B)

    for (b in seq_len(B)) {

      z_star <-
        sample(
          z,
          size = n,
          replace = FALSE
        )

      permuted[b] <-
        statistic(y, z_star)
    }

    if (alternative == "greater") {

      b_extreme <-
        sum(permuted >= observed)

    } else if (alternative == "less") {

      b_extreme <-
        sum(permuted <= observed)

    } else {

      b_extreme <-
        sum(
          abs(permuted) >=
          abs(observed)
        )
    }

    p_value <-
      (b_extreme + 1) /
      (B + 1)
  }

  list(
    statistic = observed,
    p_value = p_value,
    permutation_statistics = permuted
  )
}

Using the Function

result <- permutation_test(
  y = y,
  z = z,
  alternative = "greater"
)

result$statistic
result$p_value

The result contains the observed treatment statistic, the permutation p-value, and the permutation distribution.

Visualizing the Permutation Distribution

A simple histogram can be used to display the reference distribution.

hist(
  result$permutation_statistics,
  breaks = 30,
  main = "Permutation Distribution",
  xlab = "Permuted treatment effect"
)

abline(
  v = result$statistic,
  lwd = 3
)

The observed treatment effect appears as a vertical reference line.

If the observed statistic lies far into the tail of the permutation distribution, the permutation p-value will be small.

Permutation Tests and Confidence Intervals

A permutation test directly provides a hypothesis test, but investigators often also want an effect estimate and confidence interval.

For example, the observed treatment effect might be:

\[ \hat\Delta = \bar Y_T-\bar Y_C \]

The permutation p-value answers a hypothesis-testing question, while a confidence interval addresses uncertainty around the estimated treatment effect.

Do not report only the p-value. Clinical interpretation should generally include the estimated treatment effect, an appropriate measure of uncertainty, and the p-value or other inferential result.

Permutation Tests Are Not Automatically Bootstrap Methods

Permutation tests and bootstrap methods are both resampling techniques, but they solve different problems.

Feature Permutation Bootstrap
Primary purpose Hypothesis testing Estimation of sampling variability
Basic operation Rearrange labels/signs Resample observations
Reference distribution Null/randomization distribution Estimated sampling distribution
Typical question How unusual is the observed statistic under \(H_0\)? How variable is the estimated statistic?

Both methods can be useful in clinical research, but they should not be described as interchangeable.

Permutation Tests and Parametric Tests

Permutation tests can be thought of as an alternative way to obtain a reference distribution.

Method Reference Distribution
t-test Theoretical t distribution
Chi-square test Theoretical chi-square distribution
F-test Theoretical F distribution
Permutation test Empirical permutation distribution
Randomization test Distribution induced by allowable treatment assignments

The advantage of permutation testing is that the reference distribution can be closely connected to the observed data and experimental design.

Advantages of Permutation Tests

  • Can work well with small samples.
  • Do not require a normal reference distribution for the test statistic.
  • Can accommodate many different test statistics.
  • Can be closely tied to randomized treatment assignment.
  • Can provide exact finite-sample inference in appropriate settings.
  • Can be adapted to complex statistics.
  • Can provide an intuitive empirical null distribution.

Limitations of Permutation Tests

  • They are not assumption-free.
  • Exchangeability must be justified.
  • Complex randomization schemes require careful implementation.
  • Exact enumeration can become computationally impossible.
  • Monte Carlo tests introduce simulation error.
  • Missing data can complicate the permutation scheme.
  • Covariate adjustment may require specialized methods.
  • Interpretation depends on the null hypothesis and statistic.

Common Mistake: Permuting the Outcome Instead of the Assignment

In a randomized treatment comparison, the usual permutation operation is to rearrange the treatment assignments while keeping the observed outcomes fixed.

The purpose is to recreate the assignments that could have occurred under the null.

Randomly generating entirely new outcome values would answer a different question.

Common Mistake: Ignoring the Randomization Scheme

Suppose treatment was randomized using blocks.

If an analyst randomly permutes treatment labels across the entire dataset, the resulting distribution may contain assignments that were impossible under the actual trial design.

This can invalidate the intended randomization inference.

Best practice: Document the exact permutation mechanism, including blocking, stratification, matching, clustering, or other restrictions imposed by the trial design.

Common Mistake: Treating a Small p-Value as Proof of a Clinical Effect

A permutation p-value measures the extremeness of the observed statistic under the specified null framework.

It does not measure:

  • The probability that the treatment works
  • The probability that the null hypothesis is true
  • The clinical importance of the treatment effect
  • The probability that the result will replicate

Clinical interpretation still requires consideration of effect size, precision, safety, endpoint relevance, and study design.

Common Mistake: Assuming More Permutations Change the Statistical Answer

Increasing the number of Monte Carlo permutations primarily reduces simulation error.

It does not change the underlying null hypothesis or the scientific meaning of the statistic.

For example, increasing:

\[ B=10{,}000 \]
to:

\[ B=1{,}000{,}000 \]

can produce a more precise estimate of the same permutation probability, but it does not make an invalid permutation scheme valid.

Common Mistake: Using Permutation Tests Because the Data Are "Non-Normal"

Non-normality alone does not automatically imply that permutation testing is the correct solution.

The more fundamental question is: what permutations are justified by the study design and null hypothesis?

For a randomized trial, the randomization mechanism may provide a compelling basis for permutation inference.

For observational data, however, unrestricted treatment-label permutation is generally not justified because treatment assignment was not randomized.

Permutation Tests in Observational Studies

Permutation methods can sometimes be used in observational research, but exchangeability must be justified carefully.

In an observational study, treatment groups may differ systematically in:

  • Age
  • Disease severity
  • Baseline risk
  • Comorbidities
  • Prior treatment

Simply shuffling treatment labels can destroy these relationships and produce an inappropriate null distribution.

Randomized vs. observational data: Permutation testing has a particularly natural interpretation when treatment assignment was randomized. In observational studies, additional structure or modeling is generally required to justify the permutation scheme.

Permutation Tests in Adaptive Trials

Adaptive clinical trials introduce another consideration.

If treatment assignment probabilities change during the trial, the permutation distribution must reflect the adaptive randomization mechanism.

The relevant reference distribution is not necessarily the set of all assignments with equal probability.

Instead, assignments should be weighted or generated according to the actual design when randomization-based inference is intended.

Permutation Tests and Interim Analyses

Repeatedly calculating an ordinary permutation p-value at multiple unplanned interim analyses can create the same type of multiplicity problem encountered with repeated conventional hypothesis tests.

If interim decisions are planned, the overall testing procedure should account for the sequential design.

A permutation method does not automatically eliminate the need for multiplicity or type I error control.

Permutation Tests and Multiple Endpoints

Clinical trials often have multiple endpoints.

Suppose investigators evaluate:

  • Overall survival
  • Progression-free survival
  • Objective response rate
  • A biomarker endpoint

Running a separate permutation test for every endpoint does not automatically control the family-wise type I error rate.

Permutation methods can, however, be incorporated into multiplicity-adjusted procedures by constructing joint permutation distributions for appropriate sets of statistics.

Permutation-Based Multiple Testing

Suppose there are \(K\) test statistics:

\[ T_1,T_2,\ldots,T_K \]

For each permutation, calculate all \(K\) statistics.

The resulting joint distribution can be used to evaluate the maximum statistic:

\[ T_{\max} = \max_{k=1,\ldots,K}|T_k| \]

This can provide a basis for simultaneous inference under suitable assumptions and permutation schemes.

Permutation Tests and Missing Data

Missing observations present another important challenge.

Suppose treatment assignment is known for every patient, but the primary endpoint is missing for some patients.

A naive permutation test that simply drops missing observations may alter the effective randomization structure.

The appropriate approach depends on:

  • The estimand
  • The missing-data mechanism
  • The analysis population
  • The randomization scheme
  • The prespecified analysis method

Permutation inference should therefore be integrated with the missing-data strategy rather than treated as a completely separate computational step.

Permutation Tests and Protocol Prespecification

If permutation testing is to be used for a confirmatory clinical endpoint, the statistical analysis plan should specify the procedure in advance.

At minimum, document:

  • Primary endpoint
  • Estimand
  • Null hypothesis
  • Test statistic
  • Permutation mechanism
  • Randomization restrictions
  • Number of permutations
  • Exact vs. Monte Carlo implementation
  • One-sided or two-sided testing
  • p-value convention
  • Random-number seed for reproducible simulations
  • Handling of missing observations
  • Handling of ties
  • Multiplicity adjustments

Reproducibility of Monte Carlo Permutation Tests

Monte Carlo procedures use random numbers.

Therefore, a statistical programmer should set and document a random-number seed.

set.seed(20260913)

The exact seed is less important than having a documented, reproducible procedure.

Ties in Permutation Distributions

Discrete clinical endpoints can produce many tied permutation statistics.

For example, binary response data may generate only a relatively small number of distinct treatment-effect values even though many treatment assignments are possible.

The p-value calculation must therefore specify how equality is handled.

For a one-sided test, a common rule is:

\[ T^{*}\ge T_{\mathrm{obs}} \]

rather than strictly:

\[ T^{*}>T_{\mathrm{obs}} \]

The choice should be prespecified and used consistently.

One-Sided vs. Two-Sided Permutation Tests

A one-sided alternative might be:

\[ H_A:\Delta>0 \]

while a two-sided alternative is:

\[ H_A:\Delta\ne0 \]

For a two-sided permutation test, a common approach is to compare absolute statistics:

\[ |T^{*}|\ge|T_{\mathrm{obs}}| \]

The definition should be established before examining the treatment effect.

Interpreting a Permutation p-Value

Suppose the permutation p-value is:

\[ p_{\mathrm{perm}}=0.018 \]

A reasonable interpretation is:

Under the specified null hypothesis and permutation scheme, approximately 1.8% of allowable treatment assignments produce a test statistic at least as extreme as the observed statistic.

This provides evidence against the null hypothesis.

It does not mean there is a 98.2% probability that the treatment is effective.

Permutation Test vs. Bootstrap: A Clinical Example

Suppose a treatment reduces a biomarker by an estimated 12 units.

A permutation test might ask:

How unusual would a treatment difference of at least 12 units be if treatment assignment had no effect?

A bootstrap procedure might instead ask:

How variable is the estimated 12-unit treatment difference across samples resembling the observed study?

The two procedures therefore answer different inferential questions.

When a Permutation Test Is Particularly Attractive

  • A randomized trial provides a clear treatment-assignment mechanism.
  • The sample size is relatively small.
  • The endpoint has a difficult or unusual distribution.
  • A conventional parametric reference distribution is questionable.
  • A scientifically meaningful custom test statistic is available.
  • An exact test is computationally feasible.
  • The randomization structure can be faithfully reproduced.

When a Permutation Test May Not Be Appropriate

  • The observations are not exchangeable under the proposed permutations.
  • The treatment assignment mechanism is ignored.
  • The data contain substantial clustering that is not preserved.
  • The analysis involves complex censoring that the permutation scheme does not handle appropriately.
  • The procedure is being used simply because the endpoint is non-normal.
  • The observational nature of the data makes unrestricted label permutation unjustified.
  • The permutation test does not correspond to the estimand of interest.

A Practical Clinical-Trial Decision Framework

1
Was treatment assignment randomized?
2
What exact null hypothesis is being tested?
3
Which observations or treatment assignments are exchangeable under that null?
4
What test statistic best represents the clinical question?
5
Can all allowable permutations be enumerated?
6
If not, how many Monte Carlo permutations are needed?
7
How will blocking, stratification, matching, or clustering be preserved?
8
How will missing observations and ties be handled?
9
How will multiplicity and interim analyses be handled?
10
How will the result be reported alongside the treatment effect and uncertainty?

What Should Be Reported in a Statistical Analysis Plan?

A permutation test should be described sufficiently precisely that an independent statistical programmer could reproduce it.

A strong SAP description should include:

  • The clinical estimand
  • The null and alternative hypotheses
  • The endpoint definition
  • The treatment-effect statistic
  • The randomization/permutation mechanism
  • Any blocking or stratification restrictions
  • The number of permutations
  • Whether the test is exact or Monte Carlo
  • The p-value calculation
  • The direction of the alternative
  • Rules for ties
  • Handling of missing data
  • Multiplicity adjustment
  • Random-number seed, if applicable

Worked Example Summary

Component Example
Study Randomized two-arm clinical trial
Endpoint Continuous change from baseline
Treatment sample size 5
Control sample size 5
Observed treatment mean 7.0
Observed control mean 2.4
Observed difference 4.6
Number of exact assignments 252
Reference distribution All allowable treatment assignments
One-sided statistic \(\bar Y_T-\bar Y_C\)
Two-sided statistic \(|\bar Y_T-\bar Y_C|\)

The Most Important Concept

The most important point is that a permutation test is not simply a technique for avoiding normality assumptions.

Its fundamental logic is:

\[ \text{Observed data} \longrightarrow \text{allowable rearrangements under }H_0 \longrightarrow \text{null distribution} \longrightarrow p\text{-value} \]

The quality of the inference therefore depends on whether the rearrangements are scientifically and statistically justified.

In a randomized clinical trial, the treatment assignment mechanism provides a particularly strong foundation for this approach.

The test statistic can then be compared with the distribution generated by valid alternative assignments.

Bottom line: Permutation tests construct an empirical null distribution by rearranging treatment assignments, signs, or other exchangeable components of the data. They can provide exact finite-sample inference when all allowable permutations are enumerated and can be extended through Monte Carlo sampling when exact enumeration is computationally infeasible. In clinical trials, the permutation scheme should respect the actual randomization design, including blocking, stratification, matching, and other restrictions. Permutation tests are powerful tools, but they are not assumption-free: exchangeability, the null hypothesis, the test statistic, and the randomization mechanism must all be considered carefully.

References

Fisher, R.A. (1935). The Design of Experiments. Oliver and Boyd.
Good, P. (2005). Permutation, Parametric and Bootstrap Tests of Hypotheses. 3rd ed. Springer.
Lehmann, E.L. & Romano, J.P. (2005). Testing Statistical Hypotheses. 3rd ed. Springer.
Pesarin, F. & Salmaso, L. (2010). Permutation Tests for Complex Data: Theory, Applications and Software. Wiley.
Higgins, J.J. (2004). Introduction to Modern Nonparametric Statistics. Brooks/Cole.
Edgington, E.S. & Onghena, P. (2007). Randomization Tests. 4th ed. Chapman & Hall/CRC.
Winkler, A.M., Ridgway, G.R., Webster, M.A., Smith, S.M. & Nichols, T.E. (2014). Permutation inference for the general linear model. NeuroImage, 92, 381–397.