Permutation Tests for Correlation

Why Permute Responses to Test Correlation?

A sample correlation describes the direction and strength of a linear pattern in paired quantitative data. But an observed correlation can arise from chance variation even when the variables have no underlying association. A permutation test asks how unusual the observed correlation would be if the response values had no systematic connection to the predictor values.

The procedure keeps the predictor values fixed and rearranges the responses among them. Each rearrangement makes a new set of pairs, and its correlation is calculated. The resulting correlations form a null distribution against which the observed correlation can be compared. This is a test of association based on the particular data and assumptions; it does not establish that changes in one variable cause changes in the other.

Statistic, Hypotheses, and Assumptions

Let \(x_i\) be the predictor value and \(y_i\) the response value for observational unit \(i\), for \(i=1,\ldots,n\). The usual statistic is Pearson’s sample correlation, \(r\). It measures linear association and is dimensionless, so it has no physical units. It is defined when both variables have nonzero sample variation.

For a two-sided test of no association, the hypotheses are \(H_0\): the predictor and response are independent, versus \(H_a\): they are associated. A positive-direction alternative asks whether the association is positive; a negative-direction alternative asks whether it is negative. The direction must be chosen before examining the permutation results. The alternative is not simply “the sample correlation is nonzero”: the null model used here is independence, which is stronger than a population correlation of zero.

The permutation calculation relies on conditions that make response reassignment reasonable:

• Paired quantitative measurements: each observational unit contributes one predictor and one response value, and the statistic is defined for the data.
• Independent observational units: the pairs come from units that can reasonably be treated as independent. Repeated measurements on the same unit or clustered observations require a different analysis.
• Exchangeable responses under the null: under \(H_0\), the response values have the same distribution across the fixed predictor values. In particular, independence of response and predictor supports this condition for independent observations.
• No unaccounted-for structure in the pairings: the data collection must not impose a pairing, time order, or group structure that makes some response assignments more plausible than others under the null.

These assumptions are about the data and null model, not just the arithmetic. For instance, if measurements are recorded in time order and nearby responses tend to be similar, freely shuffling them may break a real dependence structure. A permutation distribution is only a suitable reference when the allowed rearrangements reflect the null model.

How the Permutation Distribution Is Built

If all \(n!\) assignments are enumerated, the test is exact under the stated null assumptions. For a two-sided test based on correlation, extremeness is measured by absolute value: count assignments with \(|r^*|\geq |r_{\text{obs}}|\). For a positive-direction test, count assignments with \(r^*\geq r_{\text{obs}}\); for a negative-direction test, count assignments with \(r^*\leq r_{\text{obs}}\). In each case, the exact p-value is the proportion of all enumerated assignments meeting that criterion.

If only a random sample of \(B\) permutations is generated, the resulting p-value is a Monte Carlo estimate. A common calculation is \((b+1)/(B+1)\), where \(b\) is the number of simulated statistics at least as extreme as the observed statistic. The added one includes the observed arrangement in the comparison and avoids reporting a p-value of zero just because no simulated permutation was as extreme.

Why This Null Distribution Is Appropriate

The key justification is that, under independence, the assignment of response values to fixed predictor values is not favored in any particular order. Conditional on the observed predictor values and the observed response values, each assignment is equally likely when responses are exchangeable under the null. Thus the observed pairing is one of the assignments represented in the permutation distribution.

More explicitly, if the \(n\) response values are treated as distinguishable, there are \(n!\) possible assignments. Under the null, each assignment has probability \(1/n!\). The fraction of assignments whose statistic is at least as extreme as the observed statistic is therefore the null probability of seeing a statistic that extreme or more extreme. This fraction is the exact permutation p-value. If some response values are tied, distinct labeled assignments can produce the same numerical arrangement; counting all labeled assignments still gives the correct weighting under the same exchangeability condition.

This argument explains both the appeal and the limitation of the method: it avoids a normal-theory model for the sampling distribution of correlation, but it does not avoid assumptions about how observations were generated. Nor does it mean that every permutation test is automatically valid; the response shuffling must match the null model.

What the Test Can and Cannot Detect

Pearson correlation is designed to summarize linear association. A permutation test using \(r\) uses that statistic, even though the null distribution is generated without assuming a normal distribution for \(r\). If a strong curved relationship has a correlation near zero, a test based on Pearson’s \(r\) may not detect it well. A non-significant result therefore does not establish that the variables are independent in every practical or scientific sense.

Conversely, the null hypothesis of independence implies zero population correlation when the relevant expectations exist, but zero correlation alone does not generally imply independence. A permutation test calibrated for independence should not be described as a test of only “zero correlation” unless additional model assumptions justify that interpretation. Always state the null model actually used.

The p-value is a probability calculated under \(H_0\): it is the probability of obtaining a statistic at least as extreme as the observed one, according to the chosen tail rule and permutation scheme. It is not the probability that \(H_0\) is true, nor the probability that the result occurred “by chance” without qualification. The size of \(r_{\text{obs}}\) communicates the observed linear association; the p-value addresses its compatibility with the null model. They answer different questions.

Common Mistakes

• Permuting paired rows together. This preserves the original pairing and does not create the null distribution described here. Keep the predictor values fixed and reassign the response values across them.
• Using a one-sided tail after seeing the result. Choosing the direction after observing \(r_{\text{obs}}\) makes the reported p-value misleading. Select the alternative before analyzing the permutation distribution, or use a two-sided test for an undirected question.
• Calling zero correlation the same as independence. They are not equivalent in general. State whether the test’s null is independence or, under a specified model, a zero population correlation.
• Interpreting a small p-value as causation. A permutation test for observational pairs can provide evidence of association under its assumptions; it does not by itself establish a causal effect.
• Ignoring dependence among observations. If responses are clustered, repeated, or ordered in a way relevant under the null, unrestricted shuffling can give an inappropriate reference distribution.