Why Permute Subject Pairings?
Two omics blocks measured on the same subjects have a natural alignment: row \(i\) of \(X\) and row \(i\) of \(Y\) describe the same subject. CCA or two-block PLS can find directions in the blocks with strong observed association. But a large fitted association does not, by itself, show how surprising that association would be if the subject matches carried no information. The fitting procedure searches over directions, so even unrelated blocks can yield a nonzero in-sample leading association.
A permutation test constructs a reference distribution by keeping the observed rows in each block but randomly breaking their pairings. It asks whether the leading association for the real matching is unusually large relative to associations produced by alternative matchings. Its validity depends on whether those matchings are interchangeable under the null, not simply on whether a permutation is easy to compute.
Choosing the Statistic and Defining the Null
The statistic should match the association method and the question. For CCA, a natural leading statistic is the largest canonical correlation, which measures the largest correlation attainable between linear combinations of the two blocks under the method's constraints. For two-block PLS, a natural statistic is the leading singular value of the cross-block covariance matrix, or of the cross-correlation matrix when the variables have been standardized. The statistic is nonnegative in either case, so unusually large values indicate stronger leading association.
For example, with centered, standardized variables, let \(R_{XY}\) be the sample cross-correlation matrix. The leading two-block PLS association statistic can be defined as \(T=\sigma_1(R_{XY})\), the largest singular value. With CCA, the corresponding statistic is the leading canonical correlation calculated using the specified covariance estimates and any regularization. These are not interchangeable numerical quantities: choose the statistic before testing and use the same definition for every permutation.
A permutation test of this kind is naturally justified under a null in which the block vectors are independent across blocks, so that the pairing carries no association. Independence is stronger than zero cross-covariance: nonlinear dependence can exist even when all cross-covariances are zero. The test's sensitivity also depends on the statistic; a leading linear association statistic is not a universal detector of every possible form of dependence.
The requirement is about the sampling or assignment structure. If subjects differ systematically by batch, family, site, or another design feature that affects the blocks, arbitrary permutations can create pairings that would not be plausible under the null. Repeated measurements also cannot generally be treated as independent rows: the unit being permuted must respect the dependence structure. When exchangeability holds only within strata, permute within those strata and define the resulting conditional null accordingly.
Computing a Permutation p-value
If all allowed permutations can be enumerated, calculate the statistic for each one, including the observed pairing. For a large permutation space, draw \(B\) random allowed permutations. In both cases, count a permutation as at least as extreme when its statistic is greater than or equal to the observed statistic; using “greater than or equal” handles ties conservatively for an upper-tail test.
Here \(T_{\mathrm{obs}}=T(X,Y)\), and \(T_b\) is the statistic from the \(b\)th random permutation. The plus-one form for Monte Carlo sampling includes the observed arrangement as the reference arrangement; it avoids reporting a p-value of zero just because none of a finite set of random permutations exceeded the observation. A small p-value means that statistics at least this large are unusual under the specified permutation null. It is not the probability that the null is true.
Worked Examples
Worked Example: An Exact Test With One Variable in Each Block
Suppose three hypothetical subjects have centered measurements \(X=(1,2,3)\) and \(Y=(1,2,3)\). Use the absolute sample correlation, \(T=|r|\), as the leading association statistic. For these data, both sample standard deviations use the same centered values \((-1,0,1)\), so the observed correlation is
There are \(3!=6\) possible pairings. The correlations for the six reorderings of \(Y\) are \(1,-1,0.5,0.5,-0.5,-0.5\). Their absolute values are \(1,1,0.5,0.5,0.5,0.5\). Thus two of the six statistics are at least the observed value, and
Although the observed matching has perfect correlation, the exact test does not provide strong evidence against the permutation null: with only three subjects, a perfect absolute correlation occurs for both the increasing and decreasing matchings. The example also shows why a magnitude-based statistic must count both signs as equally extreme.
Worked Example: A Multivariate PLS-SVD Test
Consider a hypothetical data set of \(n=30\) matched subjects with two standardized variables in each block. Suppose the observed cross-correlation matrix is \(R_{XY}=\begin{pmatrix}0.60&0.10\\0.20&0.50\end{pmatrix}\). Use the leading singular value as the two-block PLS statistic. Its square is the largest eigenvalue of \(R_{XY}^{\mathsf T}R_{XY}\):
Therefore \(T_{\mathrm{obs}}=\sqrt{0.5046}\approx0.7104\). Now suppose 999 randomly sampled subject-pairing permutations produce 23 statistics at least \(0.7104\). The Monte Carlo p-value is
Under the stated exchangeability assumption, this is evidence against the null of independent block vectors: an association statistic at least as large as the observed value was uncommon under the randomized pairings. The calculation does not establish that the association is causal, and it does not show that every variable contributes meaningfully to the leading component.
Worked Example: Permuting Within Strata
Suppose a hypothetical study has six subjects split across two sites, with three subjects per site. Site is a design feature that must be preserved, and the null permits pairing exchangeability only within each site. A fixed leading-association analysis gives \(T_{\mathrm{obs}}=0.62\). The allowed permutations are the \(3!\) reorderings at the first site combined with the \(3!\) reorderings at the second, for \(3!\times3!=36\) assignments. If exact enumeration finds four statistics at least \(0.62\), then
The conclusion is conditional on the within-site permutation scheme: the observed statistic is not especially unusual among matchings that retain site. Permuting across all six subjects instead would answer a different question and could break the exchangeability condition by pairing observations from unlike sites.
Why the Whole Analysis May Need to Be Repeated
A valid permutation comparison must account for the process that produced the observed statistic. If the analysis uses a fixed, prespecified ridge penalty and a defined CCA statistic, apply that same procedure to every permuted data set. If the analysis selects a penalty, chooses features, selects a sparsity level, or chooses between CCA and PLS by examining the paired data, that selection can increase the observed association. To calibrate the complete selection procedure, repeat the data-dependent selection within each permutation and calculate the resulting leading statistic.
Unsupervised transformations determined separately within each block may sometimes be held fixed if they do not use the cross-block pairing. But any step that used the paired association, including supervised feature screening, must be treated as part of the procedure being tested. The safe principle is to specify the analysis pipeline and distinguish fixed choices from choices informed by the observed cross-block match.
Choose the association method, its leading statistic, and whether the test is upper-tail. State the null and the allowed permutation scheme.
Identify the exchangeable unit and any strata, clusters, repeated measures, or batch restrictions. Do not permute across units that are not exchangeable under the null.
Break the allowed pairings and calculate the same statistic, repeating any data-dependent selection that is part of the analysis.
Count permutation statistics at least as large as observed, calculate the appropriate p-value, and report the statistic, permutation scheme, number of permutations, and interpretation of the null.
Why the Permutation Comparison Is Justified
Under the null and the exchangeability condition, the observed pairing and all allowed reassigned pairings have the same distribution. Consider first the exact test. If the possible statistics are distinct, the rank of the observed statistic among the statistics from all allowed pairings is uniformly distributed under the null. Therefore the chance that this rank falls among the most extreme fraction \(\alpha\) is at most \(\alpha\). Tied statistics, counted using “greater than or equal to,” make the test conservative rather than anti-conservative.
This argument does not require the statistic to follow a normal or \(t\) distribution. It does require the permutation scheme to represent the null assignments correctly. With sampled rather than exhaustive permutations, the plus-one estimate reflects the finite random sample of allowed alternatives; its precision depends on \(B\). For example, a p-value near \(0.02\) estimated from only a few dozen permutations is necessarily coarse, so a larger \(B\) is appropriate when a more stable tail estimate is needed.
Diagnostics, Alternatives, and Limitations
Before interpreting a result, check that the permuted rows preserve each block's marginal data and that the pairing alone changes. Verify that the statistic is calculated identically each time and that the permutation distribution is not dominated by coding errors, failed model fits, or different preprocessing. In sparse or regularized analyses, confirm that the fitting and selection steps are repeated whenever they depended on the observed pairing. When permutations are restricted, report the restrictions and the conditioning they imply.
A permutation test is not automatically valid because it is nonparametric. Unrestricted shuffling can be wrong for clustered subjects, repeated observations, or systematic batch differences. Residualizing covariates and then shuffling residuals is also not a universally valid fix; residual exchangeability depends on the model and design. A suitable design-specific scheme or a different inferential method may be needed.
An asymptotic test uses a reference distribution justified by a model or large-sample approximation and may be computationally cheaper, but it can be unreliable when its assumptions or approximations are poor. A bootstrap typically resamples observations to assess sampling variability; resampling paired subjects preserves their pairing and does not, by itself, create a null distribution of no cross-block association. Permutation is attractive when the assignment structure is clear and the full statistic can be recomputed, but it tests the specified permutation null, not every conceivable absence of scientific relationship.
Common Mistakes
- Permuting individual entries instead of subject rows. This destroys within-block structure. Reorder whole rows of one block so that each block retains its observed multivariate rows.
- Assuming that any row permutation is valid. This ignores exchangeability. Restrict permutations to valid units or strata, or use a design-appropriate alternative.
- Comparing the observed optimized statistic with null statistics from a simpler fit. This gives the observed analysis an unfair search advantage. Apply the same optimization and repeat data-dependent selection under each permutation.
- Interpreting a small p-value as proof of a meaningful or causal relationship. It is evidence against the stated permutation null, not a measure of practical importance or causation.
- Calling a result significant because no sampled permutation exceeded it. With \(B\) random permutations, use the plus-one formula; the estimate is not zero.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- An analysis uses \(B=499\) random permutations, of which 14 statistics are at least as large as observed. What is the Monte Carlo p-value?
- Why is permuting \(Y\)'s individual feature values independently generally not the same as permuting its subject rows?
- A feature-screening step uses the observed \(X\)-\(Y\) association before fitting sparse CCA. What must happen to that step during each permutation?
- In a study with two sites, exchangeability is plausible only within site. What permutation scheme is appropriate, and what does its test condition on?
Solutions. (1) \((1+14)/(499+1)=15/500=0.030\). (2) Row permutation preserves the observed joint pattern among features within each \(Y\) subject; independently shuffling entries changes that within-block structure and no longer tests only the subject matching. (3) Repeat the screening inside every permutation because it used the paired association and is part of the data-adaptive procedure being calibrated. (4) Permute whole \(Y\) rows separately within each site. The test compares the observed pairing with alternative pairings that retain site membership, so its null is conditional on those site strata.