Residual Permutations With Estimated Nuisance Effects

Why Residual Permutations Need Care

Permutation tests are attractive because they can avoid specifying a parametric distribution for a test statistic. In a linear model with nuisance effects, however, the observed response is not just an error vector: it contains a fitted nuisance component. Permuting the raw responses may scramble that component and create a reference distribution for the wrong problem. Residual permutation tries to remove the nuisance fit first, but the residuals themselves are altered by estimating that fit.

The distinction matters especially for a multivariate response. Each observational unit contributes a row of responses, and a permutation should usually move that row as a unit. The method’s calibration then depends on whether the residuals have the required permutation symmetry, or whether the use of fitted residuals is only a large-sample approximation.

Model and the Freedman–Lane Construction

Let \(Y\) be an \(n\times q\) response matrix, with one row per observational unit and \(q\) response variables. Write the full linear model as

Here \(X_0\) is the \(n\times p_0\) nuisance design, \(X_1\) is the \(n\times p_1\) design for the effect of interest, and \(B_0,B_1\) are coefficient matrices. The null hypothesis is \(H_0:B_1=0\). Assume the combined design has the intended rank, and define the reduced model using \(X_0\) alone. Let \(P_0\) be the orthogonal projection onto the column space of \(X_0\), and let \(M_0=I-P_0\). The reduced fitted values and residuals are \(P_0Y\) and \(R_0=M_0Y\).

The observed statistic is computed from \(Y\), and the reference distribution is formed from the statistics computed on the permuted pseudo-responses. For a complete enumeration of a finite permutation group \(\mathcal G\), a common upper-tail p-value is \( |\mathcal G|^{-1}\sum_{\pi\in\mathcal G} I\{T(Y^{(\pi)})\geq T(Y)\}\). If only \(B\) random permutations are drawn, the usual finite-simulation version includes the observed arrangement and uses \((1+\sum_{b=1}^B I\{T(Y^{(b)})\geq T(Y)\})/(B+1)\). These formulas handle ties conservatively when the comparison includes equality.

What Is Being Compared?

The residual scheme determines which part of the fitted data is held fixed and which part is rearranged. The table summarizes common choices; their names do not by themselves guarantee valid calibration.

Raw-response permutation is appropriate only when the null distribution is invariant to the proposed row permutations, including the nuisance structure. Full-model residual permutation is a distinct procedure, not another name for Freedman–Lane. Because the full fit has used the effect being tested, its residuals need not represent the null errors in the same way that reduced-model residuals are intended to. Neither alternative should be adopted merely because it is computationally convenient.

When Is Residual Permutation Exact?

A finite-sample exact permutation test needs a genuine symmetry under the null. The following sufficient condition makes precise one setting where Freedman–Lane has that symmetry. A permutation group \(\mathcal G\) is a collection of permutation matrices closed under composition and inverse. Assume the error matrix is invariant in distribution under every allowed permutation: \(P_\pi E\overset{d}=E\). Independent, identically distributed error rows are a familiar sufficient condition.

Proof. The column-space condition implies that \(P_\pi P_0=P_0P_\pi\), and therefore \(P_\pi M_0=M_0P_\pi\). Under the null, \(Y=X_0B_0+E\), so \(R_0=M_0E\). For each allowed permutation, \(P_\pi R_0=P_\pi M_0E=M_0P_\pi E\overset{d}=M_0E=R_0\). Thus the reduced residual matrix is invariant in distribution under the group. The test statistic is unchanged by the nuisance fitted term \(P_0Y\), since that term lies in the column space of \(X_0\). Consequently the observed statistic and the statistics over the permutation orbit are governed by the same group symmetry. Under a finite group, an invariant distribution assigns equal probability to arrangements within each orbit, so the rank of the observed statistic among its orbit values is uniform in the absence of ties and conservative when ties are counted in the upper tail. This gives the stated level bound. \(\square\)

The condition is sufficient, not necessary. With an intercept-only nuisance design, every row permutation preserves the nuisance column space, so iid error rows give this symmetry. For a general covariate design, most permutations do not preserve its column space, and the theorem does not establish exactness.

Why Calibration Is Often Approximate

For a general nuisance design, reduced residuals are projected errors, not the original errors. Under independent rows with common covariance matrix \(\Sigma\), their covariance across observational units is \(\operatorname{Cov}(R_{0,i\cdot},R_{0,j\cdot})=(M_0)_{ij}\Sigma\). Unless \(M_0\) has the symmetry required by the allowed permutations, the residual rows are not exchangeable. A permutation of them then creates a reference distribution that does not have the exact finite-sample null law.

There is a useful approximation argument, but it is not an automatic validity theorem. If the nuisance design has fixed rank \(r\), the error rows are independent with mean zero and covariance \(\Sigma\), then

Indeed, the expectation is \(\operatorname{tr}(\Sigma)\operatorname{tr}(P_0)/n\), and \(\operatorname{tr}(P_0)=r\). If \(r\) stays fixed as \(n\) grows, this quantity tends to zero. Thus projection changes the errors by a vanishing amount on average squared per observation. This helps explain why residual permutations can work approximately in some large samples, especially with low-dimensional nuisance fits and suitably regular, studentized statistics. It does not prove that a particular permutation test is asymptotically valid: leverage, design imbalance, heteroscedasticity, dependence, and the statistic’s behavior all matter.

For an error distribution invariant under row permutations, iid rows are a standard assumption; equal means and covariances alone are not enough. If row distributions differ, or if errors are dependent, unrestricted residual permutations generally lack the needed symmetry. More specialized permutation restrictions require a design-specific justification and should not be inferred from the Freedman–Lane formula alone.

Common Mistake

The practical value of the distinction is substantial. A well-chosen residual scheme can protect the nuisance fit while allowing a flexible statistic and a data-driven reference distribution. But reporting a permutation p-value does not, by itself, establish that the p-value is exact. State which residuals were permuted, whether whole response rows moved together, what assumptions were made about the errors, and whether calibration is exact or approximate under those assumptions.