Rotation Tests for Self-Contained Enrichment

Why Rotate Residuals?

A pathway can contain genes that move together across samples. If a test treats their gene-level statistics as independent, their shared variation can make the evidence appear stronger than it is. Rotation tests address this dependence by constructing null data sets through a shared transformation of the sample-level residuals. Because every gene receives the same rotation, the procedure retains important features of the joint residual data while testing whether the pathway as a whole responds to a specified contrast.

The target is self-contained: the null says that none of the genes in the set has an effect for the contrast. It does not ask whether the set has stronger effects than genes outside it. A rotation p-value therefore supports evidence against that all-genes-null hypothesis; it is not by itself a measure of pathway specificity or biological importance.

Model and Rotation Construction

Suppose expression for \(n\) independent biological samples is modeled for each gene \(g\) by a linear model. Write \(y_g\) for the \(n\)-vector of expression measurements, \(X_0\) for the design matrix containing nuisance terms under the null, and \(\beta_{0g}\) for the corresponding coefficients. The null model is \(y_g=X_0\beta_{0g}+\varepsilon_g\). For example, if the full model includes treatment and batch, but the tested contrast is treatment, the null model retains batch while excluding the treatment effect.

Let \(P_0\) be the projection onto the column space of \(X_0\). The fitted null values are \(P_0y_g\), and the residuals are \(r_g=(I-P_0)y_g\). Collect the residual vectors as columns of an \(n\)-by-\(|S|\) matrix \(R\). A rotation \(Q\) acts on the sample dimension within the residual subspace. It is orthogonal there, so it preserves lengths and angles in that subspace. A rotated data set is formed by adding the rotated residuals to the null fitted values, then refitting the full model and recalculating gene-level statistics.

Here \(Y\) contains expression columns for the genes being tested; \(Q\) acts as an orthogonal transformation on the residual subspace and leaves the fitted null space unchanged. The same \(Q\) is used for every gene in the set. A different independent rotation for each gene would fail to retain their joint residual structure.

The basic exactness argument requires a suitable null error distribution. A standard sufficient model is independent samples with Gaussian errors that are spherical across samples, allowing arbitrary covariance among genes. In matrix terms, errors have identity covariance across the sample dimension, up to a common scale, while the gene dimension can have a general covariance matrix. Under this model, conditional on the null fit and residual cross-products, residual orientation in the residual subspace is rotation-invariant. Rotating that orientation therefore generates valid null replicates. If sample errors are not spherical—for example, because of unmodeled heteroskedasticity or dependence—the rotation reference may not be valid without an appropriate model or transformation.

What the Shared Rotation Preserves

The shared rotation has a precise algebraic property. Since \(Q\) is orthogonal on the residual subspace, \(Q'Q\) acts as the identity there. Thus the residual cross-product matrix is unchanged:

In particular, the uncentered cross-product between residual columns \(r_g\) and \(r_h\), \(r_g'r_h\), is preserved. Their normalized uncentered cross-product (the cosine of the angle between the vectors) is preserved as well. This helps retain the dependence structure needed for pathway-level inference.

Do not turn that statement into a claim that usual centered Pearson correlations across residual coordinates are necessarily preserved. Such correlations subtract each residual vector's coordinate mean before calculating the cross-product; a rotation can change those means. For instance, rotating \(z_1=(1,0)'\) and \(z_2=(1,2)'\) by 90 degrees gives \((0,1)'\) and \((-2,1)'\). The uncentered cross-product remains 1, but the centered Pearson correlation across the two coordinates changes from \(-1\) to \(+1\). The valid preservation claim is about uncentered cross-products and normalized uncentered correlations, not automatically about centered Pearson correlations.

Set-Level Statistics and Direction

After each rotation, gene-level statistics are recomputed from the full model. For a contrast, a signed \(t\)-statistic is useful: positive values indicate effects in the contrast's positive direction, and negative values indicate the opposite direction. The set-level statistic determines which pattern the test is designed to detect.

• Mean signed statistic: \(T_{\text{mean}}=m^{-1}\sum_{g\in S}t_g\), where \(m=|S|\). Large positive values support a coordinated positive response; small values in the lower tail support a coordinated negative response. The two directions can be tested separately.
• Mean squared statistic: \(T_{\text{msq}}=m^{-1}\sum_{g\in S}t_g^2\). This is large when the set contains substantial effects in either direction, even if positive and negative effects cancel in the signed mean. It does not, on its own, identify which direction predominates.
• Maxmean statistic: define \(U=m^{-1}\sum_{g\in S}\max(t_g,0)\) and \(D=m^{-1}\sum_{g\in S}\max(-t_g,0)\). Then \(T_{\text{maxmean}}=\max(U,D)\). The statistic can respond to a coherent subset moving up or down without requiring the whole set to have one common direction. Inspect whether \(U\) or \(D\) is larger to describe the observed direction.

The empirical rotation p-value is the fraction of null reference statistics at least as extreme as the observed one, with a finite-simulation correction. For an upper-tail test with \(B\) rotations and \(b\) rotated statistics greater than or equal to the observed statistic, use \(p=(b+1)/(B+1)\). For a lower-tail test, count rotated statistics less than or equal to the observed statistic. For two-sided tests, define the two-sided extremeness rule in advance—for example, compare absolute values when the statistic is a signed mean—and apply the same rule to every rotation.

Worked Examples

Justification, Diagnostics, and Alternatives

The core justification can be summarized in three steps. Under the self-contained null, the reduced model contains the correct mean structure. Under spherical errors, the residual orientation is invariant to orthogonal transformations in its residual subspace. Consequently, the observed residual orientation and its rotated versions are exchangeable under the null, conditional on the preserved residual cross-products. Recomputing the statistic for each version gives its null reference distribution. The plus-one p-value avoids reporting zero merely because a finite set of rotations happened not to produce a more extreme statistic.

Diagnostics are informative but cannot prove rotation invariance. A few influential samples can dominate residual orientation and set statistics; examine residual plots and sample-level influence. If residual spread changes systematically with fitted values, the spherical assumption is doubtful. If repeated measures or other sample dependencies are ignored, treating samples as freely rotatable is not justified. Very small residual degrees of freedom also mean few distinct directions are available, so p-values can be coarse and model estimates unstable.

Rotation tests differ from gene-set permutations, which reshuffle gene membership and typically address a competitive question. They also differ from phenotype-label permutations, which require an appropriate exchangeability scheme for the sample labels and design. A rotation test instead uses the fitted sample-level model and rotates residuals under its null. A parametric test of a set summary is another option, but its calibration may rely on an assumed reference distribution for the summary and may handle gene dependence less directly. Each approach targets a particular null and depends on its own assumptions; they are not interchangeable merely because each returns a pathway p-value.

Rotation inference has limits. It tests the specified set and contrast, not whether the pathway is uniquely more responsive than the rest of the genome. The result depends on gene-level statistics and the chosen set summary. A mean can conceal opposing effects; a mean-squared statistic can signal effects without giving their direction; maxmean can emphasize a subset rather than a uniform pathway-wide shift. Finally, a raw pathway p-value does not account for testing many pathways. Interpret it alongside the model, the statistic, the rotation count, and the broader analysis plan.

Common Pitfall

Wrong reasoning: “The shared rotation preserves the residual cross-products, so it preserves Pearson correlations between gene residuals.” This confuses uncentered cross-products with centered correlations. Pearson correlation subtracts each vector's coordinate mean, and those means can change under rotation. Correction: state exactly what is preserved: the uncentered residual cross-products \(r_g'r_h\) and their normalized cosines. Do not claim centered Pearson correlation is preserved unless its invariance has been separately established.

A second frequent error is to call a small self-contained p-value evidence that the pathway is more enriched than other genes. The null tested here concerns effects within the set. A comparison with genes outside the set is a different, competitive question.