Rank-Based Competitive Enrichment Tests

Why Rank Gene-Level Scores?

A gene set may be enriched for strong gene-level evidence even when its scores are on different scales across analyses. Ranking the eligible genes offers a simple comparison: does the set tend to appear nearer the strong-evidence end of the ordered list than a set of the same size chosen from the background? This is a competitive question because the set is compared with genes outside it, not merely tested for evidence within the set.

A rank test discards information about the distances between scores. It uses their ordering, so a small difference and a large difference between adjacent scores both contribute one step in rank. That can make the test less sensitive to extreme score magnitudes, but the result still depends on the score definition, the eligible-gene universe, and the direction in which genes are ranked.

Definition: Let the eligible universe contain $N$ genes, each with one gene-level score. Choose a direction so that stronger evidence corresponds to a smaller rank: rank 1 is strongest, and rank $N$ is weakest. For a gene set containing $m$ eligible genes, its rank sum is $W=\sum_{i\in S}R_i$, and its mean rank is $\bar{R}_S=W/m$, where $R_i$ is gene $i$'s rank. Smaller-than-background ranks indicate competitive enrichment for stronger scores.

The comparison is conditional on the chosen universe and on the set size $m$. Genes not in the eligible universe do not contribute ranks or count toward $m$. Scores must be oriented consistently before ranking: for example, if a smaller p-value represents stronger evidence, it must receive a smaller rank. When a score has the opposite direction, reverse its ranking first. The test concerns the score ordering, not the biological correctness of the gene mapping or the causal interpretation of the genes.

The Competitive Null and Its Expected Rank

The rank-based competitive null treats set membership as exchangeable among the $N$ eligible genes: conditional on the observed ranks and the fixed set size $m$, each subset of $m$ genes is equally plausible under the null. In words, membership in the named set does not preferentially select genes with stronger or weaker ranks. This is a null model for a competitive comparison, not a claim that no gene in the set is associated with the phenotype.

Null expectation: With no ties, ranks are $1,\ldots,N$. Under the competitive null, the expected rank sum and expected mean rank for a set of size $m$ are $$ E(W)=\frac{m(N+1)}{2}, \qquad E(\bar{R}_S)=\frac{N+1}{2}. $$ Thus, the expected mean rank does not depend on set size, while the expected rank sum does.

The null expectation follows because each gene has probability $m/N$ of being included in a uniformly selected set of size $m$. Therefore, the expected rank sum is the sum of all ranks multiplied by $m/N$:

E(W)=\frac{m}{N}\sum_{r=1}^{N}r =\frac{m}{N}\frac{N(N+1)}{2} =\frac{m(N+1)}{2}.

Dividing by $m$ gives the expected mean rank, $(N+1)/2$. A mean rank below this value points toward stronger scores in the set; a mean rank above it points toward weaker scores. The difference in mean rank is descriptive. To quantify how unusual the observed rank sum is under the null, its variability must also be considered.

For a random set of fixed size $m$, the no-tie null variance of the rank sum is

\operatorname{Var}(W)=\frac{m(N-m)(N+1)}{12}.

One justification uses an indicator $I_i$, equal to 1 if gene $i$ is selected and 0 otherwise. Under uniform sampling of $m$ genes, $E(I_i)=m/N$, $\operatorname{Var}(I_i)=m(N-m)/N^2$, and for distinct genes $i$ and $j$, $\operatorname{Cov}(I_i,I_j)=-m(N-m)/[N^2(N-1)]$. Writing $W=\sum_i I_iR_i$, using these variances and covariances, and noting that the ranks have mean $(N+1)/2$ and squared deviations summing to $N(N^2-1)/12$, gives

\operatorname{Var}(W) =\frac{m(N-m)}{N(N-1)} \sum_{i=1}^{N}\left(R_i-\frac{N+1}{2}\right)^2 =\frac{m(N-m)(N+1)}{12}.

The variance shrinks as $m$ approaches $N$: a set containing nearly every eligible gene has less freedom to differ from the background. For a quick standardized summary, one may compare the observed sum with its null expectation using $Z=[W-E(W)]/\sqrt{\operatorname{Var}(W)}$. With ranks oriented so that smaller is stronger, a negative $Z$ points toward enrichment. A normal approximation to this statistic is not an exact finite-sample probability; when the set is small or the rank distribution is discrete, the exact conditional distribution can differ noticeably.

Worked Example: Rank Sum Under a No-Tie Null

Suppose eight eligible genes have distinct scores, ranked from strongest to weakest as $1,\ldots,8$. A hypothetical set contains genes with ranks 1, 2, and 5. Calculate its rank sum and mean rank, compare them with their null expectations, and find the exact one-sided probability of a rank sum this small or smaller under the competitive null.

The set size is $m=3$, the universe size is $N=8$, and the observed ranks sum to

W=1+2+5=8,\qquad \bar{R}_S=\frac{8}{3}=2.667.

Under the null, the expected rank sum is $3(8+1)/2=13.5$, and the expected mean rank is $9/2=4.5$. The observed mean rank of 2.667 is below 4.5, in the direction of stronger scores. The no-tie null variance is $3(8-3)(8+1)/12=11.25$, so the standard deviation is $\sqrt{11.25}=3.354$. The standardized difference is $(8-13.5)/3.354=-1.640$.

For the exact one-sided probability, there are $\binom{8}{3}=56$ equally likely subsets. The rank triples with sum at most 8 are $(1,2,3)$, $(1,2,4)$, $(1,2,5)$, and $(1,3,4)$: four subsets. Thus the exact conditional p-value is $4/56=0.07143$. This is evidence in the enrichment direction, but it is not below 0.05. The normal approximation would give a different tail probability, illustrating why the exact discrete calculation can matter for a small set.

Tied Scores and Midranks

Distinct ranks are not appropriate when two or more genes have identical scores. Assign each tied gene the average of the rank positions that the tied genes would occupy. These are called midranks. For example, if two genes tie for positions 2 and 3, each receives rank $(2+3)/2=2.5$. Midranks preserve the average of the original rank positions, so the mean of all $N$ gene ranks remains $(N+1)/2$. Consequently, the null expectations above remain valid with ties.

Ties do change the variability of the rank sum. For tie blocks of sizes $t_1,t_2,\ldots$, the sum of squared deviations of the midranks from their mean is

\sum_{i=1}^{N}(R_i-\bar{R})^2 =\frac{N(N^2-1)-\sum_j(t_j^3-t_j)}{12}, \qquad \bar{R}=\frac{N+1}{2}.

The corresponding conditional null variance is

\operatorname{Var}(W) =\frac{m(N-m)}{N(N-1)} \left[\frac{N(N^2-1)-\sum_j(t_j^3-t_j)}{12}\right].

This is the same fixed-size sampling variance derived above, using the squared deviations of the actual midranks. The term $\sum_j(t_j^3-t_j)$ reduces the variance relative to distinct ranks. Ties therefore affect not only the reported ranks but also the null distribution used to assess how unusual the rank sum is. If all scores tie, every gene receives the same midrank, every set has the expected mean rank, and there is no rank-based evidence of competitive enrichment.

Worked Example: Midranks and a Tie-Corrected Null

Consider seven hypothetical genes ordered from strongest to weakest score: A has score 9, B and C each have score 8, D and E each have score 5, F has score 2, and G has score 1. A hypothetical set contains A, B, and C. Assign midranks and calculate its rank-sum deviation and tie-corrected null variance.

Gene A is alone in rank 1. B and C share positions 2 and 3, so each receives rank 2.5. D and E share positions 4 and 5, so each receives rank 4.5. F and G receive ranks 6 and 7. The set's rank sum is $W=1+2.5+2.5=6$, and its mean rank is $6/3=2$. The null expectations are $E(W)=3(7+1)/2=12$ and $E(\bar{R}_S)=4$. The observed sum is 6 below its expectation, indicating a shift toward stronger scores.

There are two tie blocks of size 2, so $\sum_j(t_j^3-t_j)=2(2^3-2)=12$. The squared deviations of all ranks sum to $[7(7^2-1)-12]/12=27$. Therefore,

\operatorname{Var}(W)=\frac{3(7-3)}{7(7-1)}(27)=7.714, \qquad \frac{W-E(W)}{\sqrt{\operatorname{Var}(W)}} =\frac{6-12}{\sqrt{7.714}}=-2.160.

For an exact lower-tail calculation, there are $\binom{7}{3}=35$ equally likely gene subsets. Only the subset containing A, B, and C has a midrank sum at most 6, so the exact one-sided probability is $1/35=0.02857$. The tie correction describes the variability under the stated random-membership null; it does not make the tied scores distinct or restore score information that ranking has discarded.

Unequal Set Sizes and Interpretation

The rank sum grows with set size even when the average rank is unchanged. Comparing raw rank sums across differently sized gene sets is therefore misleading: the appropriate null expectation for each set is $m(N+1)/2$. Mean rank makes the direction easier to compare across sizes, but uncertainty also depends on size. At a fixed difference between observed and expected mean ranks, a larger set will generally have a more precise rank-sum comparison under the exchangeable null.

Worked Example: Why Set Size Changes the Comparison

In a hypothetical universe of $N=10$ genes with distinct ranks, consider two sets. Set A has ranks 1 and 5; Set B has ranks 1, 2, 3, 4, and 5. Compare their mean-rank deviations and standardized rank sums. The sets overlap, which is allowed for this comparison.

Set A has $m=2$, $W=1+5=6$, and mean rank $6/2=3$. Its expected sum is $2(11)/2=11$, so the observed sum is 5 below expectation. Its null variance is $2(10-2)(11)/12=14.67$, giving $Z=(6-11)/\sqrt{14.67}=-1.305$.

Set B has $m=5$, $W=1+2+3+4+5=15$, and mean rank $15/5=3$, the same as Set A. Its expected sum is $5(11)/2=27.5$, so the observed sum is 12.5 below expectation. Its null variance is $5(10-5)(11)/12=22.92$, giving $Z=(15-27.5)/\sqrt{22.92}=-2.611$.

Both sets have mean rank 3, which is 2.5 ranks below the background expectation of 5.5. Their raw sums, 6 and 15, cannot be compared directly because the set sizes differ. The standardized values differ because the larger set provides more information about its rank distribution under the fixed-size null. These calculations use a normal-scale summary; an exact p-value would be based on the conditional distribution for each set size.

Assumptions, Scope, and Common Pitfalls

The rank-based competitive test is straightforward only when its comparison is well defined. Before interpreting it, verify that each eligible gene has one score, the score direction is set correctly, and the membership labels correspond to the same eligible universe used for ranking. The null assumes exchangeability of set membership across that universe conditional on the observed scores and set size. If some genes are systematically more likely to enter a set for reasons related to their scores, the simple exchangeable null may not represent the intended comparison.

A rank test answers whether the set tends to occupy stronger or weaker positions relative to the eligible background. It does not estimate how much larger the original scores are, and a significant competitive result does not establish that every member is associated or that the set causes the phenotype. Conversely, failure to find a rank shift is not proof that no member of the set is relevant.

Wrong reasoning: “The set's rank sum is small, so it must be enriched.” Correction: Compare the sum with its size-specific null expectation and variability; smaller sums are not unusual merely because a set has fewer genes.
Wrong reasoning: “Two tied genes can be ordered arbitrarily without consequence.” Correction: Give tied genes midranks and account for ties in the null variance or exact rank distribution.
Wrong reasoning: “A low mean rank proves strong gene-level effects.” Correction: It establishes a tendency toward stronger positions in this ranked list, not a magnitude on the original score scale or a causal effect.
Wrong reasoning: “A rank result has the same meaning for any background.” Correction: It is conditional on the specified eligible-gene universe; changing that universe can change both ranks and the competitive comparison.

Key takeaway: Rank-based competitive enrichment compares a gene set's rank sum or mean rank with the null distribution for a uniformly selected set of the same size from a defined eligible universe. The null mean rank is $(N+1)/2$; ties require midranks and a tie-aware variance, and unequal set sizes require size-specific interpretation.

Check Your Understanding

Use ranks in which rank 1 denotes the strongest gene-level evidence.

For $N=20$ eligible genes and a set of $m=4$, what are the null expected rank sum and null expected mean rank?
Why does the competitive null condition on the set size?
Three tied genes occupy rank positions 6, 7, and 8. What midrank should each receive?
Two gene sets have the same mean rank but different sizes. Why might their standardized rank sums differ?
What does a significantly low rank sum support, and what does it not establish?