Why Two Enrichment Methods Can Disagree
Over-representation analysis (ORA) asks whether a thresholded list contains more genes from a gene set than expected under a specified sampling model. Ranked gene-set enrichment analysis (GSEA) instead asks whether members of a set tend to accumulate toward one end of a ranked gene list. Because one method reduces gene-level evidence to membership in a list and the other uses the ordering of genes, disagreement is not automatically a contradiction. It can reflect different features of the same data, different assumptions, or sensitivity to analytic choices.
A useful diagnosis starts by checking that the methods actually concern the same genes and gene set: use the same eligible-gene universe where appropriate, harmonize identifiers, and confirm which set members are represented. Then examine the cutoff, the set’s size, where its genes lie in the ranked list, how the ranking was constructed, and how the null distribution handles dependence among genes. A small p-value from either method does not by itself resolve these questions.
What Each Method Is Sensitive To
Suppose the universe contains \(N\) genes, a gene set contains \(K\) of them, and a thresholded list contains \(n\) genes, of which \(x\) belong to the set. Under the standard ORA null, conditional on these sizes, the selected list behaves like a uniformly chosen subset of \(n\) genes from the universe. The overlap \(X\) then has a hypergeometric distribution. Its expected value and the observed fold enrichment are:
The expected overlap is a reference, not a p-value. A one-sided ORA p-value is \(P(X\ge x)\) under this hypergeometric model. The model is most defensible when the universe correctly represents eligible genes and, under the null, selection into the list is not systematically more likely for some genes than others. Differences in expression, gene length, or measurement precision can undermine that assumption if they affect selection and are related to set membership.
A common GSEA running sum gives more weight to genes with larger absolute ranking statistics at set members; a simpler unweighted version gives each hit the same increment. For an unweighted score and a list of \(N\) genes with \(K\) set members, the running sum increases by \(1/K\) at a hit and decreases by \(1/(N-K)\) at a miss. The enrichment score (ES) is the largest positive or negative departure from zero. The chosen weighting and null procedure affect the result, so an ES alone is not a p-value.
The main contrast follows from these definitions. ORA can miss a coordinated but modest shift if few genes cross its cutoff. GSEA can detect that tendency if set members cluster in the ranking. Conversely, a handful of very strong genes may yield ORA enrichment while the remaining set members are distributed throughout the ranking, limiting the GSEA running-sum departure. Neither method is simply a more sensitive version of the other.
Worked Example: How a Cutoff Changes the ORA Summary
Consider a hypothetical ranked analysis with \(N=1{,}000\) eligible genes and a set of \(K=100\) genes. The top 40 genes include 9 set members; the top 80 include 10. Compare the fold enrichment at each cutoff.
At \(n=40\), the expected overlap is \(40(100/1{,}000)=4\). The observed proportion of set members in the list is \(9/40=0.225\), while the universe proportion is \(100/1{,}000=0.100\). Thus the fold enrichment is \(0.225/0.100=2.25\).
At \(n=80\), the expected overlap is \(80(100/1{,}000)=8\). The observed proportion is \(10/80=0.125\), giving fold enrichment \(0.125/0.100=1.25\).
The top-80 list contains just one additional set member, so the broader list has weaker fold enrichment. The counts do not establish whether either ORA result is statistically significant: that requires the appropriate hypergeometric tail probability and, in a collection of tested sets, attention to multiple testing. The comparison does show why the cutoff is consequential. A threshold should be scientifically justified, not selected after inspecting which value gives the most favorable result.
Trace the Signal Before Comparing P-Values
A running-sum calculation makes the role of ordering concrete. The next example is deliberately small so each step can be checked by hand. Its ES illustrates the statistic; it is not a basis for a reliable GSEA significance claim. In an actual analysis, inference also depends on a suitable null distribution and the design of the study.
Worked Example: A Strong GSEA Pattern With Weak ORA Evidence
Imagine 12 eligible genes ordered from most positive to least positive by a signed statistic. A set contains four genes, at positions 1, 2, 3, and 4. For an unweighted running sum, each hit adds \(1/4=0.25\), and each of the other eight genes subtracts \(1/8=0.125\).
The running sum after the first four genes is \(0.25,\ 0.50,\ 0.75,\ 1.00\). After the next eight nonmembers it decreases by \(8(0.125)=1.00\), returning to zero. The maximum positive departure is therefore \(ES=1.00\). All set members are at the positive end of this toy ranking.
Now threshold only the top two genes. The list has \(n=2\), with \(x=2\) set members. Under the ORA null, the probability of this overlap is \(P(X\ge2)=\binom{4}{2}/\binom{12}{2}=6/66=0.0909\), since two is the largest possible overlap. Thus this small example has a clear concentrated ranking pattern, but the corresponding ORA tail probability is 0.0909. The difference illustrates how a short thresholded list can limit ORA evidence even when set members cluster at the top. It does not imply that the GSEA result is significant: the ES has no permutation-based reference here.
Signal distribution matters beyond whether genes are “positive” or “negative.” A set may have a few extreme members and many ordinary members, or many modest members shifted in the same direction. A weighted GSEA score can be especially responsive to extreme statistics; an unweighted score gives hit genes equal increments. Inspecting the positions and gene-level statistics of set members helps distinguish a broad pattern from one driven by a few genes. The leading edge—the members contributing to the running-sum departure—can help describe that pattern, but should not be treated as an independent confirmation.
Worked Example: Many Set Members, but No Clear Ranked Concentration
Suppose a hypothetical analysis has \(N=1{,}000\) eligible genes, a pathway with \(K=200\) members, and a thresholded list of \(n=50\) genes containing \(x=15\) pathway members. The expected overlap is \(50(200/1{,}000)=10\), and the fold enrichment is \((15/50)/(200/1{,}000)=0.30/0.20=1.50\). There are more pathway genes in the list than the null expectation, although the size of the overlap alone does not determine its p-value.
Now suppose the 15 pathway genes are spread across the ranking: some are near the positive end, some near the middle, and some near the negative end. A threshold might collect enough members for an ORA signal, while the opposing or scattered positions prevent a large running-sum departure. This is a plausible reason for stronger ORA than GSEA evidence. To diagnose it, examine the ordered positions and signed statistics of all represented pathway genes, not just the 15 selected genes. Also check that “pathway size” means the number of pathway genes actually represented in the eligible universe, rather than a database count that includes unmeasured genes.
Ranking Choices and Dependence Among Genes
The ranking statistic should match the contrast of interest and be comparable across genes. A signed statistic can preserve direction; a statistic based only on significance or absolute magnitude may place strong effects from both directions at the same end. If the biological question concerns coordinated up-regulation, a ranking that discards sign can therefore answer a different question. Rankings based on adjusted p-values can also be awkward when many values tie or when significance depends on gene-specific precision. Check the statistic’s direction, ties, missing values, and whether a small change in its definition materially rearranges set members.
Correlation is another key source of disagreement. The hypergeometric ORA calculation treats selection as if, under its null, every subset of \(n\) genes were equally plausible. When genes are correlated, genes in a pathway can move together, so selecting several of them is not equivalent to independently drawing several genes. The usual hypergeometric tail may then misrepresent uncertainty. GSEA’s null also depends on how it is constructed: permuting gene labels or set membership can break gene-gene correlation, whereas valid phenotype-label permutations can preserve the observed correlation structure across genes. Phenotype permutations require labels to be exchangeable under the null; paired, blocked, or otherwise structured designs need permutations that respect that structure. No permutation scheme repairs an invalid study design.
Worked Example: Correlated Genes Can Make an Overlap Look Like Many Independent Events
Consider a hypothetical experiment in which 10 genes from a pathway respond together because they share a regulatory program. A thresholded list contains 8 of those 10 genes. If the eligible universe has 1,000 genes and the pathway has 20 members, an ordinary hypergeometric calculation uses the observed \(N=1{,}000\), \(K=20\), \(n\), and \(x=8\). It does not know that those eight genes may have moved together.
If the pathway genes were independent opportunities for selection, eight selected members could be interpreted as multiple pieces of evidence against random overlap. If their shared response makes their selection highly dependent, the same eight observations may largely reflect one coordinated module. The overlap and its fold enrichment remain descriptive, but the standard hypergeometric p-value relies on a sampling model that may not capture this dependence. Diagnose this possibility by examining expression correlation or known shared regulation and by checking whether the GSEA permutation preserves the relevant dependence. Do not report a numerical corrected p-value without a method and assumptions that justify it.
A Practical Diagnostic Sequence
Confirm gene identifiers, represented pathway members, eligible universe, direction of the contrast, and the list of tested pathways. Differences here can create apparent disagreement before the methods are compared.
Compare ORA at a small number of defensible, prespecified thresholds. Record list size, overlap, expected overlap, fold enrichment, and uncertainty; do not choose a cutoff solely because it produces the smallest p-value.
Check the effective set size after restricting to eligible genes. Plot or tabulate member positions and statistics to see whether signal is broad, concentrated, scattered, or mixed in direction.
Check whether direction-preserving, scientifically appropriate gene-level statistics give a similar pattern. A materially different result may reveal that the inference depends on the ranking definition.
For ORA, assess whether the universe and exchangeable-selection model are plausible. For GSEA, identify what is permuted and whether the permutation respects gene correlation and the experimental design.
These checks are diagnostic rather than a menu for searching until one method agrees with the preferred interpretation. A useful comparison keeps the biological question fixed and explains what feature each result reflects. If the discrepancy disappears under modest, scientifically reasonable choices, describe the result as sensitive. If it persists, explain the distinct signals and assumptions rather than averaging p-values or declaring one method universally correct.
Common Mistake and Limitations
Common mistake: “ORA is significant and GSEA is not, so the pathway must contain a small set of decisive genes.” That conclusion does not follow from the two p-values alone. The ORA result may depend on the chosen cutoff, the universe, or gene-level selection bias; the GSEA result may depend on ranking, weighting, set size, or a permutation scheme that does not preserve dependence. Inspect the overlap and rank positions, check assumptions, and describe what the data support without claiming causation.
Neither method removes all limitations of gene-set analysis. Pathways can overlap, databases can be incomplete, and pathway definitions may include genes that are not measurable in the study. Multiple testing affects interpretation across sets, while correlation can affect the calibration of individual tests. Results from ORA and GSEA are complementary summaries, not independent replications: they often use the same expression data and gene annotations.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In a universe of 500 genes, a set has 50 members and a thresholded list has 20 genes, including 4 set members. What are the expected overlap and fold enrichment?
- Why can a pathway have a strong ranked pattern but weak ORA evidence at a strict cutoff?
- What is one reason a standard hypergeometric ORA p-value may be poorly calibrated when genes are correlated?
- Why can a ranking based on absolute statistics answer a different question from a signed ranking?
- What should be checked before interpreting ORA–GSEA disagreement as a biological difference?
Solutions. (1) The expected overlap is \(20(50/500)=2\). The observed list proportion is \(4/20=0.20\), and the universe proportion is \(50/500=0.10\), so fold enrichment is \(0.20/0.10=2.00\). (2) Set members can cluster near one end of the ranked list without enough of them crossing a stringent threshold to make the selected list show strong over-representation. (3) Correlation can make selection of multiple set genes occur together, contrary to the uniform-subset sampling assumption used by the hypergeometric null. (4) An absolute statistic groups strong effects from both directions, whereas a signed statistic distinguishes positive from negative effects; the resulting enrichment questions differ. (5) Check gene identifiers, eligible universe, represented set size, cutoff, ranking direction and statistic, gene positions, and the validity of each method’s null assumptions.