Testing Pathway Enrichment and Depletion

Why the Direction of a Pathway Test Matters

A pathway can contain more selected genes than expected, fewer selected genes than expected, or an overlap that is unusual in either direction. Those are different statistical questions. A one-sided enrichment test asks specifically about excess overlap; a one-sided depletion test asks about a deficit; a two-sided association test allows either direction to count as evidence against no association. Choosing the tail after looking at the observed overlap changes the question and can make the reported evidence misleading.

This tutorial treats the selected genes as a fixed-size set drawn from a defined tested-gene universe. The universe and pathway must be defined consistently: genes outside the tested universe do not contribute to the comparison. The earlier material on Fisher exact tests and odds ratios provides the contingency-table representation; here the emphasis is on selecting, calculating, and interpreting the correct tail.

Definition: Let $M$ be the number of genes in the tested universe, $K$ the number of those genes in a pathway, and $n$ the number of selected genes. Let $X$ be the number of selected genes that belong to the pathway. Under the null model that the $n$ selected genes are a uniformly chosen subset of size $n$ from the $M$ genes, $X$ has a hypergeometric distribution: $$ P(X=x)=\frac{\binom{K}{x}\binom{M-K}{n-x}}{\binom{M}{n}}. $$ The possible values of $x$ are the integers from $\max(0,n-(M-K))$ to $\min(K,n)$. The expected overlap under this null is $E[X]=nK/M$.

The uniform-subset null is a modeling assumption, not a claim that genes were literally selected by drawing names from a hat. It says that, conditional on the universe and the total number selected, pathway membership does not make a gene more or less likely to be selected. The model also assumes that pathway membership and selection status are correctly recorded and that the chosen universe contains the genes eligible for selection.

Three Hypotheses, Three Interpretations

For a particular observed overlap $x_{\text{obs}}$, an enrichment test assigns evidence to outcomes at least as large as the observed overlap. A depletion test assigns evidence to outcomes at least as small. A two-sided test considers outcomes that are unusual in either direction. The central null in each case is no association between pathway membership and selection, represented in the $2\times2$ table by an odds ratio of 1.

Test directions:

Enrichment: $H_0$ is no pathway-selection association; $H_a$ is enrichment, meaning selection is more common among pathway genes. The exact upper-tail p-value is $P(X\geq x_{\text{obs}})$.

Depletion: $H_0$ is no pathway-selection association; $H_a$ is depletion, meaning selection is less common among pathway genes. The exact lower-tail p-value is $P(X\leq x_{\text{obs}})$.

Two-sided association: $H_0$ is no association; $H_a$ is an association in either direction. A common Fisher exact convention sums the probabilities of all tables with fixed margins whose null probabilities are no greater than the probability of the observed table. Other two-sided conventions exist, so the convention used should be reported.

The corresponding table has selected pathway genes $a=x_{\text{obs}}$, selected non-pathway genes $b=n-a$, unselected pathway genes $c=K-a$, and unselected non-pathway genes $d=M-K-b$. When all four cells are positive, the sample odds ratio is $ad/(bc)$: a value above 1 points toward enrichment, and a value below 1 points toward depletion. A zero cell can yield an estimated odds ratio of zero or infinity; this does not prevent calculating a conditional exact test.

The tail probabilities follow directly from the null distribution. For example, the enrichment p-value is the sum of the probabilities of all possible overlaps at least as large as the observed one:

p_{\text{enrich}}=\sum_{x=x_{\text{obs}}}^{\min(K,n)} \frac{\binom{K}{x}\binom{M-K}{n-x}}{\binom{M}{n}}.

This is a p-value because it is the probability, under the specified null model, of an overlap at least as far in the enrichment direction as the observed overlap. It is not the probability that the null hypothesis is true. The depletion formula instead sums from the smallest possible overlap through $x_{\text{obs}}$.

Worked Exact Tests

Worked Example: Testing for Enrichment

Suppose a hypothetical tested universe contains $M=20$ genes. A pathway contains $K=6$ of them, and $n=5$ genes are selected. Four selected genes are in the pathway. The expected overlap under the null is $$ E[X]=\frac{nK}{M}=\frac{5(6)}{20}=1.5\text{ genes}. $$ The observed overlap of 4 is above that expectation, so the prespecified question is whether there is enrichment.

The table is $a=4$, $b=5-4=1$, $c=6-4=2$, and $d=20-6-1=13$. Its odds ratio is $$ \frac{ad}{bc}=\frac{4(13)}{1(2)}=26. $$ This estimated odds ratio points toward enrichment, but the exact p-value is computed from the null overlap distribution, not from the odds ratio alone.

There are $\binom{20}{5}=15{,}504$ possible subsets of five genes. For an overlap of 4, there are $\binom{6}{4}\binom{14}{1}=15(14)=210$ subsets; for an overlap of 5, there are $\binom{6}{5}\binom{14}{0}=6$. Therefore, $$ P(X\geq4)=\frac{210+6}{15{,}504} =\frac{216}{15{,}504}=0.01393. $$ At significance level $\alpha=0.05$, this is evidence against the no-association null in the enrichment direction. In context, under the uniform-subset null with these fixed margins, the chance of an overlap of at least 4 is about 1.39%. This result does not establish that pathway activity causes the selection pattern.

Worked Example: Testing for Depletion

Now suppose a hypothetical universe has $M=20$ genes and the pathway contains $K=10$. Five genes are selected, but none belongs to the pathway. Under the null, the expected overlap is $$ E[X]=\frac{5(10)}{20}=2.5\text{ genes}. $$ An observed overlap of zero is in the depletion direction. The table has $a=0$, $b=5$, $c=10$, and $d=5$, so the odds ratio is $0(5)/(5(10))=0$.

For a depletion test, include all overlaps no greater than zero; in this case the only such value is $X=0$. Thus, $$ P(X\leq0)=P(X=0) =\frac{\binom{10}{0}\binom{10}{5}}{\binom{20}{5}} =\frac{1(252)}{15{,}504}=0.01625. $$ At $\alpha=0.05$, this provides evidence of depletion under the specified null model. The p-value describes the probability of an overlap this small or smaller under the null, not the probability that pathway genes are biologically inactive.

Worked Example: A Two-Sided Association Test

Consider another hypothetical universe with $M=12$ genes. A pathway contains $K=4$, and $n=4$ genes are selected; the observed overlap is $x_{\text{obs}}=3$. The expected overlap is $4(4)/12=1.333$ genes. The table has $a=3$, $b=1$, $c=1$, and $d=7$, giving an odds ratio of $3(7)/(1(1))=21$. This points toward enrichment, but the question here is two-sided: could either excess or deficit be evidence of association?

The denominator for the overlap probabilities is $\binom{12}{4}=495$. For $x=0,1,2,3,4$, the respective numbers of subsets are $$ \binom{4}{x}\binom{8}{4-x}=70,\ 224,\ 168,\ 32,\ 1. $$ These counts sum to 495. Under the probability-ordering convention, tables with null probability no greater than the observed table's probability $32/495$ are counted. These are the tables for $x=3$ and $x=4$, so $$ p_{\text{two-sided}}=\frac{32+1}{495} =\frac{33}{495}=0.06667. $$ This convention does not yield evidence at $\alpha=0.05$, despite the large estimated odds ratio. A different two-sided convention that doubles the smaller directional tail would give $2(33/495)=0.1333$ here. The difference illustrates why a two-sided exact p-value should be interpreted with its convention specified.

Why the Tail Calculation Is Justified

The hypergeometric model arises by counting equally likely subsets under the uniform-subset null. Among the $\binom{M}{n}$ possible selected sets, a set with exactly $x$ pathway genes can be formed by choosing $x$ of the $K$ pathway genes and $n-x$ of the $M-K$ non-pathway genes. There are $\binom{K}{x}\binom{M-K}{n-x}$ such sets, giving the stated probability after division by the total number of sets. Summing probabilities in the direction specified by the alternative gives the exact conditional tail probability.

The expected overlap also follows by writing $X$ as a sum of indicators. For each pathway gene $i$, let $I_i=1$ if it is selected and $I_i=0$ otherwise. Each gene has selection probability $n/M$, so $E[I_i]=n/M$. Since $X$ is the sum over the $K$ pathway genes, $$ E[X]=E\left[\sum_{i=1}^{K}I_i\right] =\sum_{i=1}^{K}E[I_i]=K\frac{n}{M}. $$ This expectation is a useful reference point for direction, but it is not itself a significance test: the tail probability accounts for the full null distribution.

Fisher’s exact test uses the same conditional logic. Once the row and column totals of the pathway-membership-by-selection table are fixed, the overlap cell has the hypergeometric distribution above. Thus, a one-sided Fisher exact test and the corresponding one-sided hypergeometric test give the same directional tail probability when they use the same margins and tail definition.

Assumptions, Diagnostics, and Limitations

Before interpreting a result, check that the selected genes are drawn from the stated tested-gene universe, that pathway membership is defined within that universe, and that $M$, $K$, and $n$ match the table. Under the null, all subsets of size $n$ are treated as equally likely. If genes have systematically different selection probabilities, or the selection mechanism makes some genes more likely to be selected for reasons related to pathway membership, the uniform-subset reference distribution may not represent the null of interest.

A practical diagnostic is to reconstruct the table and verify its margins: $a+b=n$, $a+c=K$, and $a+b+c+d=M$. Also verify that the reported direction agrees with both the observed overlap relative to $nK/M$ and the odds ratio when it is finite. A stated enrichment result paired with a lower-tail p-value, or a depletion conclusion paired with an odds ratio above 1, is a warning to check the alternative and table orientation.

The test concerns over-representation or under-representation of selected genes relative to the chosen universe. It does not by itself show that a pathway is activated, identify a causal mechanism, or measure the magnitude of a biological response. Correlated genes, imperfect pathway annotations, and an inappropriate universe can limit the interpretation. Testing many pathways also creates a multiple-testing problem; an individual exact p-value is not automatically an adjusted p-value or a guarantee of a pathway-level discovery.

Common Mistakes and Alternatives

Choosing the tail after seeing the overlap. Reporting the smaller of the enrichment and depletion tails as though it were a prespecified one-sided test does not preserve the intended interpretation. Specify the direction before testing, or use an explicitly defined two-sided procedure.
Calling any large overlap enrichment without a reference. An overlap must be judged relative to $nK/M$ and its null distribution. A count of five could be unusually high for a small pathway and ordinary for a large one.
Equating two-sided with a single universal calculation. Exact two-sided p-values can use different conventions. State the convention, especially when reproducing a software result.
Reading a p-value as a probability that the null is true. A p-value is calculated assuming the null model; it measures how often the specified test statistic would be at least as extreme under that model.

The hypergeometric/Fisher approach is appropriate when the question is about overlap in a selected list and the fixed-size uniform-subset model is a reasonable reference. A two-sided test is useful when either direction matters and was not ruled out in advance. These tests differ from methods that analyze a ranked list or test changes across all genes; those methods answer different questions and should not be substituted merely because a directional overlap is not significant.

Key takeaway: Match the alternative to the scientific question before calculating: upper tail for enrichment, lower tail for depletion, and a stated two-sided convention for association in either direction. Interpret the result conditional on the tested universe, pathway size, and number selected.

Check Your Understanding

Answer each question, then compare with the brief solutions.

A pathway has $K=8$ genes in a universe of $M=40$, and $n=5$ genes are selected. Which tail tests a prespecified depletion alternative?
For the same values, calculate the expected overlap under the uniform-subset null.
A table has $a=2$, $b=3$, $c=6$, and $d=9$. Calculate its odds ratio and state which direction it points.
Why can a two-sided Fisher exact p-value differ from twice a one-sided p-value?
State one assumption whose failure could make the uniform-subset hypergeometric null inappropriate.

Solutions. 1. Use the lower tail, $P(X\leq x_{\text{obs}})$, because depletion means fewer pathway genes selected than expected. 2. $E[X]=nK/M=5(8)/40=1$ gene. 3. The odds ratio is $ad/(bc)=2(9)/(3(6))=1$, so this table’s estimated odds ratio points to neither enrichment nor depletion. 4. A two-sided probability-ordering test sums probabilities of tables no more likely under the null than the observed table; this need not equal twice a directional tail because the null distribution may be discrete and asymmetric, and conventions differ. 5. For example, if genes have different selection probabilities related to pathway membership, not all subsets of size $n$ are equally likely under the null.