Two Questions Hidden in the Word “Enrichment”
A pathway can appear active because its genes have evidence of change, because they show stronger evidence than genes outside the pathway, or both. These are different claims. A pathway whose genes all have similar moderate scores may stand out sharply from a mostly inactive background, even if the evidence is not strong enough to reject the hypothesis that all pathway effects are zero. Conversely, a pathway may contain genes with clear evidence of change while many genes outside it show similarly strong evidence.
The distinction is about the null hypothesis and its reference, not about two names for the same test. A self-contained test compares pathway genes with a no-effect reference. A competitive test compares pathway genes with genes outside the pathway. The same expression data and pathway can therefore yield different p-values without either calculation being contradictory.
The self-contained null is \(H_0:\beta_g=0\) for every \(g\in S\). Its alternative is that at least one pathway gene has a nonzero effect.
A competitive null says that pathway membership does not identify genes with stronger evidence of association than genes outside \(S\). The precise test depends on how “stronger evidence” is measured and on the reference distribution used. A common reference compares the observed pathway statistic with statistics for other gene sets of size \(m\), formed from the tested-gene universe.
These definitions use the same tested-gene universe that is required for over-representation analysis, but neither null is the hypergeometric null for a thresholded list. Here we work with continuous gene-level scores. Let \(Z_g\) be a signed, standardized score for the specified expression contrast, with positive values indicating increased expression in the chosen direction and negative values indicating decreased expression. A self-contained test based on squared scores asks whether any pathway gene departs from zero in either direction; a one-sided competitive test based on mean signed scores asks whether the pathway tends to have larger positive scores than the background.
Different Reference Distributions
A standard self-contained construction is available when the gene-level scores are independent standard normal variables under the complete pathway null. This is an explicit model assumption, not a consequence of standardizing scores alone. Under it, the sum of squared pathway scores has a chi-square distribution with \(m\) degrees of freedom. The test is sensitive to effects in either direction, since squaring removes the sign.
A competitive random-set test instead conditions on the observed scores for the tested genes and compares the pathway’s score statistic to the same statistic for all (or many sampled) subsets of \(G\) of size \(m\). For a positive-direction analysis, one possible statistic is the average signed score, \(\bar Z_S=m^{-1}\sum_{g\in S}Z_g\). Its one-sided p-value is the fraction of size-\(m\) subsets whose average score is at least as large as the observed pathway average. This calibration is exact under the reference model that, under the competitive null, the pathway’s \(m\) genes are exchangeable with the tested genes for this score statistic. It is a model for comparison, not a claim that biological pathway membership was literally randomized.
The competitive statistic and direction must be chosen to match the scientific question. For example, a signed mean is appropriate for a specified direction; a two-sided or magnitude-based statistic would answer a different question. A random-set test of the mean is not a universal test of every possible difference between pathway and background score distributions. Likewise, the self-contained chi-square test above is one particular test of the all-zero null, not the only possible self-contained procedure.
Worked Example: Competitive Evidence Without a Self-Contained Rejection
Worked Example: Competitive Evidence Without a Self-Contained Rejection
Suppose a hypothetical expression analysis tests \(N=20\) genes. Four genes in pathway \(S\) have score \(Z_g=1.5\), and the other 16 genes have score \(Z_g=0\). Treat the scores as independent standard normal under the self-contained null. For the competitive calculation, use the positive-direction random-set mean and regard all size-four subsets as equally likely under its exchangeability reference.
For the self-contained test, \(m=4\), so the observed statistic and its null distribution are
For four degrees of freedom, the upper-tail probability is \(P(\chi^2_4\ge 9)=e^{-9/2}(1+9/2)\approx0.0611\). At significance level \(\alpha=0.05\), the self-contained test does not reject the all-zero pathway null.
For the competitive test, the observed mean is \(T_S=1.5\). A size-four subset can have mean at least 1.5 only if it contains all four genes with score 1.5: any subset containing a zero has mean below 1.5. There is one such subset among \(\binom{20}{4}=4,845\) subsets. Thus
The pathway is significant by the competitive reference but not by the self-contained test at the 0.05 level. The scores are modest relative to the specified self-contained null, yet the pathway’s scores are unusually high relative to the rest of this hypothetical gene universe. This conclusion is conditional on both stated calibrations; it is not evidence that the pathway is biologically causal.
Why the Tests Can Disagree
A self-contained test asks whether the pathway would be surprising if all its effects were zero. Its evidence depends on pathway scores relative to that null distribution. A competitive test asks whether this particular set is unusually strong compared with other sets of the same size drawn from the tested-gene universe. If background genes are also affected, the competitive contrast can be weak even when the pathway itself is clearly non-null. Conversely, if pathway genes have modest scores but background genes mostly have weak scores, the pathway may look highly competitive while not crossing a self-contained threshold.
The difference can be summarized as a change in reference, not necessarily a change in the observed data:
| Feature | Self-contained formulation | Competitive formulation |
|---|---|---|
| Reference | No effect for every gene in the pathway | Scores for pathway genes compared with scores outside it or with size-matched gene sets |
| Core question | Does any pathway gene depart from its no-effect model? | Does this pathway show stronger evidence than the tested-gene background? |
| Typical calibration | Gene-level null model, such as a justified chi-square distribution for independent normal scores | Random-set or other competitive reference distribution |
| Potential sensitivity | Effects in the pathway, including effects shared by much of the transcriptome | Pathway-specific excess relative to the background |
Worked Example: Strong Pathway Effects but No Competitive Contrast
In a second hypothetical data set, suppose the 20 tested genes all have \(Z_g=2\), and the pathway again contains four genes. Under the independent standard-normal self-contained null, its statistic is
The self-contained test rejects at \(\alpha=0.05\). But every size-four subset has mean score 2, exactly the pathway’s observed mean. The one-sided competitive random-set p-value is therefore \(1\). The pathway has substantial evidence against no effect, but no score advantage over the background in this constructed example. Competitive nonsignificance does not mean that the pathway genes have no effects.
Justifying the Self-Contained Calibration
The chi-square result follows directly from the assumed score distribution. If the self-contained null is true and the pathway scores are independent \(N(0,1)\), then each squared score \(Z_g^2\) has a chi-square distribution with one degree of freedom. The sum of \(m\) independent chi-square variables with one degree of freedom has a chi-square distribution with \(m\) degrees of freedom. Therefore \(Q_S=\sum_{g\in S}Z_g^2\sim\chi^2_m\).
The assumptions matter. Independent scores with mean zero and variance one are not automatically normal, and their squared sum is not automatically chi-square. If the score distribution is not normal, if the standard errors are not correctly calibrated, or if scores are dependent, this reference distribution may not give the stated p-value. In particular, correlation among genes can alter the variability of the sum. The independence model is useful for demonstrating the distinction between the null formulations, but it should not be treated as a default guarantee for real expression data.
Worked Example: Checking the Exact Competitive Calculation
Worked Example: Checking the Exact Competitive Calculation
Keep the first hypothetical data set: four scores are 1.5 and 16 are zero. Now consider a different size-four gene set containing one score of 1.5 and three zeros. Its mean is \(1.5/4=0.375\), and its self-contained statistic is \(1.5^2=2.25\). Its self-contained p-value is
For the one-sided competitive test, subsets with mean at least 0.375 are precisely those containing at least one of the four nonzero-score genes. Of the 4,845 size-four subsets, \(\binom{16}{4}=1,820\) contain no such gene. Hence
Neither test is significant. This example also shows why the competitive p-value is not obtained by testing the pathway mean against zero: it is calculated relative to the size-matched sets in the observed gene universe.
Diagnostics, Limitations, and Common Mistakes
Before interpreting a result, check that the gene universe contains the genes that could actually have been tested in the expression analysis, and that pathway membership is mapped consistently to that universe. The random-set calculation is changed by the choice of universe. Also confirm that the score and direction match the question: a signed score mean can cancel positive and negative changes, whereas a squared-score self-contained statistic can detect either direction but does not itself describe direction.
A competitive result relies on a meaningful comparison with the background. If genes outside the set are not an appropriate reference, a competitive p-value may answer a formally correct but scientifically unhelpful question. A self-contained result likewise depends on a valid gene-level null calibration. The simple chi-square calculation requires independent standard normal scores under the null; it should not be applied merely because the scores were called “z-scores.” Discrete, skewed, estimated, or correlated scores may require a different calibration. A nominal p-value also does not account for testing many pathways; the same significance threshold cannot be interpreted as controlling the overall false-positive rate across a large pathway collection.
- Mistake: treating the nulls as interchangeable. A pathway can reject the all-zero null without being stronger than the background, or be strong relative to the background without rejecting its all-zero null. State the reference in words before interpreting the p-value.
- Mistake: claiming that competitive nonsignificance proves no pathway effects. It only means the chosen competitive statistic was not unusually large against its reference at the selected threshold.
- Mistake: assuming standardization proves a chi-square null. Unit variance alone does not imply normality or independence. Verify the distributional assumptions or use a calibration appropriate to the scores.
- Mistake: saying enrichment means causation. Enrichment describes evidence in a set of genes under a statistical reference model; it does not establish that the pathway causes the observed phenotype.
Check Your Understanding
Answer each question using the null formulation and reference distribution, not only the word “enriched.”
- A pathway has scores above zero, but its competitive test is not significant. What can and cannot be concluded?
- Under independent standard normal scores, a pathway of three genes has \(Z\)-scores 2, 0, and 0. Calculate its self-contained statistic and state its null distribution.
- In a size-matched competitive random-set test, why is the observed pathway mean compared with other sets of the same size rather than directly with zero?
- Name two assumptions needed for the chi-square calibration \(Q_S\sim\chi^2_m\).
- What is the interpretation of a significant self-contained test and a nonsignificant competitive test in the same analysis?
Solutions. (1) There is not sufficient evidence that the pathway is stronger than the tested-gene background under the chosen competitive test; this does not show that all pathway effects are zero. (2) \(Q_S=2^2+0^2+0^2=4\), with a \(\chi^2_3\) null distribution if the three scores are independent standard normal under the null. (3) The competitive question concerns relative strength: the size-matched sets provide the reference for what a pathway of that size could score in the observed gene universe. (4) The pathway scores must be independent and standard normal under the all-zero null; correct score calibration is also required. (5) The data provide evidence against no effect for at least one pathway gene, but do not show that the pathway is stronger than the background by the chosen competitive measure.