What Does a Self-Contained Test Ask?
A pathway can be biologically interesting because its genes show evidence of association with a phenotype. A self-contained test asks whether the genes inside a specified set show any association, without comparing their evidence with that of genes outside the set. This makes its central question different from a competitive test, which asks whether the set is more strongly associated than the rest of a defined gene universe.
Suppose \(S\) is a prespecified set of \(m\) genes, and let \(\beta_g\) denote the gene-level association effect for gene \(g\). The effect may, for example, be a log-odds ratio per allele under a gene-level model. The precise gene-level score depends on the analysis that produced it; here the task is to test the evidence for the set as a whole.
The word global matters: the null is a joint statement about all genes in the set. A set can reject this null because one gene has a strong association, several genes have modest evidence, or a chosen statistic is sensitive to another pattern of effects. A rejection alone does not identify which pattern occurred.
Joint Tests of the Gene-Level Effects
When gene-level effect estimates and their covariance are available, a direct approach tests the vector of effects jointly. Write \(\hat{\boldsymbol{\beta}}_S\) for the \(m\)-vector of estimated effects and \(V\) for its estimated covariance matrix. Under the global null, assume the estimates are approximately multivariate normal with mean zero and covariance \(V\), and assume \(V\) is full rank. The Wald statistic is
Under these conditions, \(Q\) has an approximate chi-square distribution with \(m\) degrees of freedom under \(H_0\). The p-value is the probability, under that null distribution, of a statistic at least as large as the observed \(Q\). If the covariance matrix is not full rank, this ordinary inverse and its stated degrees of freedom are not valid; the test must instead be formulated for the estimable dimensions.
Worked Example: A Two-Gene Joint Wald Test
Suppose a hypothetical pathway contains two genes. The gene-level effect estimates for a binary phenotype are \(\hat{\beta}_1=0.30\) and \(\hat{\beta}_2=0.20\) log-odds per allele, each with standard error \(0.10\). For this example, assume the estimates are independent, so their covariance matrix is diagonal. Also assume the estimates are approximately normal under the null.
The covariance matrix has the squared standard errors on its diagonal:
Its inverse is the diagonal matrix with entries \(100\). Therefore,
Under the two-degree-of-freedom null distribution, the upper-tail probability is \(P(\chi^2_2\ge13.00)=e^{-13/2}\approx0.00150\). This is evidence against the joint null that both gene effects are zero. It is not evidence that both effects are individually significant, and the estimates describe association, not causation.
Combining Gene-Level P-Values
A joint effect vector is not always available. If valid gene-level p-values are available, a set-level statistic can combine them. Fisher’s method is a standard option. Let \(p_1,\ldots,p_m\) be the p-values for the genes in the set and define
Small gene-level p-values contribute larger positive amounts to \(T\). When all gene-level null hypotheses are true and their p-values are independent and continuously distributed, each p-value is uniform on \((0,1)\), and \(T\) follows a chi-square distribution with \(2m\) degrees of freedom. This provides a calibrated global-null p-value under those assumptions. The independence condition is substantive: valid marginal p-values alone do not ensure that the usual Fisher reference distribution applies.
Worked Example: Fisher’s Test for Three Genes
Consider a hypothetical three-gene set with two-sided gene-level statistics \(Z=0.1,\ 1.8,\) and \(2.4\). Suppose each statistic is standard normal under its gene-level null, and suppose the three statistics are independent under the global null. The corresponding two-sided p-values are approximately \(0.9203,\ 0.07186,\) and \(0.01640\).
The Fisher statistic is
Using \(\ln(0.9203)\approx-0.0831\), \(\ln(0.07186)\approx-2.633\), and \(\ln(0.01640)\approx-4.111\), the calculation is
With \(2m=6\) degrees of freedom, the upper-tail probability is approximately \(0.0338\). Thus the set shows evidence against the global null at the 0.05 level, conditional on the stated independence and calibration assumptions. The large contribution from the gene with \(p=0.01640\) is important: the combined result does not imply that all three genes are associated.
What the Test Does—and Does Not—Compare
A self-contained test uses evidence within \(S\), not the evidence in \(S\) relative to a background. As a result, a set can have a small self-contained p-value even when genes outside the set have similarly strong evidence. Conversely, a set might rank above much of the background without showing compelling evidence against the null that all of its own effects are zero. The two tests answer different scientific questions, so their p-values need not agree.
For a concrete comparison, consider the three absolute gene-level statistics from the previous example, \(0.1,\ 1.8,\) and \(2.4\). Suppose the absolute statistics for three genes outside the set are \(0.2,\ 0.3,\) and \(0.4\). A simple exact rank-based competitive test ranks all six values from smallest to largest, with no ties, and asks whether the set’s rank sum is unusually large. This test uses an exchangeability assumption under the competitive null: conditional on the six scores and on set size, the three set labels are treated as equally likely assignments.
Worked Example: Comparing Within-Set and Competitive Evidence
The ordered scores and ranks are \(0.1\) (set, rank 1), \(0.2\) (outside, rank 2), \(0.3\) (outside, rank 3), \(0.4\) (outside, rank 4), \(1.8\) (set, rank 5), and \(2.4\) (set, rank 6). The set’s rank sum is
Under the exact competitive null, each of the \(\binom{6}{3}=20\) assignments of three ranks to the set is equally likely. For an upper-tail test, count assignments with rank sum at least 12. The assignments with sums greater than 12 are \((2,5,6)\), \((3,4,6)\), \((3,5,6)\), and \((4,5,6)\), giving four assignments. There are three assignments with sum exactly 12: \((1,5,6)\), \((2,4,6)\), and \((3,4,5)\). Thus,
The competitive rank test does not show strong evidence that the set ranks above the background. For clarity about the opposite tail, the assignments with sums greater than 12 are four, so 16 of the 20 assignments have sums at most 12; that lower-tail probability is \(16/20=0.80\). This does not change the upper-tail p-value. In contrast, Fisher’s within-set test for these three genes gave approximately \(0.0338\). The difference is coherent: the set has evidence against its own global null, but its scores are not unusually high relative to this small background sample under the chosen rank test.
Choosing and Interpreting a Self-Contained Test
The test should match the available evidence and the scientific question. A joint Wald test directly uses effect estimates and their covariance matrix. Fisher’s method combines gene-level p-values and can detect evidence concentrated in one or more small p-values, but its usual chi-square calibration requires independent null p-values. Other combining rules weight or emphasize evidence differently; no single rule is best for every alternative.
Several conditions deserve attention before interpreting a nominal p-value:
- Valid gene-level tests: Each input statistic or p-value must be calibrated for its gene-level null under the model used to obtain it.
- Dependence: Gene-level statistics may be correlated. Applying an independence-based reference distribution to dependent statistics can miscalibrate the set-level p-value.
- Set definition: The tested genes and the combining procedure should be specified in a way consistent with the inferential procedure. Selecting a set after inspecting the same association results can make a nominal p-value misleading.
- Scope of inference: A set-level rejection supports evidence of association within the set; it does not by itself identify a causal pathway, a causal gene, or a biological mechanism.
A self-contained p-value is not a measure of how much stronger the set is than other genes, nor does it give the probability that the null is true. If many gene sets are tested, a small raw p-value also does not by itself resolve the multiplicity of those tests. Report the tested set, the input evidence, the combining rule, its calibration assumptions, and the precise null being assessed.
Common Mistake
A common error is to interpret a significant self-contained result as proof that a pathway is enriched relative to the rest of the genome. That conclusion does not follow: the self-contained null contains no comparison with genes outside the set. The correction is to describe the result as evidence against the joint null of no association among the set’s genes, and to use a competitive test if the scientific question is explicitly about relative enrichment.
Check Your Understanding
Answer these questions using the definitions and assumptions above.
- Write the self-contained null and alternative for a set containing five genes.
- In a joint Wald test, what role does the covariance matrix play, and why must it be full rank for the stated inverse-based statistic?
- What assumption permits the usual chi-square reference distribution for Fisher’s statistic?
- A set-level test rejects the global null. Does this show that every gene in the set is associated? Explain.
- How can a set have a small self-contained p-value but a non-significant competitive p-value?