Why Enrichment Reports Need an Analysis Record
An enrichment result is not identified by a pathway name and a p-value alone. The same expression data can produce different results when analysts use different gene identifiers, gene-set releases, tested universes, ranking statistics, size filters, or null models. A reproducible report therefore describes the complete chain from the measured genes to the reported pathway-level results.
This is not a demand to print every line of code in the article. It is a demand to make the analysis choices explicit enough that another analyst can reconstruct the input and procedure, or understand exactly why reconstruction is not possible. The earlier material on over-representation analysis (ORA), gene set enrichment analysis (GSEA), and multiple pathway testing supplies the methods; here the emphasis is on documenting them without leaving consequential choices implicit.
Describe the Inputs Before the Test
Begin with the gene-set resource. State its name, release or version, download date or stable identifier, organism, and any filtering used to create the collection. If sets were restricted to particular evidence codes, ontology branches, or annotations, say so. Give the identifier system used for both the expression data and gene sets, and describe how identifiers were mapped. Report how unmapped identifiers, one-to-many mappings, duplicated gene identifiers, and duplicated set members were handled. These details matter because they determine both set membership and pathway size.
Next define the tested universe, denoted \(U\): the genes eligible to contribute to the enrichment test. It is not automatically every gene in the genome or every gene in the gene-set database. In an expression analysis it is usually restricted to genes that were measurable and passed the study's pre-analysis eligibility rules. For a standard hypergeometric ORA, the selected genes \(L\) must be a subset of \(U\), and the pathway \(S\) is evaluated through its members in \(U\). State the universe rule and the resulting number of genes, not just a phrase such as “all genes.”
For ORA, report the precise gene-level selection rule: for example, a specified adjusted p-value threshold together with a specified effect-size threshold, and whether the rule was applied in one direction or both. For a ranked analysis, name the statistic used to order genes, its sign convention, the treatment of ties, and the handling of genes with missing or non-finite values. “Genes were ranked by differential expression” is not enough: a signed test statistic and an unsigned p-value produce different orders and biological interpretations.
Worked Example: Documenting a Small ORA Input
Consider a hypothetical teaching analysis with eight eligible genes, \(U=\{A,B,C,D,E,F,G,H\}\). Suppose the selected list is \(L=\{A,B\}\), and the pathway is \(S=\{A,B\}\). State the quantities that determine the ORA overlap.
The universe size is \(N=|U|=8\); the pathway size within the universe is \(K=|S\cap U|=2\); the selected-list size is \(n=|L|=2\); and the overlap is \(k=|S\cap L|=2\). These are the four counts to report. Under the standard hypergeometric ORA null, conditional on \(N\), \(K\), and \(n\), the selected genes are treated as an equally likely subset of size \(n\) from \(U\). The one-sided over-representation probability is therefore
Here \(X\) is the number of selected genes in the pathway. This calculation is a deliberately tiny hypothetical example, not evidence about a real biological pathway. A report for a real analysis must give the actual universe rule and counts: substituting the whole genome for the genes that could have been selected would change the null distribution.
Name the Test, Null Model, and Filtering Rules
State whether the procedure is competitive or self-contained. A competitive test asks whether a gene set shows a stronger signal than genes outside the set, under its specified null. A self-contained test asks whether genes in the set show evidence of association under a null of no association within the set. These answer different questions; a report cannot leave the distinction to readers to infer from a software package name.
For the chosen procedure, name the test statistic and explain how its reference distribution or resampling scheme is obtained. If permutations are used, state what is permuted (for example, sample labels or gene labels), how many resamples were used, and any restrictions on permissible permutations. If a parametric approximation or rotation procedure is used, identify it and state the model assumptions relevant to its null calibration. Report whether intergene correlation is represented or ignored where that distinction applies. Do not describe a p-value as if it arose from a different null than the one actually tested.
Give the pathway-size filters as applied to the set after intersecting it with the tested universe. Report the minimum and maximum sizes and whether these bounds are inclusive. Also state how many sets were tested after filtering. Size filtering changes both the tests conducted and the family for multiplicity adjustment; it is not merely a display preference.
For ranked analyses, report the ranking statistic and its direction, the enrichment-score procedure and weighting choice, and the null construction or permutation scheme. If the software reports a normalized enrichment score, retain the method-specific meaning and do not describe it as a fold change or as the proportion of pathway genes affected. If a leading-edge subset is reported, identify the definition used by the procedure. That subset summarizes genes contributing to the enrichment signal; it is not, by itself, independent confirmation of the pathway.
Worked Example: Reporting a Ranked Analysis
In a hypothetical expression study, an analyst ranks 6,000 eligible genes by signed moderated \(t\)-statistics, with positive values indicating higher expression in condition A than condition B. A specified GSEA implementation tests sets containing 15 through 500 eligible genes, uses 10,000 phenotype-label permutations, and reports a normalized enrichment score and permutation p-value. Show how these choices become a reproducible method description.
A concise methods record would say: “Genes were ranked by signed moderated \(t\)-statistics (positive values indicate higher expression in condition A); all 6,000 eligible genes were retained, with ties resolved by the stated software rule. Gene sets were intersected with the tested universe and sets of 15–500 members, inclusive, were tested. Enrichment used [named implementation and version], with [specified weighting setting] and 10,000 phenotype-label permutations; the null was generated by permuting condition labels according to the study design. The reported statistics were the normalized enrichment score, nominal permutation p-value, and multiplicity-adjusted value.” The bracketed settings must be replaced by actual values, not omitted.
Suppose the hypothetical output for one pathway has 40 genes in the tested universe, a normalized enrichment score of 1.82, a nominal p-value of 0.006, and an adjusted value of 0.041. The report should add the pathway identifier and name, indicate the positive direction under the stated sign convention, and give the leading-edge genes or a stable supplementary-file location for them. It should not claim a 1.82-fold expression increase: the normalized enrichment score is a pathway-level enrichment statistic, not an expression ratio.
Multiplicity and Pathway-Level Results
Report the multiplicity procedure, the set of hypotheses included in its adjustment family, and the adjusted values. For example, “Benjamini–Hochberg false-discovery-rate adjustment across all 240 pathways that passed the stated size filters” is more informative than “FDR corrected.” Make clear whether adjustment was performed separately by collection, direction, contrast, or analysis, if applicable. The adjusted value addresses the stated family and procedure; it is not the probability that a particular pathway is truly associated.
A pathway result should be traceable and interpretable. Include a stable pathway identifier as well as its name, since names may change or be shared. For ORA, report the pathway size in the universe, selected-list size, overlap count, overlap genes, test statistic or effect summary, nominal p-value, and adjusted value. For ranked enrichment, report the score and direction, nominal p-value, adjusted value, and leading-edge information when available. Include the denominator and units or definition for any reported proportion. Avoid tables that show only names and adjusted p-values: without set sizes and overlap details, readers cannot tell whether a result rests on a broad overlap or a few genes.
Worked Example: Reporting an Adjustment Family
Suppose a hypothetical analysis tests three pathways after filtering, with sorted nominal p-values \(0.002\), \(0.012\), and \(0.040\). Under Benjamini–Hochberg adjustment, the initial rank-based values are \(3p_{(i)}/i\), with a monotonicity correction from the largest rank downward.
Thus the adjusted values in the original sorted order are \(0.006\), \(0.018\), and \(0.040\). The report must say that the adjustment family contained these three tested pathways. If 300 pathways had passed filtering instead, these adjusted values would not describe adjustment across that 300-pathway family.
A Practical Reporting Sequence and Audit
A short structured record helps prevent omissions. The sequence below follows the dependency of the analysis: the database and identifiers define membership, the universe and gene-level rule define the input, the test defines the null, and filtering and adjustment define which pathway results are reported.
Record collection, release, organism, identifier type, mapping rules, and any set-construction restrictions.
Give the tested-universe rule and size, then state the selection thresholds or complete ranking and direction rule.
Name the test, statistic, null model, resampling or calibration procedure, and pathway-size bounds.
State the adjustment family and procedure, pathway-level fields, software versions, and any random seed needed for stochastic computation.
A useful diagnostic is to check that counts reconcile. The selected list should be drawn from the declared universe; each reported pathway size should refer to the tested-universe intersection; the number of adjusted tests should match the declared family; and pathway identifiers should match the cited database release. If a software run drops genes silently, or if a table reports only pathways meeting a cutoff without identifying how many tests were conducted, the account is not adequate for interpreting adjusted results.
Different approaches are not interchangeable substitutes for missing details. ORA is naturally reported through a selected list and its universe, whereas ranked enrichment requires the full ranking and its signed statistic. Competitive and self-contained tests also have distinct null questions. A complete report allows readers to distinguish those choices; it does not imply that one procedure is universally preferable.
Reproducibility has limits. A well-documented analysis can be rerun, but it does not remove uncertainty in the expression measurements, establish causation, or guarantee that a result will replicate in a new sample. Database annotations change over time, and different valid mapping policies may yield different memberships. Record the original resource version and, where possible, retain the exact gene-set file, input ranking or selected list, analysis code, and result table as supplementary materials.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- An ORA report says “all genes were the background.” What must be clarified?
- A ranked analysis reports a normalized enrichment score but gives no sign convention for the gene-level statistic. Why is this a problem?
- Three pathways pass size filters, but adjusted values were computed over 200 pathways. What must the report identify?
- Give two details needed to reproduce the gene-set membership used in an analysis.
Solutions. (1) “All genes” must be replaced by an operational universe rule and its resulting gene count, including how genes unavailable for testing were treated. (2) The direction of pathway enrichment cannot be interpreted without knowing what positive and negative gene-level ranks mean; report the ranking statistic and its sign convention. (3) State that the adjustment family contained 200 hypotheses and explain which pathways entered it, including the filtering rule; do not imply adjustment was across only the three displayed results. (4) Examples include the collection and release, organism, identifier system, mapping policy, and any annotation or evidence-code restrictions.