Diagnostics and Sensitivity Analysis for Enrichment Results

Why Enrichment Results Need Diagnostics

A pathway-level result compresses information from many genes into a statistic and a p-value. That summary can conceal important differences: evidence may be spread across a set, concentrated in one gene, associated with gene size, or dependent on how variants were assigned to genes. A small p-value from one analysis therefore does not, by itself, show that the result is robust to reasonable analytic choices.

Diagnostics and sensitivity analysis examine those choices while keeping the scientific question visible. They do not replace a valid enrichment test, and they do not establish a biological mechanism. Their purpose is to show what contributes to the result, whether plausible alternatives materially change it, and whether the null procedure is appropriate for the data and design.

Inspect the Gene-Level Evidence

Let \(T_g\) denote the gene-level score for gene \(g\), with larger values representing stronger evidence under the chosen scoring scheme. The exact scale matters: a signed score retains direction, whereas a nonnegative score such as \(-\log_{10}(p_g)\) measures strength without retaining direction. A score plot should label its scale and show the genes in the set, rather than displaying only a pathway summary.

Useful plots show the gene scores in pathway order or as a ranked dot plot, with gene labels for the strongest contributors. If available, display scores for the full analysis universe in the same way or provide a reference distribution. Mark genes with missing scores separately: absence of an annotated or measurable score is not evidence of no association. A plot can reveal whether the result is broadly supported, whether a few genes dominate, and whether the pathway contains unusually large or well-covered genes.

A simple descriptive check is to calculate the pathway mean score and the mean after omitting each gene in turn. If \(A\) is a gene set with \(m\) genes, then \( \bar{T}_A = \frac{1}{m}\sum_{g\in A}T_g \). The leave-one-out value for gene \(j\) is \( \bar{T}_{A,-j} = \frac{m\bar{T}_A-T_j}{m-1} \). This identity follows by subtracting \(T_j\) from the sum of the \(m\) scores, leaving \(m\bar{T}_A-T_j\), and dividing by the \(m-1\) remaining genes. It is a diagnostic of concentration, not a replacement p-value: genes may be correlated, and removing a gene changes the set being tested.

Check Dependence on Gene Size

Gene size can influence gene-level evidence because genes with more variants or greater genomic span may have more opportunities to produce an extreme statistic. Functional annotation coverage and linkage disequilibrium can also affect scores. Earlier material on gene-size bias and covariate-adjusted enrichment establishes why these properties may matter; here, the diagnostic question is whether the pathway conclusion changes after the relevant property is handled.

Compare the unadjusted result with a size-adjusted analysis using the same gene universe and a clearly stated adjustment. For example, one may compare pathway and non-pathway scores after fitting a prespecified model that includes a measure such as log variant count, or compare genes within size-matched strata. The adjustment must be appropriate to the score and the sampling or permutation design. A regression coefficient or residual is not automatically a calibrated enrichment p-value; inference still requires a valid standard error or null procedure that respects gene-score dependence and the analysis design.

Inspect overlap as well as the adjusted estimate. If pathway genes occupy a size range with few or no non-pathway genes, the data provide little basis for separating pathway membership from size. A model can still return a coefficient, but it may rely on extrapolation. Report that limitation rather than treating adjustment as a guarantee that bias has been removed.

Vary Gene Boundaries and Gene-Set Definitions

A gene score can depend on which variants are assigned to that gene. Alternative defensible boundaries might use the annotated gene body alone or extend it by a prespecified upstream or downstream window. If the variant-to-gene mapping changes, recompute gene scores using the same score construction and then rerun the pathway analysis. Merely changing labels while keeping scores fixed does not evaluate boundary sensitivity when the score itself depends on variant assignment.

Pathway membership also depends on the chosen gene-set definition. Relevant alternatives may include a different ontology release, a documented evidence tier, or a prespecified rule for including parent and child terms. Compare these definitions transparently: give the set size, shared and changed members, the gene-level scores of changed members, and the resulting pathway statistic and uncertainty or p-value. Keep the gene universe consistent where possible; if it changes, report that too, since the universe affects the interpretation and calibration of competitive tests.

A useful comparison separates changes in scores from changes in membership. First hold the set definition fixed and recompute scores under alternative boundaries. Then, where appropriate, hold the score pipeline fixed and vary the set definition. This helps identify whether sensitivity arises from variant-to-gene mapping, pathway annotation, or both. The alternatives should be scientifically defensible and described, not selected because they produce the smallest p-value.

Assess Calibration and Report Robustness

Sensitivity and calibration answer related but different questions. Sensitivity asks whether a finding changes under reasonable analytic choices. Calibration asks whether the inferential procedure produces appropriate error rates under its null. For a continuous exact null p-value, the p-value is uniform on 0 to 1 under the null; discrete procedures may instead be conservative. In pathway analyses, dependence among genes and overlap among sets mean that many reported pathway p-values are not independent. A collection of small p-values therefore cannot be interpreted as though each pathway were an unrelated test.

Calibration checks should use a null procedure that reflects the actual analysis, including relevant gene-score dependence, gene properties, and any design restrictions. Where appropriate, inspect the empirical distribution of null statistics or p-values across repeated null data sets, and compare observed tail rates with the rates expected under the stated null. A permutation scheme that breaks linkage disequilibrium or ignores size-related structure may generate an unsuitable reference distribution even if it produces many resamples. Earlier material on permutation nulls and LD-aware analysis describes these design concerns; the diagnostic task is to verify that the chosen null matches the present analysis.

Summarize robustness with a compact comparison: primary definition, boundary alternatives, set-definition alternatives, size-adjusted analysis, and the relevant statistic and p-value for each. Also report set size, the universe, the score scale, and notable gene-level contributors. A pathway that remains directionally and materially similar across credible choices is more robust to those choices, but this is not proof of causation or of biological function. A result that changes substantially should be described as sensitive, with the source of sensitivity made explicit.

Common Mistake: Treating the Best Result as the Robust Result

A common mistake is to try several gene boundaries, ontology releases, and adjustment models, then report only the version with the smallest p-value as evidence that the pathway is enriched. This reasoning confuses a favorable result among many choices with stability across choices. Searching over alternatives can increase false-positive risk, and the alternatives may not be independent.

The correction is to identify a primary analysis on scientific grounds, state which alternatives were examined, and compare the full set of results. Sensitivity checks are evidence about dependence on assumptions; they are not a menu from which to select the most attractive answer. If alternatives were explored extensively, that exploration should be disclosed and the inferential claims should account for the selection process or be treated as exploratory.