Why Circularity Matters
Biological knowledge can make a multi-omics model more interpretable and can guide estimation when the number of measured features is large. A pathway database, interaction network, or regulatory map may inform which effects are expected to be related. The risk is not simply that a prior is strong. It is that the data being analyzed may already have influenced which knowledge was chosen, how it was encoded, or which pathway was emphasized.
Suppose investigators first inspect which genes are associated with a clinical outcome, select the pathway containing the most promising genes, and then fit a model that favors effects in that pathway. The same outcome information has helped choose the pathway and estimate its effects. The resulting analysis may look like independent confirmation from biological knowledge, although the knowledge-based choice was partly tailored to the observed result.
Circularity is about information flow, not about whether a pathway is biologically plausible. A pathway can be well established and still be used circularly if the same study outcomes helped select it and are then used to claim evidence for it. Conversely, prior information from a genuinely independent source is not automatically circular merely because it concerns the same disease or biological process. Its provenance, dependence on the current data, and intended interpretation must be examined.
Where Reuse Enters the Analysis
Potential reuse can occur at several stages: choosing a database or tissue-specific pathway, filtering genes, deciding which interactions to encode, estimating prior strength, selecting a model, or choosing a reported result. The relevant question is: could the outcome values in the evaluation set have affected this decision? If yes, an ordinary analysis that treats the decision as fixed may overstate the strength of evidence or predictive performance.
For a simple Bayesian illustration, let \(H\) denote a hypothesis about an effect and let \(D\) denote current study data. Bayes' rule expresses posterior odds as prior odds multiplied by the likelihood ratio:
This factorization uses prior odds that represent information available before observing \(D\), or an appropriately modeled source of information distinct from \(D\). If the prior odds were selected after inspecting \(D\), multiplying by the likelihood ratio as if the prior were fixed can reuse the same evidence. The formula itself remains a probability identity when the prior is a genuine prior distribution; the problem is assigning the usual prior interpretation to a data-dependent choice while ignoring how it was made.
Worked Example: A Prior Chosen After Seeing the Effect
Consider a hypothetical standardized gene effect \(\theta\). The study yields an estimate \(y=2\) with known standard error \(1\), under the sampling model \(Y\mid\theta\sim N(\theta,1)\). For comparison, suppose a prior is \(N(m,1)\). Under the normal likelihood and this normal prior, the posterior variance is \(1/(1+1)=0.5\), and its mean is \((y+m)/2\).
Independent prior. If an independent source established \(m=0\) before this study, the posterior mean is \((2+0)/2=1\) and the posterior variance is \(0.5\). The prior pulls the estimate toward zero.
Outcome-informed prior. Suppose instead that investigators inspect the study result \(y=2\), choose a pathway because it contains this promising gene, and then set the gene's prior mean to \(m=2\). The same calculation gives posterior mean \((2+2)/2=2\) and variance \(0.5\). The arithmetic is correct for a prior fixed at \(m=2\), but the usual posterior interpretation is not justified by that calculation alone: the observed value helped determine \(m\). The analysis has not received independent confirmation from the prior.
This example isolates the issue rather than proposing a data-dependent prior procedure. A formal empirical-Bayes or selection model could account for how the prior was estimated or selected, but simply plugging in the outcome-informed value and proceeding as if it were external does not do so.
Pathway Selection and the Meaning of a Test
A pathway-level test ordinarily asks how compatible observed data are with a null hypothesis for a pathway specified by a particular rule. If the pathway is chosen because it has the most favorable observed association, the reported test is conditional on a selection process. A nominal p-value computed as though that pathway had been specified in advance does not generally account for the search over candidate pathways.
For example, imagine a hypothetical analysis of 200 genes. A pathway contains 20 genes, and 10 genes are selected as outcome-associated. If six of the selected genes belong to the pathway, that overlap may appear notable. But if investigators searched 50 pathways and reported whichever had the largest overlap, the pathway was not an independent target chosen before seeing the outcome associations. The overlap count \(6\) describes the observed data; it does not, by itself, provide a selection-adjusted measure of evidence. Appropriate adjustment must reflect the actual search and selection rule.
The same concern applies when pathways are selected using one omics layer and then described as independently supported by another layer from the same participants. Correlated measurements, shared samples, and outcome-guided selection can connect the decisions even when the final model combines different molecular modalities. Calling the second layer “validation” does not make it independent.
Worked Example: Prespecification Versus Selecting the Largest Overlap
Suppose a hypothetical team has 50 candidate pathways and plans to test whether outcome-associated genes are enriched in one pathway. In a prespecified analysis, the team names Pathway A before examining the outcomes. After analysis, 10 of 200 genes are selected and 3 are in Pathway A. The reported overlap is \(3\), and the inferential procedure can be designed for that prespecified pathway, subject to its stated sampling and testing assumptions.
Now suppose the team examines all 50 pathways first and chooses Pathway B because it contains 6 of the 10 selected genes, the largest overlap. A test that treats Pathway B as if it had been chosen in advance ignores the search. The relevant procedure must account for choosing the maximum across the 50 candidates—for instance, by applying the entire selection-and-testing pipeline in each valid permutation under the null. The count \(6\) is not itself enough to calculate a corrected p-value; the gene selection rule, pathway structure, and dependence among tests are needed.
The safeguard is not to avoid pathway analysis. It is to define the target before outcome inspection, or to incorporate pathway selection into the inferential procedure and report that selection explicitly.
Validation Must Include the Whole Development Process
A held-out set is useful only if it is held out from every outcome-informed development decision. If pathway selection, prior-strength calibration, or feature screening uses all outcomes before cross-validation begins, the held-out folds are no longer untouched. Predictions for those folds may benefit indirectly from their own outcomes through the earlier selection step.
In \(K\)-fold cross-validation, the observations are split into \(K\) folds. For each outer fold, model development—including outcome-informed pathway selection, prior calibration, and tuning—must be carried out using only the other \(K-1\) folds. The resulting model then predicts the untouched outer fold. If tuning is substantial, an inner cross-validation within each outer training set can choose tuning parameters. The outer-fold outcomes are used to evaluate predictions, not to make development decisions.
For each pathway, prior parameter, and filter, record whether it was fixed externally, chosen using outcomes, or estimated from the current data.
Use an independent validation cohort, a discovery/validation split, or nested cross-validation. Keep validation outcomes out of pathway selection, prior calibration, feature selection, and tuning.
In each outer training set, repeat any step that used outcomes. Apply the resulting fixed choices to that outer fold without revising them from its outcomes.
Describe data sources, selection rules, repeated steps, and any shared samples or outcomes. State whether conclusions are exploratory, confirmatory, or predictive.
Worked Example: Diagnosing Leakage in Cross-Validation
In a hypothetical prediction study, 100 participants have matched molecular measurements and a binary outcome. An analyst first uses all 100 outcomes to select a pathway and its genes, then performs five-fold cross-validation with 20 participants in each fold. Across the five held-out folds, the model classifies 82 of 100 participants correctly:
This value is not a clean estimate of performance for a procedure that selects pathways in new data: each fold's outcomes contributed to the initial pathway choice. It may be optimistic, although the size and direction of the bias depend on the data and selection process.
The team repeats the analysis using nested cross-validation. Within each outer training set, it selects the pathway and tunes the model using only that training set; it then predicts the untouched outer fold. Suppose the resulting outer predictions classify 66 of 100 correctly:
A separate hypothetical external cohort then yields 64 correct classifications out of 100, or \(64/100=0.64\). The nested estimate and external result are closer than the leaky estimate in this example, but that agreement is not guaranteed in general. The key correction is procedural: every outcome-informed development step is confined to training data, and performance is calculated from predictions for observations not used in those steps.
Practical Safeguards, Diagnostics, and Limits
The strongest safeguard is often a clearly documented information boundary. Biological knowledge may be specified using established sources before examining the current outcomes. If current data are needed to learn prior strength or choose among pathways, that dependence should be modeled or confined to training data. An independent validation cohort should not contribute outcomes to any such choice.
When no independent cohort is available, useful safeguards include a prespecified primary pathway, discovery/validation splitting, nested cross-validation, and permutation procedures that repeat the full selection pipeline. Permutations are valid only when their null-generation scheme respects the design and relevant dependence; shuffling outcomes indiscriminately can give an inappropriate reference distribution. A split also costs precision, so it is a design trade-off rather than a guarantee of high power.
A provenance table can make hidden reuse visible. For each knowledge source, record its version and date, tissue or context filters, whether it includes the study participants or study outcomes, and whether it influenced feature selection or prior calibration. For each reported analysis, state which decisions were made before outcome inspection and which were data-dependent. If an external resource may contain evidence from the same cohort, investigate that possibility where feasible and describe uncertainty rather than assuming independence.
Diagnostics include comparing results under prespecified and data-selected pathways, checking whether conclusions persist in independent data, auditing the sequence of analysis decisions, and verifying that each validation fold was excluded from all outcome-informed steps. Large differences between a leaky and properly nested estimate are a warning, not a universal correction factor. No diagnostic can turn an undocumented, outcome-guided pathway choice into a prespecified one after the fact.
The alternatives answer different questions. A prespecified pathway analysis emphasizes inference about a defined biological hypothesis. A discovery analysis can search broadly but should label the selected findings as exploratory and seek independent confirmation. Nested validation evaluates the performance of a complete learning procedure, including pathway selection. An external cohort offers a direct assessment in new data, but differences in population, assay, or clinical practice may limit transportability. None removes all uncertainty; each makes the source of evidence clearer.
A common mistake is to reason, “The pathway is biologically established, so selecting it after seeing the association does not matter.” The correction is that biological credibility and statistical independence are different properties. A strong biological rationale may support the pathway's plausibility, but it does not undo outcome-informed selection or justify treating the same observations as independent confirmation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A pathway was named before the study began using an external database. The study outcomes are then used to estimate gene effects under that pathway prior. Is this automatically circular? What should still be checked?
- An analyst screens genes using all outcomes, then uses five-fold cross-validation to report prediction accuracy. What is the problem, and what change addresses it?
- A team searches 40 pathways and reports the one with the smallest nominal p-value as confirmatory. Why may that p-value be misleading?
- In one outer fold, 18 of 25 held-out participants are classified correctly after all pathway selection was performed using only the outer training data. Calculate the fold accuracy and state what it measures.
Solutions. (1) Not automatically. The pathway choice is external and prespecified, but investigators should check whether the source incorporates the same cohort or outcomes and document any dependence; the prior and model assumptions also need to be appropriate. (2) The held-out outcomes influenced gene screening, so the folds are not independent of development. Repeat screening and all other outcome-informed steps within each outer training set, then evaluate predictions in its untouched fold. (3) The pathway was selected through a search over 40 candidates. A nominal p-value that treats the winner as prespecified ignores selection; the inferential procedure must account for the search or the result must be labeled exploratory and validated independently. (4) The accuracy is \(18/25=0.72\), or 72%. It is the proportion correctly classified in that untouched outer fold for the model-development procedure described; it is not, by itself, a guarantee of performance in another population.