Why Feature Overlap Changes a Validation Claim
A validation cohort can be large, carefully measured, and still be unable to evaluate every discovery finding. Different assay panels, preprocessing rules, or feature definitions may leave some molecular features unavailable or non-comparable across cohorts. The key question is therefore not only whether the validation results agree with discovery, but also which findings could be checked at all.
A result about the features shared between cohorts is not automatically a result about the full discovery feature set. If many discovery candidates are absent from the validation data, successful replication among the available candidates may support a narrower claim than the original signature or finding set. Coverage makes that limitation explicit.
These two coverage measures answer different questions. Platform-level coverage describes how much of the discovery feature space is represented in validation. Finding-level coverage describes what fraction of the particular discovery candidates can be evaluated. A candidate set can have high finding-level coverage even when platform-level coverage is low, if the candidates happen to be concentrated among shared features. Conversely, a high platform-level percentage does not guarantee that all the most important candidates are available.
Define Comparability Before Counting
Set intersection is meaningful only when a feature in one cohort represents the same target as the feature in the other. Use a matching rule that is explicit about identifiers, feature granularity, and any required harmonization. For example, a gene-level expression measurement should not be matched to a transcript-level measurement merely because the labels are similar. A matching decision may also need to account for platform comparability, which is the subject of the earlier material on cross-platform measurement comparability.
The eligible feature sets should reflect the analysis actually possible under the validation design. A feature can be present in a raw file but not eligible if it failed quality control or cannot be mapped under the prespecified rule. Conversely, a feature should not be declared comparable only after seeing that its validation result supports the discovery claim. Define eligibility and matching without reference to the validation outcome to reduce outcome-driven selection.
Coverage is descriptive: it counts what can be evaluated, not whether the evaluated findings replicate. It does not itself require a probability model. However, using the observed shared features to make claims about all discovery features requires an additional justification. In particular, high coverage does not prove that missing features are unimportant, and a representative-looking subset does not establish that feature availability is unrelated to effect or replication status.
State which discovery features are in the denominator, such as all eligible discovery features or only the prespecified candidate findings. Do not switch denominators without naming the change.
Identify which features have a defensible counterpart in the validation cohort, using identifiers and measurement definitions determined independently of validation outcomes.
Give numerator and denominator as counts as well as a proportion, then describe what the count represents in context.
Assess replication only for the findings that are measurable and comparable. State separately what is unresolved because a feature was unavailable.
Calculating Coverage and Interpreting the Denominator
Suppose the discovery analysis includes \(N_D=|D|\) eligible features and the comparable shared set contains \(N_C=|C|\). Then platform-level coverage is:
If \(N_S=|S|\) discovery findings were selected for validation and \(N_{SC}=|S_C|\) are in the common set, finding-level coverage is:
For the finding-level percentage, the denominator is the prespecified set of selected findings, not the number that happened to be available. Dividing by the available findings instead would answer a different question and could make incomplete coverage appear complete. Always show the counts so readers can see what the percentage summarizes.
Worked Example: Separating Platform and Finding Coverage
In a hypothetical analysis, the discovery cohort has 800 eligible features. The validation cohort has a defensibly matched, comparable counterpart for 520 of those features. A prespecified set of 20 discovery findings includes 13 features from that common set. Calculate both coverage measures.
Platform-level coverage. Here \(N_D=800\) and \(N_C=520\):
Thus, 65.0% of the eligible discovery feature universe is represented by comparable validation measurements.
Finding-level coverage. Here \(N_S=20\) and \(N_{SC}=13\):
In this example, the percentages happen to be the same, but their meanings differ: one concerns all 800 eligible features, while the other concerns the 20 selected findings. If 9 of the 13 evaluable candidates showed a prespecified direction of effect, the direction agreement among evaluable candidates would be \(9/13=69.2\%\). The other 7 candidates were not evaluated; they are not direction disagreements or failed replications. The result supports a statement about the 13 evaluable findings, not an unqualified statement about all 20.
When Coverage Is Uneven
A single overall coverage percentage can conceal important gaps. Availability may differ across feature classes, biological regions, or groups of candidates. If a validation assay includes common metabolites but omits many lipid species, for instance, the overall percentage alone may not show that a particular scientific claim is poorly covered. Where relevant, report coverage in prespecified categories as well as overall coverage.
An unevenly covered candidate set also matters for generalization. If the measured subset differs systematically from the omitted subset, the observed replication rate among evaluable findings may not represent the rate that would have been observed for all candidates. The descriptive fraction of successful results in the measured subset remains valid for that subset, but extending it to unavailable findings needs assumptions or additional data.
Worked Example: Finding a Coverage Gap Within a Candidate Set
A hypothetical discovery analysis selects 40 findings: 25 protein features and 15 lipid features. The validation assay can evaluate 24 of the proteins and 6 of the lipids. Calculate total finding-level coverage and the coverage within each feature class.
The total number evaluable is \(24+6=30\), so total coverage is:
Within the protein candidates, coverage is \(24/25=96.0\%\). Within the lipid candidates, coverage is \(6/15=40.0\%\). The overall 75.0% figure therefore hides a substantial gap for lipid findings.
Suppose 18 of the 24 evaluable proteins and 4 of the 6 evaluable lipids meet a prespecified replication criterion. There are \(18+4=22\) successful results among 30 evaluable candidates, or \(22/30=73.3\%\). That is a result for the measurable subset. Reporting only “22 of 40 findings replicated” would incorrectly treat the 10 unavailable candidates as failures; reporting only “73.3% replicated” without the coverage counts could imply more complete evaluation than occurred. A clear account gives the evaluable and total counts, with the class-specific gaps.
What Can Be Concluded About Unavailable Findings?
An unavailable finding has no observed validation outcome under the current measurement design. It is not evidence for replication, and it is not evidence against replication. One way to make this uncertainty concrete is to calculate logical bounds for the proportion of all selected findings that would meet a binary criterion, if that outcome were defined for every finding. These are bounds, not confidence intervals: they describe what is compatible with the observed evaluable results when outcomes for unavailable findings are unknown.
Let \(N=|S|\) be the total number of selected findings, \(m=|S_C|\) the number evaluable, and \(r\) the number meeting the criterion among those \(m\). Without assumptions about the unavailable findings, the smallest possible total number meeting the criterion is \(r\), and the largest is \(r+(N-m)\). Dividing by \(N\) gives:
The lower bound treats every unavailable outcome as not meeting the criterion; the upper bound treats every unavailable outcome as meeting it. Neither endpoint estimates the unknown outcomes. The range simply shows how little or much the missing evaluations could change the all-candidate summary.
Worked Example: Bounding an All-Candidate Replication Fraction
In a hypothetical validation set, 20 findings were selected, 12 were evaluable, and 9 of those 12 met the prespecified replication criterion. Calculate the possible range for the proportion meeting that criterion across all 20 findings, without assuming anything about the 8 unavailable findings.
Here \(N=20\), \(m=12\), and \(r=9\). If none of the 8 unavailable findings would meet the criterion, the total is 9:
If all 8 unavailable findings would meet it, the total is \(9+8=17\):
Thus, without further assumptions, the all-candidate proportion could range from 45.0% to 85.0%. This wide interval is not a 95% confidence interval and does not quantify sampling uncertainty. It is a logical range reflecting unknown outcomes for the unavailable findings. The directly observed fraction, \(9/12=75.0\%\), applies to the 12 evaluable findings.
Choosing an Analysis and Avoiding Overclaiming
When features overlap only partially, the most direct approach is to evaluate the prespecified candidates in the common set and report the coverage alongside the validation outcomes. This keeps the analysis tied to measurements that exist and are comparable in both cohorts. Other choices can be useful but answer different questions:
| Approach | What it answers | Main limitation |
|---|---|---|
| Evaluate only candidates in the common set | How the measurable, comparable discovery candidates perform in validation | Does not directly establish results for unavailable candidates |
| Report coverage by feature class as well as overall | Whether gaps are concentrated in particular types of findings | Does not recover outcomes for missing features |
| Make a claim about the full discovery set from the shared subset | A broader generalization, if justified by additional assumptions or evidence | Can be misleading if the shared subset is not representative |
Avoid changing the candidate set after observing validation outcomes simply to improve coverage or apparent agreement. If a different set is of interest, label it as a new analysis, state how it was formed, and report its denominator. Likewise, do not silently replace an unavailable feature with a loosely related one: a proxy or aggregation changes the measurement target and must be justified as such.
Coverage is also distinct from cohort sample size. A study may have many participants but still measure few of the discovery features, or measure many features in a small cohort. These limitations affect different parts of the validation question and should not be collapsed into one summary.
Common Mistakes and Diagnostics
- Counting unavailable features as failures. A feature with no comparable validation measurement has no observed validation outcome. Report it as unevaluable, not as a failed replication.
- Using the wrong denominator. The proportion of selected findings that can be evaluated uses all selected findings as its denominator. State the numerator and denominator explicitly.
- Assuming a high percentage guarantees broad representation. Check whether coverage is concentrated in some feature classes while missing others. Report relevant subgroup counts.
- Matching features by label alone. Similar names do not establish identical measurement targets. Check identifiers, granularity, and assay comparability under a prespecified rule.
- Generalizing from the shared subset without qualification. The observed results directly describe the evaluable subset. Broader claims require a reason that the subset supports them.
Useful warning signs include a large gap between platform-level and finding-level coverage, substantial differences in coverage between feature classes, or many selected findings excluded by matching and quality-control rules. These do not make the validation useless; they indicate that the scope of the conclusion needs to be narrower and the exclusions made visible.
Check Your Understanding
Answer each question using the definitions and distinctions developed above.
- A discovery analysis has 500 eligible features, 350 are comparable in validation, and 16 of 25 selected findings are among those shared features. What are platform-level and finding-level coverage?
- Why is a candidate with no comparable validation measurement not a failed replication?
- Of 30 selected findings, 18 are evaluable and 12 meet the criterion. Without assumptions about the other findings, what are the logical bounds for the all-candidate proportion meeting the criterion?
- Overall candidate coverage is 80%, but coverage is 95% for proteins and 35% for metabolites. What should be added to a report, and why?
- What is wrong with defining feature matches after inspecting which validation results agree with discovery?
Solutions. (1) Platform-level coverage is \(350/500=70.0\%\). Finding-level coverage is \(16/25=64.0\%\); the denominators refer to different sets. (2) No validation outcome was observed for that feature, so it cannot count as either agreement or disagreement. (3) Here \(N=30\), \(m=18\), and \(r=12\). The lower bound is \(12/30=40.0\%\); the upper bound is \((12+12)/30=80.0\%\). These are logical bounds, not a confidence interval. (4) Report the feature-class coverage counts or percentages alongside the overall figure, because the overall value conceals a substantial metabolite coverage gap. (5) Outcome-informed matching can selectively retain agreeing results and distort the validation summary; define matching without using the validation outcomes.