Why Benchmark Design Matters
A functional prediction score is useful only insofar as its performance has been evaluated against an appropriate target. A score may rank variants consistently with a particular assay, yet perform differently for another molecular process, tissue, variant class, or candidate-variant population. Benchmarking therefore asks a specific question: how well does this score distinguish or predict a clearly defined set of benchmark outcomes in a population it was not trained or tuned on?
The answer depends on more than a performance statistic. If variants used to develop the score also appear in the test set, the result can be optimistic. If positive and negative controls are chosen in a way that makes them unusually easy to distinguish, the result may not represent real prioritization. If positives are rare, accuracy can be high even when the score misses most positives. And if the benchmark labels were generated using the score itself, apparent agreement is not independent validation.
Define the Target Before Choosing the Metric
State what counts as a positive and what population the evaluation is intended to describe. For example, a benchmark might ask whether a score ranks variants with a specified experimentally measured molecular effect above matched variants without that effect. That is a narrower claim than whether the score identifies causal variants for a disease. A positive label tied to one assay, cell type, and effect definition supports conclusions about that benchmark target, not every possible meaning of “functional.”
Positive controls should have outcome evidence independent of the score being assessed, such as results from an assay that directly measures the prespecified molecular outcome. Negative controls should be chosen to represent a defensible comparison group. A variant with no reported evidence is not automatically a true negative: it may simply have been untested. When feasible, controls should be similar to positives on characteristics that affect score distributions or assay opportunity, such as variant class, genomic context, or measurement process. Matching does not make labels perfect, but it can reduce the chance that the benchmark is testing an irrelevant contrast.
A test set should resemble the intended use population, or its limitations should be explicit. A benchmark of experimentally selected variants may differ sharply from the variants encountered when prioritizing candidates at a locus. Metrics computed on an enriched benchmark can still compare methods under that benchmark design, but quantities that depend on the positive fraction—especially predictive values—will not automatically describe a population with a different positive fraction.
Prevent Leakage by Design
Data leakage occurs when information unavailable for a genuine prediction has influenced evaluation. The clearest case is placing the same variant in training and testing. Less obvious leakage can arise when nearby or correlated variants are split across sets, when variants from the same locus share strong signals, or when a score was trained on a public data source that overlaps the benchmark. Choosing a threshold after inspecting test outcomes is also leakage: the test set has then helped tune the procedure it is supposed to evaluate.
A robust design separates data before fitting or tuning and keeps the test outcomes untouched until evaluation. The appropriate grouping depends on how the score was developed and how it will be used. If variants from the same locus could share information, split by locus rather than by individual variant. If training and test data may overlap by study, assay, or source, audit those sources and consider holding out whole studies. Any normalization, feature selection, imputation, or calibration estimated from data should be learned using training data only, then applied unchanged to the test set.
Write down the positive-label rule, the comparison group, and the variants or settings to which conclusions are intended to apply.
Check whether variants, loci, studies, or label-generating data overlap with score training or tuning. Choose a split that respects relevant dependence.
Choose metrics and any decision threshold using training or validation data, not the held-out outcomes.
Report the benchmark composition, metrics, uncertainty where appropriate, and the limits imposed by its labels and sampling design.
Worked Example: Choosing a Leakage-Resistant Split
Suppose a hypothetical score was trained using 4,000 variants from several genomic loci. A proposed benchmark contains 600 assay-labelled variants from 30 loci, and 80 benchmark variants are also present in the score's training data. The intended evaluation is performance at loci not used for training.
A random split of the 600 variants is inadequate. It could put a variant from a locus in training and another variant from the same locus in testing, and it does nothing about the 80 exact overlaps. First, remove or otherwise resolve the exact training-benchmark overlaps. Then assign whole loci, not individual variants, to development and test sets. For example, 20 loci could be used for development and 10 held out for the final evaluation.
The 10 test loci must not be used to choose score settings, features, or a decision threshold. If assay labels from those loci informed the original score, the benchmark is not held out in the relevant sense and must not be described as independent evaluation. This design does not guarantee that the 10 loci represent every future locus, but it makes the test more directly relevant to performance on unseen loci.
Discrimination, Thresholds, and Class Imbalance
A score can be assessed at a fixed threshold or across all possible thresholds. At a chosen threshold, label a variant as score-positive when its score is at least that threshold. The counts are true positives (TP), false negatives (FN), false positives (FP), and true negatives (TN), each defined relative to the benchmark labels. Sensitivity is the fraction of benchmark positives detected; specificity is the fraction of benchmark negatives not called positive.
Accuracy can conceal poor detection when positives are uncommon. The positive predictive value, \(PPV=TP/(TP+FP)\), answers what fraction of score-positive variants are benchmark-positive in this particular evaluation. It depends on the positive fraction as well as the score's error rates. Thus, a PPV from a deliberately balanced benchmark should not be presented as the expected fraction of true positives in a population with a much lower prevalence.
Worked Example: A High Accuracy That Misses the Imbalance
In a hypothetical held-out benchmark, 20 of 100 variants are positive and 80 are negative. At a threshold fixed before examining these test outcomes, the score calls 15 positives correctly, misses 5 positives, calls 20 negatives positive, and correctly leaves 60 negatives below threshold. The threshold was fixed using separate development data, and no benchmark variants or loci overlapped with score training.
The sensitivity is \(15/(15+5)=0.75\), or 75%. The specificity is \(60/(60+20)=0.75\), also 75%. Accuracy is \((15+60)/100=0.75\), or 75%. However, calling every variant negative would yield accuracy \(80/100=0.80\), while detecting no positives. Accuracy alone therefore does not show whether the score is useful for finding positives.
The PPV at this threshold is \(15/(15+20)=15/35=0.4286\), or 42.86%: among score-positive variants in this benchmark, that fraction carries the benchmark's positive label. The negative predictive value is \(60/(60+5)=60/65=0.9231\), or 92.31%. Neither predictive value can be transferred unchanged to a population with a different positive fraction.
For ranking, the receiver operating characteristic (ROC) curve plots sensitivity against \(1-\text{specificity}\) as the threshold varies. Its area under the curve (AUC) has a useful pairwise interpretation: for a randomly selected benchmark positive and negative, it is the probability that the positive receives the higher score, counting a tie as one-half. The empirical AUC is the fraction of positive-negative pairs correctly ordered, with half-credit for tied scores. AUC measures ranking, not whether reported scores are calibrated probabilities, and it does not identify a useful operating threshold by itself.
Worked Example: Calculating AUC From Ranked Scores
Consider a hypothetical held-out benchmark with positive scores 0.9 and 0.7, and negative scores 0.8, 0.4, and 0.2. The scores are used only for ranking here; they are not assumed to be probabilities. There are \(2\times3=6\) positive-negative pairs.
The positive score 0.9 exceeds all three negative scores, contributing 3 correctly ordered pairs. The positive score 0.7 exceeds 0.4 and 0.2, but not 0.8, contributing 2. There are no ties. Therefore the empirical AUC is \((3+2)/6=5/6=0.8333\). In this small benchmark, a randomly selected positive outranks a randomly selected negative in five-sixths of the pairs. The calculation does not establish that the score will have the same ranking performance in other variant populations.
Calibration Is Different From Ranking
A score is calibrated as a probability for a stated outcome and population when variants assigned probability \(q\) have that outcome at a frequency close to \(q\), across relevant probability ranges. Calibration is not defined for a score that is only an arbitrary ranking value. A well-ranked score can systematically overstate or understate probabilities; conversely, a score can be calibrated on average yet have limited ability to separate individual positives from negatives.
Calibration assessment requires reliable labels and a test population whose outcome frequency is relevant to the intended use, or a clear adjustment for a changed positive fraction. One simple diagnostic groups predicted probabilities into bins and compares the mean predicted probability with the observed positive fraction in each bin. This diagnostic is descriptive, and results can be unstable in small bins. Calibration methods, including recalibration, must be fitted on development data rather than on the held-out test outcomes used to report performance.
Worked Example: Checking Calibration in Two Probability Groups
Suppose a hypothetical score outputs probabilities for the benchmark's positive outcome. In a held-out set, 100 variants have predicted probability 0.10 and 20 are positive; another 100 have predicted probability 0.40 and 50 are positive. The labels are independent of score construction, and each variant contributes one observation.
In the first group, the observed positive fraction is \(20/100=0.20\), compared with a mean predicted probability of 0.10. The score underpredicts the observed fraction in this group by 0.10. In the second group, the observed fraction is \(50/100=0.50\), compared with 0.40, again an underprediction of 0.10. This is evidence of miscalibration for these benchmark groups, although two bins alone cannot establish calibration behavior at every probability value or in another population.
For a numerical summary, the Brier score is the mean squared difference between predicted probabilities and binary outcomes: \(n^{-1}\sum_{i=1}^{n}(q_i-y_i)^2\), where \(q_i\) is the predicted probability and \(y_i\) is 1 for a positive label and 0 otherwise. In the first group, its contribution per variant is \([20(0.10-1)^2+80(0.10-0)^2]/100=(16.2+0.8)/100=0.17\). In the second group it is \([50(0.40-1)^2+50(0.40-0)^2]/100=(18+8)/100=0.26\). The overall Brier score is \((0.17+0.26)/2=0.215\). Lower values indicate smaller squared probability errors for this labelled test set, but the value depends on the outcome prevalence and does not by itself isolate calibration from discrimination.
Label Circularity and Limits of Transfer
A benchmark is circular when its labels or inclusion rules are derived from the same score, or from information that substantially overlaps with the score's training target. For example, defining “positive” variants as those above a threshold of the score and then showing that the score separates positives from negatives merely restates the definition. Likewise, evaluating against labels produced using a training resource may test recovery of that resource rather than independent functional validity. Such comparisons can be useful for describing concordance, but they are not independent validation.
Even with independent labels and careful separation, performance is conditional on the benchmark. Results may change across variant classes, genomic contexts, tissues, assay systems, ancestry groups, or positive fractions. A control set of easy negatives can inflate discrimination relative to a harder, matched comparison. An assay's measured outcome may be an imperfect proxy for the biological question of interest. These are not reasons to avoid benchmarking; they define what the benchmark can support.
Worked Example: Recognizing a Circular Benchmark
A hypothetical group defines 50 “positive” variants as the highest-scoring variants from a functional prediction tool and 50 “negative” variants as the lowest-scoring variants from that same tool. The tool assigns higher scores to all 50 positives than to all 50 negatives, so the resulting AUC is \(2500/2500=1\).
The arithmetic is correct, but the conclusion “the tool has perfect independent functional prediction” is not. The labels were assigned from the score, so the evaluation asks whether the score agrees with its own ranking rule. To assess functional prediction, the benchmark needs outcome labels obtained independently of the tool, with held-out variants and a comparison group appropriate to the intended use. The circular result describes construction of the groups, not predictive validity.
Common Mistakes
- Randomly splitting correlated variants. A random variant-level split can place related variants or variants from the same locus on both sides. Use grouped splits when the intended claim concerns performance on unseen loci or studies.
- Calling untested variants negatives. Lack of an assay result is not evidence of a negative outcome. Distinguish confirmed comparison controls from variants with unknown status.
- Reporting accuracy without the positive fraction. Always show benchmark composition and metrics that reveal positive and negative detection, such as sensitivity and specificity.
- Interpreting AUC as calibration or a threshold recommendation. AUC describes pairwise ranking across thresholds. It does not show that numerical scores are probabilities or determine which threshold is appropriate.
- Claiming broad transfer from a narrow benchmark. State the assay, variant population, and control design. A result on one benchmark supports conclusions about that setting, not every functional outcome.
Check Your Understanding
Use these questions to test your reasoning about benchmark design and interpretation.
- Why might splitting individual variants at random overstate performance when the intended use is prioritization at previously unseen loci?
- A benchmark has 5% positives. Explain why an accuracy of 95% could be uninformative, and name two additional quantities that would help interpret the result.
- A score has a high AUC but predicted probabilities are systematically larger than observed positive fractions. Which aspect of performance is strong, and which is not?
- Why is a variant with no published assay result not automatically an appropriate negative control?
- What can be concluded if a score performs well against independent assay labels in one cell type, and what cannot be concluded without further evaluation?