Calibrating Rare-Variant Filters With Replicates

Why Calibrate Rare-Variant Filters?

A rare-variant filter is intended to remove unreliable calls while retaining true variants. Its settings therefore affect two kinds of error: a true variant may be filtered out, or an artifact may be retained and treated as a variant. Quality metrics such as those considered in earlier material on rare variant calls can rank or describe calls, but their usefulness depends on how well they distinguish true from false calls in the data and study conditions at hand.

Replicate sequencing and truth sets provide complementary evidence. Replicates show whether the same sample produces consistent results when measured again. A truth set provides calls independently judged to be true or false, allowing direct estimates of filter performance. Neither source is perfect: replicates can share systematic errors, and a truth set may be small, selective, or imperfectly representative of the study data.

What Replicates Measure

Compare replicate calls at the same sample and genomic positions, using consistent genotype definitions and a pre-specified set of positions that can be assessed in both replicates. Genotype concordance is the proportion of these comparisons with matching genotypes. For rare calls, overall concordance can be misleading: most positions may be reference in both replicates, so many matching reference genotypes can obscure inconsistent alternate calls. Report a rare-call-specific measure as well, such as positive agreement among comparisons where at least one replicate reports the alternate genotype.

Agreement is evidence of reproducibility, not proof of accuracy. Both replicates can make the same error if they share a mapping artifact, a problematic region, or another systematic source of error. Conversely, low agreement may reflect stochastic sampling or differences in coverage rather than a fixed error in one replicate. Analyze only comparable, callable observations and describe how discordances are counted; do not treat a missing genotype as a matching reference genotype.

Measuring Performance Against a Truth Set

Let a filter retain a call when its quality score \(Q\) meets a chosen threshold \(t\), using the convention \(Q \ge t\). For each truth-set genotype, compare the filter decision with its established status. A position should be counted as a truth-set negative only when the reference establishes that it is callable and does not carry the variant; an uncertain or unassessed position is not automatically a negative.

These measures answer different questions. Sensitivity describes how many true variants survive; FPR describes how often negatives are retained. Positive predictive value is especially dependent on the fraction of true variants among the evaluated units. A truth set deliberately enriched for known variants may therefore yield a positive predictive value unlike that in a study-wide set of candidate calls. Do not infer a study-wide predictive value from a differently composed validation set without accounting for that difference.

Uncertainty and What the Validation Supports

A measured sensitivity or FPR is an estimate from a finite number of truth-set units. For an estimate based on \(m\) independent units with the same true probability \(\theta\), each unit can be represented as a Bernoulli outcome. The count of successes has a binomial distribution, and the estimated proportion has approximate standard error \(\sqrt{\hat{p}(1-\hat{p})/m}\), where \(\hat{p}\) is the observed proportion. This approximation describes sampling variation; it does not account for incorrect truth labels, selection bias, or differences between validation and study samples.

For example, if sensitivity is estimated from only 30 known variants, each variant has substantial influence on the estimate. A confidence interval communicates that uncertainty. The Wilson interval is one standard interval for a binomial proportion. With \(x\) successes among \(m\) units and \(z=1.96\) for a 95% interval, its center and half-width are

The interval is a method with a long-run coverage interpretation under its binomial assumptions; it is not a probability statement that the fixed parameter is inside this particular interval. With small counts, report the denominators and interval rather than treating a rounded point estimate as precise.

Choosing a Threshold From Validation Evidence

A threshold should be selected by considering both observed performance and the scope of the evidence. First, evaluate candidate thresholds on the same defined truth-set units so that the comparison is fair. Then consider the study’s priorities: a setting intended to minimize false retained calls may tolerate lower sensitivity, while a setting intended to preserve as many true variants as possible may accept a higher FPR. State the relevant measure, its denominator, and the trade-off rather than naming a threshold “optimal” without a criterion.

The validation set should resemble the study data in the conditions that can affect calling and filtering. Relevant checks include sample preparation and sequencing batch, genomic context, coverage, and variant types represented. A validation set that omits a difficult class of regions or contains few rare variants cannot establish performance for that class. If performance varies across important conditions, report the variation and avoid relying only on a pooled average that hides it.

Threshold choice and performance assessment should also be separated when possible. Choosing a threshold to perform well on a small truth set and reporting its performance on that same set can make the result look more favorable than it will be on new data. Use an independent validation set for an unbiased assessment when available. If the available evidence is limited, describe the threshold as supported by that evidence, not universally validated.

Common Mistakes

• Equating replicate agreement with truth. Matching calls show reproducibility, but shared systematic errors can make both replicates wrong. Compare with an independent truth set to assess accuracy.
• Reporting only overall concordance. Agreement at common reference positions can dominate the total. Include a measure focused on alternate calls and state its denominator.
• Reporting sensitivity without its denominator. A sensitivity of 90% based on 10 variants is not as informative as the same estimate based on a much larger, representative set. Report counts and uncertainty.
• Calling unassessed positions negatives. A truth-set unit must be reliably assessed before it can contribute to the FPR denominator. Keep uncertain and uncallable units out of that calculation.
• Choosing a threshold from one metric alone. A lower FPR can come with more false negatives. State the study-relevant trade-off and the limits of the validation set.