Choosing Frequency and Quality Inclusion Thresholds

Why Inclusion Thresholds Need to Be Explicit

A gene-level analysis begins with a set of variants, not with a single gene-wide measurement. Even after choosing a functional mask, the analyst must decide which variants are sufficiently rare and sufficiently well measured to include. Those decisions shape the data entering the analysis: a frequency cutoff can exclude commoner variants, while a quality rule can remove calls or sites that are not reliable enough for the intended use.

There is no universal frequency or sequencing-quality cutoff appropriate for every study. Frequency estimates depend on the population and data source; quality metrics depend on the assay, sequencing pipeline, and validation evidence. The goal is therefore not to memorize one set of numbers. It is to specify rules before examining association results, justify them for the study, and show whether reasonable alternatives change the included variants.

The functional mask defines which consequences are eligible; the frequency and quality criteria further restrict that candidate set. These are distinct decisions. A variant can have a consequence allowed by the mask but be too frequent for the study’s rarity definition, or it can meet the frequency rule but fail a quality screen.

Frequency and Quality Are Different Filters

For a biallelic variant, the minor-allele frequency is the frequency of the less common allele in a stated reference population or study sample. The frequency criterion \(f_v \leq t_F\) says how rare a variant must be to qualify. The cutoff should be tied to the scientific target and the population relevant to the analysis. A frequency value from a broad reference panel may not represent a particular ancestry group well, and a pooled study estimate can obscure differences among groups.

Call quality addresses whether the observed sequence evidence supports the call. It is not a measure of biological importance. Common site-level metrics include sequencing depth, genotype quality, allele balance, and site-level quality scores. Their interpretation depends on how the pipeline defines them. For instance, depth counts reads supporting a call, while genotype quality summarizes confidence in a genotype call under a specified model. A high depth alone does not guarantee a correct call if reads are biased or misaligned.

It is important to distinguish site-level filters from genotype-level filters. A site-level rule removes a variant from the analysis when the site as a whole is unreliable. A genotype-level rule marks an individual’s call at a site as missing or otherwise unusable when that particular call fails. If a study requires complete high-quality genotype data, a site might be excluded when too many calls fail; alternatively, an analysis may retain the site while handling missing genotypes appropriately. The policy should be stated rather than left implicit.

A quality rule may combine multiple criteria. For example, a hypothetical protocol could require a site-level quality score of at least 30, adequate call rate, and no evidence of a problematic batch pattern. A separate genotype-level rule might require genotype quality of at least 20 for an individual call. Such example values are not universal standards: the assay’s calibration and validation determine whether cutoffs are defensible.

Writing the Rule as a Set Definition

Let \(M\) be the set of variants eligible under the chosen functional mask. For each \(v \in M\), let \(f_v\) be its selected minor-allele-frequency estimate, and let \(Q_v(t_Q)\) indicate whether it passes the quality criteria specified by \(t_Q\). The inclusion set is

This form makes the logic visible: membership requires every stated condition. In practice, \(Q_v(t_Q)\) may represent several criteria, such as adequate depth and site quality, rather than one numerical metric. The precise definition should name the metrics, their thresholds, how missing values are treated, and whether each criterion applies to sites, genotypes, or both.

A useful basic check follows directly from the rule. If the mask and quality rule are fixed, raising the frequency cutoff cannot remove a variant that previously qualified; it can only keep the set unchanged or add variants. If the frequency cutoff and mask are fixed, relaxing a quality rule cannot make a previously passing variant fail. These monotonicity checks are simple ways to detect implementation errors.

Compare Rules by Comparing Sets

A sensitivity analysis asks what changes when a defensible alternative inclusion rule is used. The comparison is about the variants, not just the final number of variants. Two rules can each select four variants while selecting different four-variant sets. At minimum, report the number included under each rule, which variants are added or removed relative to the primary rule, and the overlap between sets.

For sets \(S_A\) and \(S_B\), the intersection \(S_A \cap S_B\) contains variants included by both rules. The set difference \(S_A \setminus S_B\) contains variants included by rule A but not rule B. A convenient summary of overall overlap is the Jaccard similarity, the size of the intersection divided by the size of the union:

The value ranges from 0 (no variants in common) to 1 (identical sets), provided the union is nonempty. It is a descriptive summary, not a test of whether either rule is correct. Listing the actual variants remains important, especially when a small number of included variants may drive a gene-level result.

Quality sensitivity can be examined in the same way. For example, compare a primary call-rate threshold with a somewhat more permissive threshold while holding the mask and frequency rule fixed. A report should say exactly which criterion changed and identify variants that entered or left. If several thresholds change at once, it becomes difficult to determine which change caused the set difference.

A Practical Sensitivity-Analysis Plan

A useful analysis plan separates primary rules from alternatives and records how each rule was chosen. Frequency alternatives should reflect plausible definitions of rarity and relevant reference-population information. Quality alternatives should be informed by validation, replicate concordance, known error patterns, and the assay’s operating characteristics. The earlier material on calibrating filters with replicates is relevant when such evidence is available.

For each alternative, change one component at a time where possible. A small grid of frequency cutoffs and quality rules can also be useful, but label every combination and provide the resulting set sizes and memberships. If the purpose is to assess whether conclusions are robust, evaluate the downstream analysis under each prespecified rule; do not choose the threshold that yields the smallest p-value or the most favorable estimate. Such result-driven selection makes the analysis difficult to interpret and can inflate false-positive evidence.

Frequency estimates may themselves be uncertain, particularly when counts are small. The frequency source, ancestry grouping, and handling of variants absent from a reference database should therefore be documented. A cutoff is a decision rule applied to an estimate, not a guarantee that the underlying population frequency is known exactly. Likewise, passing a quality threshold means only that a call met the stated criteria; it does not prove the call is error-free.

Why Threshold Choices Matter

Thresholds affect what evidence reaches a gene-level analysis. A permissive quality rule can retain questionable calls, while an overly strict one can discard real variants. A permissive frequency rule can bring in variants outside the intended rarity range; an overly strict one can leave a sparse set. These choices affect which variants are analyzed and can therefore affect the interpretation of any later gene-level result.

The purpose of a sensitivity analysis is not to search for a preferred answer. It is to show whether the analysis depends strongly on reasonable choices and to make the consequences of those choices inspectable. A transparent rule, a justified set of alternatives, and a clear account of set changes are more informative than a cutoff presented as universally correct.

Common Mistake

• Wrong reasoning: “The frequency and quality cutoffs are standard, so there is no need to state them or check alternatives.”
• Correction: Cutoffs are study-specific decisions. Define the frequency source and quality metrics, justify the primary values, and report how plausible alternatives change variant membership. A variant’s inclusion is conditional on the stated rule; passing filters is not proof of biological relevance or perfect measurement.
• Wrong reasoning: “The alternative threshold gave a stronger association, so it is the better threshold.”
• Correction: Choosing a rule because it improves the observed result is outcome-driven selection. Set and justify alternatives independently of the association results, then report the full sensitivity analysis.