Why a Frequency Threshold Defines the Question
A burden mask specifies which variants will be grouped for analysis. A frequency threshold is one rule for deciding whether a variant belongs in that set: variants meeting the rule are retained, and the others are not. The threshold is therefore more than a convenient data-cleaning setting. It helps define the genetic hypothesis being examined.
For example, a mask intended to study very rare variants should not silently include variants that are common in a substantial part of the study population. Conversely, a cutoff that is too strict for the available data may leave out variants that the investigator meant to study. A defensible choice starts with the target hypothesis, states exactly how frequency is measured, and specifies how the cutoff is applied.
In a sample of \(n\) unrelated, diploid individuals with complete calls, the estimated frequency of an allele is its observed count divided by \(2n\), the number of observed chromosomes. The estimated MAF is the smaller of the two estimated allele frequencies. In practice, frequency estimates should use appropriately quality-controlled calls and the relevant callable chromosomes; missing or non-diploid data can change the denominator. Frequency is an estimate, not a known, error-free property of the sample.
The cutoff may be a fixed value, such as 0.01, or may depend on ancestry-specific frequency estimates. Neither approach is automatically correct for every study. The choice depends on what “rare” is meant to describe, who is represented in the analysis, and whether the target is a pooled population or particular ancestry groups.
Choosing a Fixed Cutoff
A fixed cutoff applies the same numerical rule to each variant. To choose it, first state the frequency range that matches the scientific question. A threshold of 0.01, for instance, defines a particular operational set: variants with estimated MAF at or below 1% in the specified frequency source. It does not establish a universal boundary between rare and common, and the label “rare” should not replace the actual cutoff in reporting.
Next, decide which frequency estimate will determine inclusion. Options include the study sample, an external reference population, or a specified combination of sources. Study-sample estimates describe the sample used for analysis, but may be imprecise for very low-frequency variants. External estimates can provide more chromosomes for estimating frequency, but they are informative only to the extent that their population composition, variant calling, and ascertainment are suitable for the study. These sources are not interchangeable without explanation.
The cutoff should be fixed before examining trait-association results. Choosing a threshold because it produces the smallest p-value makes the tested variant set depend on the outcome data and creates additional selection that must be addressed in the analysis plan. If several thresholds are scientifically useful, they should be identified as distinct analyses rather than presented as if one had been the sole prespecified choice.
Worked Example: Applying a Fixed Threshold
Suppose a hypothetical study has 1,000 unrelated individuals with complete diploid calls at two biallelic variants. At the first variant, the less frequent allele is observed 20 times; at the second, it is observed 24 times. Use a fixed threshold of 0.01 and the inclusion rule \(\mathrm{MAF}\leq 0.01\).
There are \(2(1{,}000)=2{,}000\) observed chromosomes per variant. The estimated MAFs are:
The first estimate is exactly 0.01, so the first variant is included under the stated “less than or equal to” rule. The second estimate is 0.012, which exceeds 0.01, so the second variant is excluded. If the rule had instead been strictly less than 0.01, the first variant would also be excluded. Stating the boundary rule prevents ambiguity at the cutoff.
Why Ancestry-Specific Frequencies Can Matter
A pooled frequency summarizes the allele frequency across the people or chromosomes in the pooled sample. When ancestry groups have different allele frequencies, that summary depends on the groups’ representation. As a result, one fixed threshold applied to a pooled estimate can classify a variant differently from a rule that uses ancestry-specific estimates.
The calculation makes this dependence explicit. For a specified allele, let \(p_k\) be its frequency in ancestry group \(k\), and let \(w_k\) be that group’s share of the observed chromosomes, with \(\sum_k w_k=1\). When the same allele is being counted in each group, the pooled frequency is the weighted average of the group frequencies. The pooled MAF is then the smaller of that frequency and its complement.
This follows by dividing the total count of the specified allele across groups by the total number of observed chromosomes. If group \(k\) contributes \(C_k\) chromosomes and the allele has frequency \(p_k\), its expected allele count is \(C_kp_k\). Summing the counts gives \(\sum_k C_kp_k\); dividing by \(\sum_k C_k\) gives \(\sum_k (C_k/\sum_j C_j)p_k\), which is the weighted-average expression above. Thus, a group with more observed chromosomes has more influence on the pooled frequency.
An ancestry-specific rule can mean different things in different analyses. For an analysis conducted separately within each ancestry group, one may define a separate mask using that group’s frequency estimates. For a combined analysis, the investigator must state how group-specific estimates determine membership—for example, whether a variant must be below the cutoff in every represented group, or whether the mask is defined for a prespecified target group. There is no single rule that should be assumed from the phrase “ancestry-specific threshold.”
Worked Example: A Pooled Estimate Can Hide a Group Difference
Consider a hypothetical study in which 80% of observed chromosomes come from ancestry group A and 20% from group B. A specified allele has estimated frequencies 0.004 in group A and 0.028 in group B. Use a 0.01 cutoff, and define separate ancestry-specific masks by including a variant in a group’s mask when its MAF in that group is at most 0.01.
Because both frequencies are below 0.5, the specified allele is the minor allele in each group. The pooled frequency is:
The pooled estimate is 0.0088, so a pooled rule using \(\mathrm{MAF}\leq 0.01\) would include the variant. Under the separate ancestry-specific rules, the variant is included in group A because \(0.004\leq 0.01\), but excluded in group B because \(0.028>0.01\). These decisions differ because they answer different questions about which populations the frequency criterion describes.
The example does not imply that ancestry groups are internally uniform or that group labels identify biological causes. It shows only that a pooled frequency can be strongly affected by group representation and can conceal a frequency that exceeds the cutoff in one group.
Making an Ancestry-Specific Rule Defensible
Ancestry-specific frequency estimates can help avoid treating a variant as rare merely because it is rare in a large, differently composed reference population. They can also prevent variants that are common in one represented group from entering a mask intended to capture rare variation in that group. The rule is most interpretable when the analysis population, ancestry definition or assignment method, frequency source, and inclusion logic are all reported.
There are practical limits. A small ancestry stratum may provide too few observed chromosomes for stable estimates of very low frequencies. A variant observed once can have a markedly different estimated frequency depending on the stratum’s denominator. Differences in sequencing coverage, calling quality, or missingness can also affect frequency comparisons; frequency estimates should not be used to compensate for unresolved call-quality problems. External reference estimates can help, but only if the reference and study data are sufficiently comparable for the intended use.
An ancestry-specific mask also changes the set of variants being analyzed across groups. Comparisons of results then need to distinguish differences in the variant sets from differences in trait associations. The frequency cutoff does not by itself establish that variants have the same functional effect, that the same variants are represented in each group, or that any observed association is causal.
What Happens When the Cutoff Is Too Broad?
A broad cutoff can admit variants whose frequencies do not fit the intended rare-variant hypothesis. The resulting set may mix variants that differ in population frequency, biological interpretation, or relevance to the target question. A burden constructed from that set no longer has the simple interpretation “the combined signal from variants below the intended rare-frequency boundary.” The altered set can change an association result, but there is no general rule that a broader threshold must increase or decrease significance.
A narrow cutoff has a different cost: it excludes variants that may have been within the intended target range, reducing the set under consideration. The right choice is therefore not the smallest possible value or the value that produces the strongest result. It is the cutoff that gives a clear, defensible operational definition of the target set, given the study population and available frequency estimates.
Worked Example: How a Broader Cutoff Changes the Mask
Suppose a hypothetical sequencing study includes 5,000 unrelated individuals with complete diploid calls. Four variants have minor-allele counts of 5, 40, 90, and 150 among 10,000 observed chromosomes. Compare fixed thresholds of 0.01 and 0.02, both using the inclusive rule.
Their estimated MAFs are:
With the 0.01 threshold, the first three variants are included and the variant at 0.0150 is excluded: \(0.0005, 0.0040, 0.0090\leq0.01\), whereas \(0.0150>0.01\). With the 0.02 threshold, all four are included because each estimate is at most 0.02. The broader rule therefore adds one variant whose estimated frequency is 1.5%.
If the stated target is variants with MAF at or below 1%, the 0.02 mask does not match that target. If the intended target is instead variants with MAF at or below 2%, the broader mask is consistent with it. The calculations decide membership; the scientific question must decide which boundary is appropriate.
Common Mistakes and a Practical Checklist
- Calling a cutoff universal. The reasoning “0.01 is the rare-variant threshold” treats a study choice as a general law. State the numerical cutoff and the population or reference source to which it applies.
- Ignoring the equality rule. “Below 1%” could mean strictly less than 0.01, while “at most 1%” includes 0.01. Define the rule explicitly.
- Assuming a pooled estimate applies equally to every group. A pooled MAF is weighted by the observed chromosome contributions. Examine whether that pooled quantity matches the target population or whether group-specific rules are needed.
- Using ancestry-specific estimates without defining the policy. The phrase alone does not say whether a variant must be rare in all groups, in any group, or only in a particular group. State the decision rule for the analysis.
- Choosing the cutoff after seeing which one gives the preferred result. That makes the mask outcome-dependent. Specify the primary threshold in advance and describe additional thresholds as additional analyses.
A useful report states the cutoff, whether the inequality is inclusive, the frequency source and denominator, the population or ancestry groups used, and how missing or low-quality calls were handled. Together, these details make the variant mask reproducible and make clear what rare-variant set the analysis was designed to examine.
Check Your Understanding
Answer each question using the stated frequency rule and population context.
- A variant has estimated MAF 0.008 in a study sample. Is it included under an inclusive 0.01 cutoff? What would change if the rule were strictly less than 0.01 and its MAF were exactly 0.01?
- Why can the pooled frequency of a variant differ from its frequency in a smaller ancestry group?
- A variant is below 0.01 in one ancestry group and above 0.01 in another. What must be specified before applying an ancestry-specific rule in a combined analysis?
- Give two reasons why an external reference frequency estimate might not be interchangeable with a study-sample estimate.
- Why is it not sound practice to select a frequency cutoff solely because it gives the smallest association p-value?