Why Threshold Choice Requires More Than a Rule of Thumb
A call rate summarizes the fraction of genotype entries that are observed rather than missing. The earlier material on variant call rate and sample call rate establishes the two relevant summaries: a rate for each variant across samples, and a rate for each sample across variants. Choosing a cutoff turns these summaries into a decision about what to retain. A stringent cutoff can remove many low-quality entries or units, but it can also discard useful variants or samples; a lenient cutoff preserves more data but allows more missingness.
There is no universal cutoff that is appropriate for every study. A defensible choice considers the observed distribution of rates, the practical number of variants or samples needed, and the consequences of excluding units. It also makes explicit the data and assumptions used to estimate retention. A cutoff is a quality-control decision, not proof that everything retained is accurate.
The denominators matter. A variant rate is calculated across samples, while a sample rate is calculated across variants. A threshold of 98% therefore means different minimum numbers of calls in these two settings whenever \(n\) and \(m\) differ. The units being filtered also differ: a variant cutoff determines which loci remain, and a sample cutoff determines which individuals remain.
Turn the Cutoff Into a Retention Count
If a rate is based on \(M\) possible calls and the cutoff is \(c\), the number of calls required for retention is \(\lceil Mc\rceil\), where \(\lceil x\rceil\) is the smallest integer at least \(x\). This integer conversion prevents ambiguity from rounding a displayed percentage. It also makes the effect of a proposed threshold concrete.
Worked Example: Translating a Variant Cutoff Into Calls
Suppose a hypothetical variant is assessed in 1,000 samples, and the proposed variant call-rate threshold is 98%. The minimum call count is
Thus the variant must have at least 980 nonmissing genotype calls and can have at most \(1{,}000-980=20\) missing calls. If it has 979 calls, its rate is \(979/1{,}000=0.979\), or 97.9%, so it does not meet the cutoff. The calculation uses the number of samples actually assessed, not a rounded sample count or a percentage alone.
A useful first step is to examine how many units would remain at several candidate cutoffs. The following hypothetical distribution summarizes call rates for 20,000 variants. The intervals are defined so their endpoints do not overlap.
| Variant call-rate interval | Number of variants |
|---|---|
| 99.5% to 100% | 15,000 |
| 98% to less than 99.5% | 3,000 |
| 95% to less than 98% | 1,500 |
| 90% to less than 95% | 400 |
| Less than 90% | 100 |
Worked Example: Comparing Variant Thresholds With the Observed Distribution
For the hypothetical distribution above, consider retaining variants with call rate at least 98%, then compare that with a cutoff of at least 95%.
At 98%, the retained count is \(15{,}000+3{,}000=18{,}000\). The proportion retained is \(18{,}000/20{,}000=0.90\), or 90%. At 95%, the retained count is \(15{,}000+3{,}000+1{,}500=19{,}500\), or \(19{,}500/20{,}000=97.5\%\). The stricter 98% cutoff excludes 1,500 more variants than the 95% cutoff in this data set.
These counts describe the consequences of the candidate rules; they do not establish which rule is scientifically preferable. The 1,500 variants between 95% and 98% may or may not be useful, and call rate alone does not tell us whether their missing calls are harmless or problematic. The cutoff should be justified against the study’s quality requirements and its need to retain informative loci.
Use Sample Retention Targets Explicitly
The same planning principle applies to samples. Before choosing a sample-level cutoff, determine how many samples the study needs to retain, or what loss the study can reasonably tolerate. Inspecting the observed rate distribution shows whether a candidate threshold meets that target. If it does not, choosing a lower threshold merely to reach the target is not automatically justified; the trade-off in data quality must also be considered.
Worked Example: Choosing a Sample Cutoff Against a Target
A hypothetical study has 120 samples. Their sample call rates are summarized as follows: 102 have rates from 99% through 100%, 10 have rates from 98% to less than 99%, 6 have rates from 95% to less than 98%, and 2 have rates below 95%. The investigators want to retain at least 115 samples if quality allows.
At a 98% cutoff, \(102+10=112\) samples remain, which is fewer than the target of 115. At a 95% cutoff, \(102+10+6=118\) remain, which exceeds the target. Thus, 95% is the less stringent of these two candidate cutoffs that meets the stated retention target. The result does not by itself prove that a 95% cutoff is acceptable: the investigators must decide whether permitting up to 5% missing calls per retained sample is suitable for the study.
If the study required at least 115 samples but also required each retained sample to have at least 98% of calls observed, these two requirements would conflict in this example. The honest conclusion is that the observed data do not meet both conditions, not that the rate summaries should be reinterpreted to make them agree.
Expected Retention Under a Simple Missingness Model
Observed distributions are usually the most direct basis for counting retention in an existing data set. A probability model can also help plan a study, but only if its assumptions are stated. Suppose a sample is assessed at \(m\) variants, each call is independently missing with the same probability \(q\), and missingness does not vary across variants or samples. Then the number \(Y\) of missing calls for that sample follows \(Y\sim\operatorname{Binomial}(m,q)\). At a sample call-rate cutoff \(c\), the sample is retained when it has at most \(m-\lceil mc\rceil\) missing calls.
If there are \(N\) comparable samples, the expected number retained is \(N\) times this retention probability. This follows by writing the retained count as a sum of \(N\) indicator variables, one for each sample; the expectation of a sum is the sum of the expectations. Independence between samples is not required for this expectation, though it affects the variability of the total count.
Worked Example: Expected Sample Retention Under a Binomial Model
Consider a deliberately small hypothetical planning calculation with \(N=200\) samples, \(m=20\) variants per sample, and an independent missing-call probability \(q=0.05\) for every call. For a 90% sample cutoff, a sample needs at least \(\lceil20(0.90)\rceil=18\) calls, so it can have at most 2 missing calls. Under the stated model, \(Y\sim\operatorname{Binomial}(20,0.05)\), and
The expected number retained is \(200(0.9245)=184.9\), approximately 185 samples. This is an expectation, not a promise that exactly 185 will pass. Also, the model assumes identical, independent missingness across calls. Real data may have variant-specific or sample-specific missingness, so the calculation is best treated as a planning approximation, not as a replacement for inspecting observed rates.
Absolute Cutoffs, Percentiles, and Practical Consequences
An absolute cutoff states a minimum acceptable call rate, such as 98%. A percentile rule instead removes a specified fraction of units ranked by their rates, such as excluding the lowest-rate 5% of variants. These answer different questions. An absolute cutoff sets a quality criterion, while a percentile rule sets the number removed relative to the observed distribution.
| Choice | What it controls | Main limitation |
|---|---|---|
| Absolute rate cutoff | The minimum call rate required for retention | The number removed depends on the data’s rate distribution |
| Percentile rule | The fraction of ranked units excluded | The retained rates may still be poor if overall quality is poor |
| Retention target | The desired number or proportion of units kept | Meeting the target does not establish acceptable quality |
For example, excluding the lowest-rate 5% of variants guarantees that 95% are not excluded by that ranking rule, subject to ties and the precise implementation. It does not guarantee that those retained variants meet a meaningful call-rate standard. If the entire distribution is low, the retained 95% can still have low rates. Conversely, if nearly all rates are high, a percentile rule may discard some variants that satisfy a reasonable absolute criterion.
Overly strict thresholds can reduce the number of available samples or loci, potentially limiting the study’s ability to address its scientific question. Overly lenient thresholds retain more units but allow more missing calls. Neither outcome is automatically preferable. A practical decision compares candidate cutoffs with observed retention counts and with explicit study requirements, then records the cutoff and its consequences.
Diagnostics, Failure Modes, and Limits
Before applying a threshold, verify that rates are calculated over the intended set of samples or variants and that the missing-value encoding has been handled correctly. A mistaken denominator or a value incorrectly treated as a call can change which units pass. Also check that the count table and the rule agree at the boundary: a threshold of at least \(c\) retains a rate exactly equal to \(c\), not only rates greater than \(c\).
- A sharp jump in exclusions near a candidate cutoff. Compare counts at nearby thresholds. A small change in cutoff that removes many additional units deserves explicit review.
- A percentile cutoff with an unacceptably low retained rate. Inspect the rates themselves; ranking units does not make the retained group adequate.
- A sample target that conflicts with the quality criterion. Report the conflict rather than lowering the cutoff without justification.
- A call rate treated as a guarantee of accuracy. A high call rate measures completeness, not whether each genotype call is correct.
- Rates rounded before applying the rule. Apply the cutoff to unrounded counts or rates where possible, so boundary decisions are reproducible.
Call-rate thresholds are also limited because a single summary compresses a pattern of missing calls into one number. Two samples with the same rate can differ in which variants are missing, and two variants with equal rates can differ in which samples lack calls. A rate cutoff does not establish why calls are missing or whether missingness is related to a scientific outcome. Those questions require separate analyses; passing a call-rate filter should not be interpreted as resolving them.
Common Mistake
A common mistake is to say, “Removing the lowest 5% guarantees that the remaining variants have acceptable call rates.” This confuses a relative ranking rule with an absolute quality standard. The rule guarantees only that the lowest-ranked fraction is excluded, subject to ties and implementation details. To claim that retained variants have acceptable rates, specify an acceptable absolute cutoff and verify that each retained rate meets it.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A variant is assessed in 250 samples. How many calls are required to meet a 96% call-rate cutoff, and how many missing calls are allowed?
- There are 80 samples: 70 have call rates at least 98%, and 76 have rates at least 95%. How many samples are excluded at each cutoff?
- Why does excluding the lowest-rate 5% of variants by a percentile rule not guarantee that the retained variants have acceptable call rates?
- Under the binomial model, suppose a sample is assessed at 10 calls, each independently missing with probability 0.10. At a 90% call-rate threshold, what is the probability that the sample is retained?
- State one reason that a high call rate does not establish that genotype calls are accurate.
Solutions. (1) The required count is \(\lceil250(0.96)\rceil=\lceil240\rceil=240\); at most \(250-240=10\) calls may be missing. (2) At 98%, \(80-70=10\) are excluded. At 95%, \(80-76=4\) are excluded. (3) The percentile rule removes a fraction ranked lowest, but the whole distribution could have low rates; the remaining 95% need not meet any specified absolute standard. (4) Retention requires at least 9 calls, or at most 1 missing call. For \(Y\sim\operatorname{Binomial}(10,0.10)\), \(P(Y\le1)=0.9^{10}+10(0.10)(0.9)^9\approx0.7361\). (5) Call rate measures the fraction of nonmissing calls, not whether those observed calls are correct; a sample can have few missing calls and still contain inaccurate genotypes.