Iterative Variant and Sample Filtering

Why Filtering Must Be Iterative

A genotype matrix records calls for samples at variants. Some entries are missing. A sample with many missing calls may fail a sample call-rate threshold; a variant with many missing calls may fail a variant call-rate threshold. These decisions are connected: removing a sample changes the denominator used for every variant call rate, while removing a variant changes the denominator used for every sample call rate.

Consequently, a list of samples and variants that pass when measured on the original matrix need not pass after filtering. Iterative filtering handles this dependence by recalculating rates on the currently retained matrix until no additional sample or variant fails. The procedure is a bookkeeping rule for call completeness, not a diagnosis of why calls are missing or a guarantee that retained calls are biologically correct.

Definition: Let $G$ be a genotype matrix with $n$ samples and $m$ variants. Define $R_{ij}=1$ if the call for sample $i$ at variant $j$ is observed, and $R_{ij}=0$ if it is missing. For retained sample set $S$ and retained variant set $V$, where both sets are nonempty, the current sample call rate and variant call rate are $$ C_i(S,V)=\frac{\sum_{j\in V}R_{ij}}{|V|},\qquad C_j(S,V)=\frac{\sum_{i\in S}R_{ij}}{|S|}. $$ A sample passes threshold $\tau_s$ when $C_i(S,V)\geq\tau_s$; a variant passes threshold $\tau_v$ when $C_j(S,V)\geq\tau_v$. Equality counts as passing.

The defining feature is that rates are calculated using only the current retained sets. At the start, $S$ and $V$ contain all samples and variants. After removals, the denominator for a sample is the number of retained variants, and the denominator for a variant is the number of retained samples. This is the operational setting for the call-rate definitions and threshold choices introduced in the earlier material on variant call rate, sample call rate, and choosing call-rate thresholds.

A Reproducible Iteration Rule

One clear convention is to use simultaneous rounds. In each round, calculate all rates using the same current matrix, mark every failing sample and variant, remove those marked items together, and then recalculate. Do not calculate some rates from an already-partly-updated matrix within the same round. This convention makes the update rule explicit and easy to reproduce.

Initialize.
Set $S$ to all samples and $V$ to all variants. Choose and record $\tau_s$ and $\tau_v$.

Calculate current rates.
Compute $C_i(S,V)$ for each retained sample and $C_j(S,V)$ for each retained variant using the same $S$ and $V$.

Remove failures.
Mark every sample below $\tau_s$ and every variant below $\tau_v$, then remove all marked items together.

Repeat or stop.
If at least one item was removed, recalculate on the reduced matrix. Stop when a full round removes nothing, provided both retained dimensions are nonempty.

The stopping condition is stability with respect to these specified thresholds: every retained sample and variant passes on the final retained matrix. It does not mean that every removed item would fail if reconsidered alone, because adding it back would change denominators. Nor does it imply that the thresholds are scientifically appropriate; those must be chosen for the study and assay.

Worked Example: Removing a Variant Makes a Sample Fail

Consider this hypothetical four-sample, four-variant matrix. “O” means observed and “–” means missing. Set both thresholds to $0.75$.

Sample	V1	V2	V3	V4
S1	O	O	O	O
S2	O	O	O	–
S3	O	O	O	–
S4	O	–	O	O

Initially, sample call rates are $4/4=1$, $3/4=0.75$, $3/4=0.75$, and $3/4=0.75$, so all samples pass. Variant rates are $4/4=1$, $3/4=0.75$, $4/4=1$, and $2/4=0.50$. Thus V4 fails and is removed in the first round.

Recalculate sample rates on V1, V2, and V3. S1 has $3/3=1$, S2 has $3/3=1$, S3 has $3/3=1$, and S4 has $2/3\approx0.667$. S4 now fails, even though it passed on the original matrix. Remove S4. On the remaining three samples and three variants, V1, V2, and V3 each have call rate $3/3=1$, and each sample has rate $3/3=1$. No further item fails, so the procedure stops with S1–S3 and V1–V3 retained.

The key change is not a change in any genotype entry: it is the denominator. S4’s call at V4 had contributed to its original numerator, but V4 was removed. This is why one pass over the original matrix would miss a sample that fails on the final matrix.

How Either Dimension Can Change the Other

Removing a sample does not always improve every variant rate. If a removed sample had an observed call at a particular variant, that call leaves the numerator as well as the sample leaving the denominator. The resulting rate depends on the calls in the removed row. Similarly, removing a variant can raise or lower a sample’s rate depending on whether that sample had an observed or missing call at that variant.

Worked Example: A Sample Removal Improves Variant Call Rates

In a second hypothetical matrix, use four samples and four variants, with both thresholds again equal to $0.75$.

Sample	V1	V2	V3	V4
S1	O	O	O	O
S2	O	O	O	O
S3	O	O	O	O
S4	–	–	–	–

Initially, S1, S2, and S3 each have rate $4/4=1$, and S4 has rate $0/4=0$, so S4 fails. Each variant has rate $3/4=0.75$, exactly the threshold, and therefore passes. After removing S4, each variant rate becomes $3/3=1$; all three retained samples also have rate $4/4=1$. The matrix is stable after this round.

This example shows that a variant’s rate can rise when a sample is removed. In the preceding example, removing a variant caused a sample rate to fall. The direction of change depends on the removed row or column’s pattern of observed and missing calls; it is not safe to assume filtering only improves all rates.

Worked Example: A Sample Removal Makes a Variant Fail

Consider another hypothetical matrix with thresholds $\tau_s=\tau_v=0.75$.

Sample	V1	V2	V3	V4
S1	O	O	O	O
S2	O	O	O	O
S3	–	O	O	O
S4	O	–	–	–

Initially, S4 has $1/4=0.25$ and fails. Each of V1–V4 has three observed calls out of four, so each variant has rate $3/4=0.75$ and passes. Remove S4 and recalculate. V1 now has $2/3\approx0.667$, so it fails; V2, V3, and V4 each have $3/3=1$. The remaining samples all pass on these three variants: S1 and S2 each have $3/3=1$, and S3 also has $3/3=1$. Remove V1, then recalculate. All three retained samples have rate $3/3=1$, and each retained variant has rate $3/3=1$, so the process stops.

Here, the removed sample had an observed call at V1. Losing that sample reduced V1’s numerator from three to two, while the denominator fell from four to three. The rate therefore fell from $0.75$ to about $0.667$. This is a concrete reason to recalculate both dimensions after every round.

Why the Procedure Terminates, and What It Guarantees

As long as both dimensions remain nonempty, every round that continues removes at least one sample or variant. No removed item is reintroduced. Starting with $n$ samples and $m$ variants, there can be at most $n+m$ individual removals. Therefore, the process must either reach a round with no failures or exhaust a dimension first. Exhaustion is not a successful stable result: call rates with a zero denominator are undefined, so the procedure should stop with a warning rather than report empty-matrix rates.

The finite-termination argument is simple. Let $N_r=|S_r|+|V_r|$ be the total number of retained rows and columns at the start of round $r$. If that round removes anything, then $N_{r+1}<N_r$. Since $N_r$ is a nonnegative integer, it cannot decrease indefinitely. If the procedure has not exhausted a dimension, it eventually reaches a round with no removals, which is precisely the stated stability condition.

This guarantee concerns the algorithm’s stopping, not the uniqueness or desirability of its retained set under every possible implementation. Simultaneous rounds provide one precise convention. An implementation that removes items one at a time and recalculates immediately may follow a different path in edge cases. For reproducibility, report whether removals were simultaneous or sequential, the thresholds, the order of operations, any minimum-size safeguards, and the number of rounds.

Diagnostics, Alternatives, and Limitations

Useful checks are mechanical. Keep a round-by-round record of the numbers of retained samples and variants, the failures in each dimension, and the rates of items removed. At termination, verify directly that every retained sample has rate at least $\tau_s$ and every retained variant has rate at least $\tau_v$, using the final denominators. Also verify that the retained matrix has nonzero dimensions and that the software’s missing-value encoding is being treated consistently.

Several seemingly reasonable alternatives answer different questions:

Approach	What it does	Main limitation
One pass on the original matrix	Removes items that fail only on the starting matrix.	Can leave failures that arise after row or column removals.
Iterative simultaneous filtering	Recalculates both sets of rates after each batch of removals until stable.	May remove many items in cascades; the result depends on thresholds and the specified update rule.
Filter only samples or only variants	Applies one dimension’s threshold without alternating updates.	Does not ensure that the other dimension passes on the final matrix.

A common mistake is to say, “S4 passed the sample threshold initially, so it must be retained.” That reasoning uses the original denominator after the matrix has changed. The correction is to calculate S4’s rate using the variants actually retained at that round. Another mistake is to assume that removing missing calls’ rows or columns always raises the opposite rates. As the third example shows, deleting a row that contains a call can lower a variant’s rate.

Iterative call-rate filtering is limited to completeness. A stable matrix does not establish that missingness is random, that the calls are accurate, or that retained samples and variants are free of other quality concerns. It can also discard data through a cascade: a marginal failure in one dimension may trigger a failure in the other. These consequences should be examined before interpreting downstream analyses, while questions about patterns and causes of missingness require separate diagnostics.

Key takeaway: Define call rates on the current retained matrix, remove failures according to an explicit update rule, and recalculate both dimensions until stable. The final pass/fail statements are valid only for the final retained denominators.

Check Your Understanding

Answer each question using the iterative rule and definitions above. Solutions follow the questions.

A variant has 8 observed calls among 10 retained samples and a threshold of $0.80$. Does it pass? What if two samples are removed and both had observed calls at this variant?
Why can a sample that passes before variant filtering fail after some variants are removed?
In a round, a sample fails but every variant passes. Should the variant rates be left unchanged for the next round? Explain.
State the stopping condition and one situation in which the procedure cannot report a stable nonempty matrix.

Solutions. (1) It passes because $8/10=0.80$, and equality passes. If both removed samples had observed calls, the updated rate is $6/8=0.75$, so it fails. (2) Removing variants changes the sample-rate denominator and may remove observed calls from its numerator. If the removed variants contain proportionally more observed calls for that sample than the retained variants, the rate on the remaining variants can be lower. (3) No. Remove the failing sample according to the round rule, then recalculate variant rates on the remaining samples; a variant can rise or fall depending on whether the removed sample had a call there. (4) Stop when a complete round removes no samples or variants and all retained items meet their thresholds on the current matrix. If removals leave no samples or no variants, the corresponding call-rate denominators are zero, so a stable nonempty result cannot be reported.