Why Relatedness Filtering Must Match the Analysis
A relatedness flag is not, by itself, a reason to remove a sample from every analysis. Close relatives may violate an independence assumption in an analysis designed for unrelated individuals, yet be the observations of interest in a family-based analysis. Filtering all flagged relatives from the study dataset can therefore remove useful family information, break pedigrees, and change the scientific question.
The practical solution is to make the rule analysis-specific. Keep the quality-controlled samples and their family relationships available for analyses that use relatives. When an analysis requires unrelated individuals, create a separate sample subset that meets that requirement. The earlier material on pairwise relatedness for QC and relatedness thresholds for sample filtering supplies the pairwise estimates and threshold choices; here the focus is how to apply them without confusing these two analysis goals.
The exact estimator, cutoff, and equality convention must be fixed before filtering. A threshold is a decision rule applied to estimates, not a biological boundary that perfectly classifies every pair. As a result, the subset is unrelated only with respect to the stated estimator and rule. Keep the full pairwise results and the rule used so that the selection can be checked and reproduced.
Separate the Family Cohort From the Unrelated Subset
For a family-based analysis, retain the relatives required by its design, subject to the study’s ordinary sample-quality and eligibility decisions. Do not apply an unrelated-only rule to that analysis cohort simply because some pairs exceed a relatedness cutoff. Instead, preserve sample identifiers, family identifiers, and the relationships needed by the planned analysis.
For an analysis that requires unrelated samples, apply the conflict rule to the eligible samples and select a subset with no flagged pair. When there are several feasible subsets, the selection should reflect the analysis objective. One transparent formulation uses a binary indicator \(x_i\), equal to 1 when sample \(i\) is retained and 0 otherwise, and a prespecified priority \(w_i\) for retaining that sample.
The constraint prevents both members of any flagged pair from entering the unrelated subset. Priorities might represent prespecified preferences such as phenotype completeness or a particular sampling objective; they should not be invented after seeing which samples would be retained. If the goal is simply to retain the largest possible number of samples, maximize \(\sum_i x_i\) directly. Equivalently, maximize a weighted sum only when every sample has the same strictly positive weight. With zero or negative weights, maximizing the sum need not maximize the number retained.
This formulation also clarifies what is and is not guaranteed. It produces a feasible set with no conflicts under the chosen rule. It does not ensure that the retained individuals represent the original cohort, preserve a particular ancestry balance, or have the most complete data for every downstream analysis unless those goals are included in the selection design.
Worked Examples
Worked Example: Keeping a Family Intact but Selecting an Unrelated Subset
Suppose a hypothetical family contains two parents, \(P_1\) and \(P_2\), and their children, \(C_1\) and \(C_2\). All four samples pass the relevant genotype-quality checks. Let the unrelated-analysis cutoff be \(t=0.177\). For this illustration, the flagged pairs are parent–child pairs and the sibling pair; the parents are not flagged as a pair. The family-based analysis needs all four people, while a separate analysis requires an unrelated subset.
Family-based cohort. Retain \(P_1\), \(P_2\), \(C_1\), and \(C_2\) for the family analysis. The flagged relationships are part of the design, not reasons to erase relatives from that cohort.
Unrelated subset. Suppose the prespecified positive priorities are \(w_{P_1}=2\), \(w_{P_2}=2\), \(w_{C_1}=5\), and \(w_{C_2}=1\). Each child conflicts with both parents, and the two children conflict with each other. Retaining both parents is feasible and gives \(2+2=4\). Retaining \(C_1\) alone gives 5, while retaining \(C_2\) alone gives 1. A parent cannot be retained with either child, and both children cannot be retained together. Thus the highest-priority feasible selection is \(\{C_1\}\), with objective value 5.
The two outputs answer different analytic needs: the family cohort preserves the pedigree, while the unrelated subset satisfies the specified pairwise rule. The choice of \(C_1\) is justified only by the stated priorities; it is not a claim that a child is generally preferable to a parent.
Worked Example: A Connected Group Does Not Mean “Keep One”
Consider three hypothetical samples \(A\), \(B\), and \(C\), with cutoff \(t=0.177\). Their estimated pairwise values are \(\hat r_{AB}=0.22\), \(\hat r_{BC}=0.24\), and \(\hat r_{AC}=0.08\). Thus \(A\) conflicts with \(B\), and \(B\) conflicts with \(C\), but \(A\) and \(C\) do not conflict under this rule.
Using equal, strictly positive weights, the constraints are \(x_A+x_B\leq1\) and \(x_B+x_C\leq1\). The selection \(A,C\) is feasible and retains two samples. Any feasible selection containing \(B\) can contain neither \(A\) nor \(C\), so it retains only one. Therefore \(A,C\) is a largest feasible subset.
All three samples belong to one connected group if connections are drawn for flagged pairs, but the endpoints \(A\) and \(C\) are not a flagged pair. Keeping only one sample from every connected group would unnecessarily discard one of them. The relevant condition is absence of a conflict between every pair in the retained subset, not membership in separate connected groups.
Worked Example: Handling Two Genotypes From One Person
Suppose a hypothetical participant, \(U\), was genotyped twice as samples \(U_1\) and \(U_2\). The duplicate check flags this pair, and both records represent the same biological person. Their call rates are 99.2% for \(U_1\) and 97.8% for \(U_2\); all other relevant checks are equal.
For an analysis whose rows represent people, retaining both records would count one person twice. Retain \(U_1\) as the representative record because it has the higher call rate, remove \(U_2\) from that analysis, and record the mapping from both assay identifiers to participant \(U\). The comparison is \(99.2\%>97.8\%\), a difference of 1.4 percentage points.
This is distinct from removing a relative from a family-based analysis: the duplicate records are two measurements of one person, whereas relatives are distinct people. If a family analysis needs a particular genotype record, the selected representative should be documented and used consistently; the duplicate record should not be treated as an additional family member. Retain both records in an audit trail even when only one is used as the participant-level observation.
Why the Selection Rule Works, and How to Check It
The no-conflict requirement follows directly from the pairwise constraints. If \(x_i=x_j=1\) for a conflicting pair, then \(x_i+x_j=2\), which violates the required inequality \(x_i+x_j\leq1\). Conversely, any selection satisfying every such inequality cannot contain both endpoints of any conflict. Thus the constraints are equivalent to the definition of a feasible unrelated subset.
There may be multiple feasible subsets, and the objective determines which one is chosen. When all weights equal a constant \(c>0\), the objective is \(c\sum_i x_i\). Since multiplying the retained count by a positive constant preserves its ordering, maximizing the weighted sum in this special case is exactly maximizing the number of retained samples. Positivity matters: if the common weight is negative, the objective rewards retaining fewer samples; if it is zero, all feasible subsets tie.
A useful representation is a graph: each sample is a vertex, and an edge joins every conflicting pair. A valid unrelated subset contains no edge between any two of its selected vertices. For small sets, feasible selections can be compared directly. For larger sets, the maximum-size or maximum-weight selection is a combinatorial optimization problem; the graph representation makes the objective and constraints explicit, but does not make every selection problem trivial. Whatever search or heuristic is used, the final selected set can be checked by verifying that no flagged edge has both endpoints retained.
Diagnostics, Alternatives, and Limitations
Before using the output, inspect the retained and excluded sample lists alongside the pairwise flags. Useful checks include confirming that every flagged pair has at most one member retained; checking that duplicate records map to the intended participant; confirming that the family cohort still contains the relatives required by its analysis; and verifying that no sample was excluded merely because it was connected to another sample through a chain of edges. Review the sizes and composition of both the family cohort and the unrelated subset.
Also examine whether the result is sensitive to plausible changes in the threshold or to the relatedness estimates near that threshold. Small changes can alter edges and therefore alter the chosen subset. If estimates may be affected by ancestry differences, poor genotype quality, or other data features, investigate those issues rather than treating the graph as error-free. A clean optimization result cannot repair an inappropriate estimator, an unsuitable cutoff, or inaccurate sample identities.
| Approach | What it does | Main caution |
|---|---|---|
| Keep the family cohort and create a separate unrelated subset | Preserves relatives for family analyses while satisfying a stated rule for unrelated-sample analyses | Requires analysis-specific sample lists and careful documentation |
| Remove one member of every flagged pair from the only study dataset | May produce an unrelated subset | Can discard relatives needed for family analyses and make the dataset unsuitable for them |
| Keep one sample per connected group | Uses graph connectivity as a simple pruning shortcut | Can discard compatible samples, as \(A\) and \(C\) show in the path example |
No single filtering choice is optimal for every analysis. An unrelated subset may be smaller, may omit informative family members, and may reflect the chosen priorities rather than a neutral sample of all participants. A family cohort, in contrast, should not be described as unrelated just because a separate unrelated subset has been made. Keep the analysis purpose, threshold, priorities, chosen sample identifiers, and exclusions together in the QC record so downstream users can identify which population each result represents.
Common Mistakes
- Applying an unrelated-sample filter to every analysis. A relative can be essential for a family-based analysis. Keep the family cohort available and make an analysis-specific subset.
- Keeping one sample per connected group. A chain of pairwise conflicts does not mean every pair in the group conflicts. Check the actual edges among proposed retained samples.
- Maximizing an arbitrary weighted sum. A weighted objective reflects the stated priorities only when those weights are appropriate. To maximize count, maximize the count directly or use equal strictly positive weights.
- Treating duplicate assays as distinct relatives. Two records for one participant are not two people. Select and document one representative for person-level analyses.
- Calling the retained samples biologically unrelated. State that they have no pair flagged under the specified estimate and cutoff; the operational rule is not proof of zero relatedness.
Check Your Understanding
Answer each question, then compare with the solutions below.
- With a cutoff of 0.20, three samples have estimates \(\hat r_{AB}=0.25\), \(\hat r_{BC}=0.23\), and \(\hat r_{AC}=0.11\). What is a largest unrelated subset?
- Why should a family-based analysis not automatically use the unrelated subset?
- When does maximizing a weighted sum of retained samples also maximize the number retained?
- Two records are flagged as duplicates but represent one participant. What should be done before a participant-level analysis?
Solutions. (1) Retain \(A\) and \(C\): their estimate \(0.11\) is below \(0.20\), while \(B\) conflicts with each of them. Two is the largest feasible count. (2) The relatives may be required by the family design; removing them can break the pedigree and discard the observations the analysis is intended to use. Maintain the family cohort and use an unrelated subset only for analyses that require it. (3) When all samples have the same strictly positive weight, because the weighted objective is that positive constant multiplied by the retained count. (4) Choose one genotype record as the participant’s representative using a documented rule, avoid counting both records as separate people, and preserve the duplicate mapping and audit trail.