Why a Relatedness Threshold Is a QC Decision
Pairwise relatedness estimates summarize how much genetic material two samples share identical by descent (IBD). In sample QC, a threshold turns those estimates into a screening rule: pairs at or above the cutoff are flagged for review. The cutoff is not a universal boundary between “related” and “unrelated.” Its meaning depends on the QC task: a rule intended to flag possible first-degree pairs differs from one intended to identify a broader range of close relatives.
The earlier material on pairwise relatedness establishes how to estimate sharing for a pair. Here the focus is how to interpret a chosen cutoff, make it fit the intended task, and handle pairs whose estimates lie close to it. A flag is a prompt for review, not by itself a decision to exclude a sample or a definitive statement about a family relationship.
PI_HAT estimates the fraction of alleles shared IBD, not the fraction of loci at which two samples happen to have the same genotype. A match by state can occur without sharing an allele IBD. The IBD proportions are estimates from the observed genotypes and marker set, so \(\hat{\pi}_{ij}\) is also an estimate, not an error-free pedigree label.
Relationship Expectations and What a Cutoff Can Separate
Under the standard pedigree model, expected IBD sharing provides useful reference points. For unrelated pairs, the expected \(Z_1\) and \(Z_2\) are zero, so expected PI_HAT is zero. Parent–offspring pairs share one allele IBD across the genome under the model: \(Z_1=1\), \(Z_2=0\), and expected PI_HAT is \(0+\tfrac{1}{2}(1)=0.5\). Full siblings have \(Z_0=0.25\), \(Z_1=0.5\), and \(Z_2=0.25\), giving expected PI_HAT \(0.25+\tfrac{1}{2}(0.5)=0.5\). Duplicate samples and monozygotic twins are expected to have PI_HAT near 1.
Second-degree relationships such as half-siblings, grandparent–grandchild pairs, and avuncular pairs have expected PI_HAT around 0.25 under the usual model. These are expectations across the genome or across repeated inheritance realizations. The estimate for any particular pair can differ, and different relationships can have the same expectation. Thus, a threshold can screen for sharing above a chosen level, but PI_HAT alone does not identify a unique pedigree.
Worked Example: Interpreting a Screening Cutoff
Suppose a hypothetical QC team sets \(t=0.20\) and flags a pair when \(\hat{\pi}_{ij}\geq 0.20\). Three hypothetical pairs have estimated PI_HAT values 0.03, 0.22, and 0.48.
For the first pair, \(0.03<0.20\), so it is not flagged by this rule. This result does not prove that the samples are unrelated; it says only that their estimated PI_HAT does not reach this screening threshold.
For the second pair, \(0.22\geq0.20\), so it is flagged. Its estimate is near the expected value for a second-degree pair, but that numerical proximity does not establish a second-degree pedigree.
For the third pair, \(0.48\geq0.20\), so it is flagged as well. The estimate is near the expected first-degree value of 0.5, but the threshold alone does not distinguish a parent–offspring pair from full siblings. The QC conclusion is that the pair meets the stated flagging rule and merits review.
Choose the Threshold From the Intended QC Task
Before selecting a number, state what the rule is meant to screen for. For example, a broad close-relatedness screen may aim to flag possible second-degree or closer pairs. A more targeted screen may aim to flag likely first-degree pairs while not routinely flagging pairs near the second-degree expectation. These tasks call for different cutoffs and different tolerance for missed or extra flags.
A threshold near 0.20 is sometimes used as a broad screen for close relatedness, while a value between the second- and first-degree expectations can be used to focus on pairs more consistent with first-degree sharing. These are operational examples, not universal standards: the appropriate values depend on the estimator, marker set, study design, and the cost of false flags versus missed pairs. A cutoff should be documented together with the estimation method and the intended screening target.
The logic of a threshold placed between two reference expectations can be made explicit. If the only two reference values of interest are 0.25 and 0.50, their midpoint is \((0.25+0.50)/2=0.375\). Under a simplified rule that assigns a value to whichever reference is closer, 0.375 is equally distant from both. This arithmetic justifies the midpoint only as a boundary under that simplified criterion. It does not account for the distributions of estimates, unequal consequences of errors, or uncertainty in real data, so it does not make 0.375 a universally optimal cutoff.
Worked Example: Matching a Cutoff to a Screening Goal
A hypothetical project wants an initial screen for pairs with estimated sharing more consistent with first-degree than second-degree reference values. It uses the midpoint \(t=0.375\), with the flagging rule \(\hat{\pi}_{ij}\geq t\). This is a screening convention, not a claim that all estimates above the cutoff are first-degree relatives.
A pair has \(\hat{\pi}_{ij}=0.36\). Since \(0.36<0.375\), the pair is not flagged by this particular rule. Its estimate is still close to the cutoff and should not be described as definitively second-degree or unrelated.
A second pair has \(\hat{\pi}_{ij}=0.41\). Since \(0.41\geq0.375\), it is flagged for review under this rule. Its estimate is 0.09 from 0.50 and 0.16 from 0.25, so it is closer to the first-degree reference value, but that comparison does not prove a first-degree relationship.
This rule serves the stated screening goal only if the project is prepared to review pairs near the boundary and if the estimator is sufficiently comparable across pairs. If the primary goal instead were to screen broadly for possible second-degree pairs, this cutoff could miss relevant pairs; the task, not the arithmetic alone, determines whether the rule is suitable.
Borderline Estimates, Diagnostics, and Failure Modes
An estimate just above or below \(t\) should not be treated as categorically different from one on the other side. Estimation variability, marker selection, and genotype quality can move a pair across the boundary. A practical protocol can define a review band around the cutoff, but its width should be justified by the method's precision and the project's QC policy rather than chosen as though it were a universal constant.
For a borderline pair, check that the estimate is based on an appropriate, comparable marker set and review relevant genotype quality and missingness. Consider whether the estimate is stable under reasonable changes to the marker set or estimation settings. If a pair changes status under small, defensible changes, record it as borderline and investigate rather than presenting the threshold classification as certain. Do not compare estimates from different estimators or marker sets as though they necessarily share identical calibration.
Important failure modes include using a cutoff without stating the flagging convention, interpreting a flag as a confirmed pedigree, and applying one threshold to a task it was not designed to serve. Population structure, ancestry differences, linkage disequilibrium, genotyping errors, missingness, and a sparse or unrepresentative marker set can affect estimates or their comparability. When these conditions are not adequately controlled, a nominal cutoff may produce misleading flags or fail to identify pairs of interest.
Worked Example: Reviewing a Pair Near the Boundary
A hypothetical project uses \(t=0.20\). One pair has an initial estimate \(\hat{\pi}_{ij}=0.198\), just below the cutoff. The team checks the analysis records and finds that a small set of markers with elevated missingness was included. After applying its prespecified marker-quality procedure, the estimate is 0.204.
Under the stated rule, the initial estimate is not flagged because \(0.198<0.20\). The revised estimate is flagged because \(0.204\geq0.20\). The change is only \(0.204-0.198=0.006\), so it would be misleading to describe the revised pair as clearly different from a pair at 0.198.
The appropriate conclusion is that the pair is borderline and meets the flagging rule on the quality-controlled marker set. The team should report which estimate and marker set govern the rule, and review the pair in context. It should not select whichever estimate produces a preferred result or treat the cutoff crossing as proof of a specific relationship.
Comparing Threshold Policies
Different QC policies trade off the breadth of screening against the number of pairs requiring review. A lower cutoff tends to flag more pairs, including estimates in the second-degree range, but may also produce more borderline or incidental flags. A higher cutoff focuses the screen on larger estimates but can miss pairs relevant to a broader close-relatedness task. The right balance depends on the purpose and the consequences of each kind of error.
| Policy choice | What it is designed to flag | Main caution |
|---|---|---|
| Broad, lower cutoff | Pairs with estimates compatible with a wider range of close relationships | More review burden; a flag is not a pedigree diagnosis |
| Higher cutoff between reference expectations | Pairs with estimates more consistent with the higher-sharing class than a lower-sharing reference | Pairs near the boundary remain ambiguous; some close pairs may fall below it |
| Relationship-category classification | Attempts to summarize estimates using expected ranges for relationship classes | Ranges can overlap, and a category label may overstate what PI_HAT alone establishes |
A threshold rule is simpler and more reproducible than interpreting every estimate informally, but it compresses information: a pair at the cutoff and one far above it both receive a flag. Reporting the estimate alongside the flag preserves useful detail. Whatever policy is selected, state the threshold, whether equality counts as a flag, the PI_HAT convention, and the marker and estimator settings used.
A Practical Decision Sequence
State which range of relatedness the QC process is intended to flag and what kinds of pairs may be missed.
Set \(t\), define whether a pair is flagged for \(\hat{\pi}_{ij}\geq t\) or for a strict inequality, and record the estimator and marker set.
Review data quality and assess whether reasonable analysis choices materially change the pair's estimate or flag status.
Give the estimate and threshold, and describe borderline cases as uncertain rather than assigning an unsupported pedigree label.
Common Mistakes
- Using a doubled PI_HAT scale. PI_HAT is \(\hat{Z}_2+\tfrac{1}{2}\hat{Z}_1\), not \(2\hat{Z}_2+\hat{Z}_1\). The doubled expression would put standard relationship expectations on the wrong scale.
- Calling the cutoff a biological boundary. A pair just above \(t\) is not necessarily meaningfully more related than one just below it. The threshold is a chosen screening rule applied to estimates.
- Equating a flag with a relationship diagnosis. A PI_HAT value can be compatible with several pedigree relationships, and estimation error can affect the value.
- Applying one cutoff to every goal. A broad screen for possible second-degree pairs and a targeted screen for higher sharing have different aims and may require different rules.
- Ignoring data comparability. Estimates based on different marker sets or affected by different quality problems may not be directly comparable.
Check Your Understanding
Answer each question using the stated PI_HAT convention and threshold rule.
- A pair has \(\hat{Z}_2=0.10\) and \(\hat{Z}_1=0.30\). What is its PI_HAT?
- With \(t=0.20\) and the rule \(\hat{\pi}_{ij}\geq t\), is an estimate of 0.20 flagged?
- Why does an estimate of 0.49 not by itself establish that a pair is parent–offspring?
- A pair's estimate changes from 0.197 to 0.203 after a defensible marker-quality change, with \(t=0.20\). What should be reported or checked?
Solutions. (1) \(\hat{\pi}_{ij}=0.10+\tfrac{1}{2}(0.30)=0.25\). (2) Yes: equality meets the stated \(\geq\) rule. (3) Parent–offspring pairs have expected PI_HAT 0.5 under the standard model, but full siblings have the same expectation, and estimates vary; PI_HAT alone does not identify the pedigree. (4) The first estimate is below \(t\), and the second is above it, so the pair is borderline. Report the governing marker set and estimate, review quality and stability, and avoid treating the crossing as a certain relationship diagnosis.