Thresholding Estimates To Detect Cryptic Relatives

Why Thresholds Need Careful Interpretation

Pairwise relatedness estimates are often used to flag samples that may represent duplicates or close relatives. A cutoff makes that screening task manageable: pairs above a chosen value are flagged for attention. But a cutoff is a decision rule applied to an estimate, not a biological boundary. Two pairs with nearly identical estimates can land on opposite sides of a cutoff, and the estimate itself is uncertain.

The scale also matters. A kinship coefficient and PI_HAT summarize relatedness differently, so a numerical threshold for one must not be applied to the other. Even on the correct scale, relationship categories are not identified with certainty by one estimate. The goal is to make a useful, transparent screening decision and to convey what that decision does—and does not—establish.

Choose the Estimator’s Scale First

The kinship coefficient \(\phi_{ij}\) describes the probability that one allele sampled from individual \(i\) and one from individual \(j\), at the same locus, are identical by descent under the coefficient’s pedigree definition. The estimated pairwise kinship coefficient is written \(\hat{\phi}_{ij}\). PI_HAT is a different summary: it estimates the proportion of the genome at which a pair shares at least one allele identical by descent. Under common idealized conditions, PI_HAT is approximately twice the kinship coefficient, but the two statistics are not interchangeable outputs.

For orientation, the conventional pedigree expectations for outbred individuals are approximately \(0.5\) for duplicates (or monozygotic twins), \(0.25\) for first-degree relatives such as parent–offspring or full siblings, \(0.125\) for second-degree relatives, and \(0.0625\) for third-degree relatives on the kinship-coefficient scale. On the PI_HAT scale, the corresponding idealized expectations are approximately \(1\), \(0.5\), \(0.25\), and \(0.125\). These are expectations under specified pedigree relationships, not fixed values that every observed pair must attain.

A frequently used set of screening boundaries for the KING-robust kinship estimate is \(0.354\), \(0.177\), \(0.0884\), and \(0.0442\). In this convention, values above \(0.354\) are duplicate or very-close-relative candidates; \(0.177\) to \(0.354\) indicates a first-degree range; \(0.0884\) to \(0.177\) a second-degree range; and \(0.0442\) to \(0.0884\) a third-degree range. Values below \(0.0442\) fall below this screening convention’s third-degree boundary. These are useful heuristics for that estimator, not universal cutoffs for every kinship method or population.

The table intentionally puts expectations beside screening ranges: they answer different questions. An expectation describes the average under a pedigree model, while a cutoff partitions estimates for practical screening. In particular, a value below a cutoff does not prove that a pair is unrelated, and a value above it does not prove one specific family relationship.

Uncertainty Near a Boundary

Pairwise estimates vary because the observed markers are a finite, imperfect sample of genomic sharing. Linkage disequilibrium reduces the amount of independent information in a marker set; missing genotypes and allele-frequency inputs can also affect an estimate. Earlier material on uncertainty in pairwise estimates and effective marker counts provides the background for interpreting this variability. Here the key consequence is practical: when an estimate is near a cutoff, its uncertainty may make a categorical label unstable.

If an estimate has a standard error \(SE\), and its sampling distribution is reasonably approximated by a normal distribution, an approximate 95% interval is the estimate plus or minus \(1.96SE\). This interval is a way to communicate uncertainty under the approximation; it is not a probability statement about the fixed relationship category.

If this interval crosses a category boundary, the data do not sharply separate the categories at that boundary under the stated approximation. The point estimate still has a value, and a prespecified rule can still flag the pair, but the classification should be reported as uncertain rather than treated as a definitive relationship assignment.

Set the Rule to Match the Screening Purpose

A cutoff is useful only in relation to a purpose. A broad screen for possible close relatives may use a lower boundary so that more candidates are reviewed, accepting that some flagged pairs will not be close relatives. A screen intended to identify only the most extreme sharing may use a higher boundary, but it will miss some genuine close relatives. This is a trade-off between sensitivity to candidates and the workload or consequences of false flags.

A defensible workflow states the estimator, scale, threshold, and intended action before interpreting results. If the data have a relevant reference set or validation information, the cutoff can be assessed against those data. If not, a conventional boundary should be described as a screening heuristic. The effect of ancestry composition, inbreeding, marker properties, and estimator assumptions should be considered, since these can alter the behavior of pairwise estimates. A single cutoff should not be assumed to perform identically in every dataset.

Thresholding also discards information. A pair estimated at \(0.178\) and one estimated at \(0.330\) both lie in the same first-degree range under the KING convention, but they are not equally close to the category’s boundaries or to one another. Retaining the continuous estimates and, when available, their standard errors makes later interpretation more informative than keeping only labels.

Pedigree Expectations Are Not Observed Values

Pedigree coefficients provide benchmarks, not targets that each pair must match exactly. Even relatives with the same pedigree relationship can differ in realized genomic sharing because inheritance is random. Marker sampling and estimation add further variability. Inbreeding can change pairwise sharing patterns, and population structure or allele-frequency misspecification can affect some estimators. The earlier material on inbreeding effects and population structure is relevant to these limitations.

Consequently, comparing an estimate with an ideal expectation is not a substitute for applying a clearly specified screening rule. For instance, a first-degree pedigree expectation of \(0.25\) does not mean that an estimate of \(0.20\) rules out first-degree relatedness. Nor does an estimate near \(0.25\) establish that a pair is first-degree: other relationships or estimation effects may produce similar values. Report the estimate and its scale, describe the rule used, and make conclusions no stronger than the evidence permits.

Common Mistakes

• Using a familiar number without checking the estimator. A cutoff developed for KING-robust kinship is not automatically valid for PI_HAT or another estimator. State which statistic is being thresholded.
• Treating the point estimate as exact. An estimate just above a boundary is not biologically different in a sharp way from one just below it. Check uncertainty and report borderline classifications cautiously.
• Calling a screening category a confirmed pedigree. A threshold can flag candidates or assign a provisional range; it does not prove parentage, sibling status, or duplication.
• Assuming pedigree expectations are observed values. Expectations describe averages under assumptions. Realized sharing and estimated values vary among pairs with the same pedigree relationship.
• Reporting only the category. Preserve the numerical estimate, estimator, scale, cutoff, and relevant uncertainty so that the reader can see how the decision was made.