Why Compare Samples Pairwise?
Genomic quality control is not only about checking each sample in isolation. Two samples can each have plausible missingness and heterozygosity yet be unexpectedly similar across the genome. Pairwise comparison helps identify samples that merit review, for example because a participant may be related to another participant despite the study’s intended sampling design. A similarity statistic is a screening signal: by itself, it does not establish why two samples are similar or dictate what action to take.
The key distinction is between identity by state (sharing the same observed allele or genotype) and identity by descent (sharing an allele inherited from a common ancestor). Genotype data directly show identity by state. A kinship estimate uses allele frequencies and sharing across many loci to estimate the extent of identity by descent under a model.
Exact genotype concordance is simple to calculate, but it is not a kinship estimate. For example, matching on a common genotype can occur frequently even among unrelated people. Conversely, a pair can share alleles by descent without having identical genotypes at every site. A frequency-adjusted estimate is therefore more informative for relatedness screening.
A Frequency-Adjusted Pairwise Estimate
Consider \(m\) autosomal variants observed in both samples. Let \(G_{i\ell}\) and \(G_{j\ell}\) be the alternate-allele dosages for samples \(i\) and \(j\) at variant \(\ell\), and let \(p_\ell\) be the alternate-allele frequency in the relevant population. The expected dosage is \(2p_\ell\). A method-of-moments estimate of kinship is
The numerator accumulates the samples’ centered genotype products. At a locus, it is positive when both samples’ dosages tend to be above their population expectation or both tend to be below it. The denominator scales the sum by the expected allele-count variation. Variants with \(p_\ell\) near 0 or 1 contribute less information because \(p_\ell(1-p_\ell)\) is small.
This estimate relies on a model: the samples are diploid at the included autosomal variants; allele frequencies represent a common, appropriately matched population; genotypes are accurately called; and the markers used to summarize the genome are not so redundant that a few linked regions dominate the result. In the idealized derivation, loci are treated as independent and the population is in Hardy–Weinberg equilibrium, with no unmodelled ancestry differences between the pair.
Why the Formula Estimates Kinship
At a locus with frequency \(p\), write each genotype dosage as the sum of two allele indicators. Under the population model, the covariance between one allele indicator from sample \(i\) and one from sample \(j\) is \(\phi_{ij}p(1-p)\): with probability \(\phi_{ij}\) the alleles are identical by descent, and otherwise their association is absent under the model. There are four cross-sample pairs of allele indicators in the product of two diploid dosages. Therefore,
Because \(E(G_i)=E(G_j)=2p\), the expected centered product is this covariance:
Summing across loci gives an expected numerator of \(\phi_{ij}\) times the denominator. Their ratio is consequently a method-of-moments estimate of kinship. In finite data, the ratio fluctuates around its target; it is not guaranteed to lie between 0 and 0.5. Estimating \(p_\ell\) rather than knowing it also adds uncertainty and can introduce bias if the frequency reference is poorly matched.
Worked Example: Calculating Kinship Across Four Variants
Suppose two hypothetical samples have genotype dosages \(G_i=(0,1,2,1)\) and \(G_j=(0,1,1,2)\) at four jointly observed autosomal variants. For this arithmetic illustration, each variant has \(p_\ell=0.5\). Assume the calls are accurate and the frequencies are appropriate for both samples.
The expected dosage at each site is \(2p_\ell=1\). The centered genotype vectors are therefore \((-1,0,1,0)\) and \((-1,0,0,1)\). Their locus-by-locus products are \(1,0,0,0\), so the numerator is \(1\). The denominator is \(4\) times the sum of four values of \(0.5(1-0.5)=0.25\):
The pair has exact genotype matches at the first two of four variants, so its genotype concordance is \(2/4=0.50\). That is a different quantity from the kinship estimate. Because this example uses only four variants, the value \(0.25\) is highly unstable and should not be treated as a reliable classification; real screening uses many suitable markers.
Interpreting Pairwise Values for Quality Control
The kinship coefficient is on a probability scale. Under the standard pedigree model, unrelated individuals from the same population have expected kinship near 0. Parent–offspring pairs and full siblings have expected kinship \(0.25\); more distant relatives have smaller expected values. These are population expectations, not hard boundaries for observed estimates. Sampling variation, allele-frequency error, ancestry differences, and genotype errors can move an estimate away from its expectation.
For quality control, compare a pair’s estimate with what the study design predicts. A large positive estimate in a study intended to contain unrelated participants is a reason to inspect both samples and the underlying data. Review sample identifiers, genotype call quality, missingness, ancestry composition, and whether the estimate is supported across the genome rather than driven by a small region. Record the result as a flag for review, not as proof of a particular pedigree relationship.
Worked Example: Why Frequency Adjustment Matters
Consider two hypothetical samples at one common variant with \(p=0.5\). Both have genotype dosage 1. They match exactly, so their genotype concordance at this site is 1. Their centered dosages are both zero, however, so this site contributes zero to the kinship numerator. Under this frequency model, the shared heterozygous genotype alone gives no positive centered covariance.
Now consider a hypothetical variant with \(p=0.1\), at which both samples have dosage 0. Their exact genotypes again match. The centered dosages are each \(0-2(0.1)=-0.2\), so the numerator contribution is \((-0.2)(-0.2)=0.04\). The denominator contribution is \(4(0.1)(0.9)=0.36\). The one-site ratio would be \(0.04/0.36\), about \(0.111\), but a one-site ratio is not a dependable genome-wide estimate. The calculation illustrates why matching genotypes do not all carry the same information: the interpretation depends on how common the genotype is under the reference frequency.
For a genome-wide estimate, contributions are summed before taking the ratio. It is not appropriate to interpret an isolated variant’s ratio as a pair’s kinship.
Worked Example: A Pairwise Flag in a Small Hypothetical Panel
Suppose three hypothetical samples, A, B, and C, are typed at eight independent autosomal variants, each with \(p_\ell=0.5\). Their dosages are:
For A and B, the centered dosage products sum to 2: the first two sites contribute \(1+1\), and the other six contribute 0. The denominator is \(4(8)(0.25)=8\), giving \(\hat{\phi}_{AB}=2/8=0.25\). For A and C, the first two products are \(1\) and \(-1\), which cancel; the remaining products are zero, giving \(\hat{\phi}_{AC}=0/8=0\). The same calculation gives \(\hat{\phi}_{BC}=0/8=0\).
In this deliberately small illustration, A–B is the pair that stands out for follow-up relative to the other pairs. The result does not establish that A and B have a particular relationship: eight variants are far too few for a dependable genome-wide assessment. In practice, the pairwise pattern would be checked using a large, carefully selected marker set and alongside other sample-level evidence.
Diagnostics, Alternatives, and Limitations
Several checks help determine whether pairwise estimates are trustworthy. First, use a consistent set of well-called, jointly observed autosomal variants and report how many contribute to each pair. Pair-specific missingness can make estimates incomparable or unstable. Second, inspect whether high estimates persist across chromosomes or genomic regions; a signal concentrated in one region may reflect local linkage or a technical artifact rather than genome-wide similarity. Third, compare allele-frequency and ancestry patterns. If two samples come from different ancestry groups, centering both using a single pooled frequency can create spurious covariance. Population structure can therefore inflate or distort estimates even when the samples are not close relatives.
Linkage disequilibrium also matters. Correlated markers do not provide as much independent information as the same number of independent markers. A dense cluster of linked variants can dominate a simple sum, so screening generally uses a large set of markers chosen to limit redundancy. Genotyping errors, allele coding inconsistencies, sample swaps, and poorly handled missing calls can also change the apparent sharing. These problems should be considered before interpreting an outlying value.
The main simple alternative is exact genotype concordance, which answers “what fraction of jointly observed sites have identical genotype calls?” It is intuitive and useful as a descriptive check, but it ignores allele frequencies and is not a kinship coefficient. Frequency-adjusted kinship is more interpretable for relationship screening under its model, but it can be sensitive to frequency mismatch and structure. More specialized estimators are designed to be more robust to some forms of population structure or to use different genotype-sharing patterns. They still require suitable data and careful interpretation; no single statistic removes the need for diagnostics.
A negative \(\hat{\phi}_{ij}\) can arise from sampling variation around zero and is not evidence of a biologically meaningful negative relationship. A value outside the usual range for kinship can likewise be a warning of small marker count, frequency mismatch, or other model failure rather than a valid pedigree estimate. Estimates also do not identify which member of a pair, if either, should be removed. That decision depends on study design and additional information.
Common Pitfall
Wrong reasoning: “These samples match at many genotypes, so they must be close relatives.” The conclusion does not follow. Common genotypes can match by chance, and concordance does not adjust for allele frequency or ancestry. Correction: treat raw concordance as a descriptive sharing measure, use a frequency-adjusted genome-wide kinship estimate for relatedness screening, and review the marker set, ancestry, missingness, and call quality before drawing conclusions.
Check Your Understanding
Answer each question, then check the solutions below.
- Two samples have dosages 0 and 2 at a variant with \(p=0.5\). What is the centered-product contribution to the kinship numerator?
- Why can exact genotype concordance be high for unrelated samples?
- For three variants with \(p_\ell=0.5\) at every site, samples have dosages \(G_i=(0,1,2)\) and \(G_j=(0,1,0)\). Calculate their kinship estimate using the stated formula.
- Name two conditions or data problems that can distort a pairwise kinship estimate.
Solutions. (1) The expected dosage is \(1\), so the centered dosages are \(-1\) and \(1\); their product is \(-1\). (2) Common genotypes can match frequently by chance, and concordance does not account for allele frequencies or ancestry. (3) The centered vectors are \((-1,0,1)\) and \((-1,0,-1)\). The numerator is \(1+0-1=0\), and the denominator is \(4(3)(0.25)=3\), so \(\hat{\phi}_{ij}=0/3=0\). (4) Examples include ancestry mismatch between the samples and allele-frequency reference, genotype errors, pair-specific missingness, and excessive linkage among included markers.