Inbreeding Effects On Pairwise Estimation

Why Inbreeding Matters for Pairwise Estimation

A person can carry more homozygous genotypes than expected under random mating without being closely related to another sampled person. This distinction matters in relatedness analysis: inbreeding describes identity by descent between the two alleles within one individual, whereas pairwise kinship concerns identity by descent between alleles sampled from two individuals. Conflating them can make a diagonal relationship value look like evidence about a pair, or can make a change in homozygosity look like excess pairwise sharing.

Inbreeding can nevertheless affect pairwise estimation. It changes genotype distributions and can increase the sampling variability of some pairwise statistics. It can also challenge methods whose calibration relies on Hardy–Weinberg genotype proportions. The direction and size of any effect depend on the estimator; there is no single correction that applies to every pairwise method.

Inbreeding and Genotype Probabilities

Suppose the reference population has allele \(A\) at frequency \(p\), and allele \(a\) at frequency \(q=1-p\). If an individual has inbreeding coefficient \(F\), the standard genotype probabilities are

These probabilities add to one. Relative to Hardy–Weinberg proportions, inbreeding shifts probability from heterozygotes to homozygotes: heterozygosity is \(2pq(1-F)\), rather than \(2pq\). The allele frequency is unchanged. For example, the expected number of \(A\) alleles per individual remains \(2p\), because the allele-count mean is determined by the marginal allele frequencies, not by whether the two alleles within a person are identical by descent.

From Within-Individual Inbreeding to Pairwise Kinship

For a diploid genotype, let \(G_i\) be the number of \(A\) alleles in individual \(i\), so \(G_i\) is 0, 1, or 2. Under the genotype probabilities above, the mean and variance are

The mean stays at \(2p\), while the variance increases with inbreeding. This is one reason that inbreeding can matter to pairwise statistics even when it does not change their expected value under independence.

To relate dosage covariance to pairwise kinship, assume both individuals draw alleles from the same reference population with frequency \(p\), and that the kinship coefficient \(\phi_{ij}\) is defined by identity by descent at the locus. For any one of the four cross-individual allele pairs, the two alleles are identical by descent with probability \(\phi_{ij}\). If they are identical by descent, their probability of both being \(A\) is \(p\); otherwise, under the reference model, it is \(p^2\). The covariance of those two allele indicators is therefore \(\phi_{ij}(p-p^2)=\phi_{ij}pq\). Summing the four cross-pairs gives the genotype-count covariance.

Consequently, a standardized product of allele-count deviations, \((G_i-2p)(G_j-2p)/(2pq)\), has expectation \(2\phi_{ij}\) under these assumptions. This gives the familiar scaling of a genomic relationship matrix: its off-diagonal expectation is twice the pairwise kinship coefficient. On the diagonal, \(i=j\), the corresponding kinship is \(\phi_{ii}=(1+F_i)/2\), so the expectation is \(1+F_i\). The diagonal therefore reflects the individual’s inbreeding as well as the baseline value of one; it is not an off-diagonal estimate of that person’s kinship with someone else.

Why Estimator Behavior Depends on the Method

The covariance calculation relies on a model: a shared reference allele frequency, appropriate allele-count centering, and the stated identity-by-descent interpretation. Under that model, individual inbreeding changes the marginal genotype variance but does not force a nonzero off-diagonal covariance for unrelated individuals. Across many loci, averaging can stabilize an estimate, but the amount of stabilization depends on marker information and on the estimator’s assumptions.

Other pairwise methods use genotype categories or homozygote and heterozygote counts rather than just a centered dosage product. Their expected values need not be protected by the same covariance argument. Inbreeding changes the frequencies of those genotype categories, so a method calibrated under Hardy–Weinberg proportions may be shifted or have altered uncertainty when those proportions do not hold. Even methods designed to be robust to population structure have particular assumptions and should not be presumed automatically immune to individual inbreeding.

A useful diagnostic is to separate three questions: is the statistic’s expected value altered under the method’s model; is its variance altered; and are the allele frequencies or other inputs estimated appropriately? These are distinct issues. A result can be unbiased in expectation but noisier, or can have a systematic shift because its assumptions or inputs fail. The tutorial on uncertainty in pairwise estimates addresses sampling variability more generally; here the key point is that inbreeding can contribute to that variability through altered genotype distributions.

A Short Derivation of the Variance Increase

The variance formula makes the effect of inbreeding explicit. Let the genotype count be \(G\). Its mean is \(E(G)=P(Aa)+2P(AA)\), so substituting the inbreeding genotype probabilities gives

Because \(G\) takes values 0, 1, and 2, its second moment is \(E(G^2)=P(Aa)+4P(AA)\). Substitution gives

Subtracting the squared mean,

Thus the mean allele count is unchanged, but the variance is multiplied by \(1+F\) relative to the outbred value \(2pq\). This is a marginal, single-individual result. It does not say that the covariance between two unrelated individuals is positive.

Practical Interpretation and Common Mistake

When examining relatedness estimates, keep within-person and between-person quantities separate. An elevated relationship-matrix diagonal can be consistent with inbreeding; an off-diagonal value concerns a pair. Similarly, a high homozygosity rate in one sample is not itself proof of pairwise relatedness. Pairwise estimates should be interpreted in light of their allele-frequency inputs, genotype model, and treatment of population structure, as well as their uncertainty.

• Wrong reasoning: “This person is more homozygous than expected, so their off-diagonal kinship estimates with everyone must be high.”
• Correction: Inbreeding changes that person’s within-individual genotype distribution. It can alter the behavior or precision of some pairwise estimators, but it does not mechanically imply kinship to every other individual.
• Wrong reasoning: “The off-diagonal standardized dosage product must be biased upward whenever either person is inbred.”
• Correction: Under the shared-frequency covariance model, unrelated independent individuals still have expected off-diagonal product zero. Inbreeding can increase variance; bias requires considering the estimator and whether its assumptions hold.
• Wrong reasoning: “All pairwise estimators respond in the same direction to inbreeding.”
• Correction: Estimators use different genotype summaries and assumptions. Inbreeding can affect their expected values, variability, or both, and the result is method-specific.