Deriving Genotype Frequencies Under Inbreeding

Why Inbreeding Changes Genotype Proportions

At a biallelic locus, Hardy–Weinberg genotype proportions follow from drawing two alleles independently from a population with fixed allele frequencies. Inbreeding changes the chance that an individual’s two alleles are identical by descent: they are copies of the same ancestral allele. Such a pair must carry the same allele at the locus, so identity by descent increases homozygosity and reduces heterozygosity.

A useful way to derive the resulting genotype proportions is to divide allele pairs into two cases: pairs that are identical by descent, and pairs that are not. The first case contributes only homozygotes. In the second, the alleles are drawn independently, so the Hardy–Weinberg probabilities apply conditionally. Mixing these cases gives the inbreeding-adjusted proportions.

These equations assume that an allele copied through identity by descent has the population allele frequencies \(p\) and \(q\), and that when the two alleles are not identical by descent, they are independent draws with those same frequencies. The model describes genotype proportions at a locus; it does not, by itself, specify how mating or ancestry produced a particular value of \(F\).

Deriving the Mixture Formula

Let \(I\) denote the event that an individual’s two alleles are identical by descent. By definition, \(P(I)=F\), and \(P(I^c)=1-F\). Conditional on \(I\), both alleles are copies of one ancestral allele. That allele is \(A_1\) with probability \(p\) and \(A_2\) with probability \(q\). Therefore,

When the alleles are not identical by descent, the model treats them as independent draws from the population gene pool. Thus the conditional probabilities are the Hardy–Weinberg proportions:

For each genotype, the law of total probability weights its probability in each case by the probability of that case. For example, the \(A_1A_1\) frequency is \(F\) times its conditional frequency when the alleles are identical by descent, plus \(1-F\) times its conditional frequency when they are not:

The last equality uses \(p-p^2=p(1-p)=pq\). The same calculation for the heterozygote and the other homozygote gives

The terms added to the two Hardy–Weinberg homozygote frequencies are equal, \(Fpq\), while the heterozygote frequency is multiplied by \(1-F\). This balance is important: the model changes the genotype proportions without changing the allele frequencies.

Checking the Frequencies and Their Allele Marginals

The three genotype frequencies must sum to one. Substituting the formulas gives

The allele frequency can also be recovered from the genotype proportions. Each \(A_1A_1\) individual contributes two \(A_1\) alleles, each \(A_1A_2\) individual contributes one, and each \(A_2A_2\) individual contributes none. Therefore, the marginal frequency of \(A_1\) is

The corresponding marginal for \(A_2\) is \(q\). Thus, the inbreeding adjustment reallocates probability among genotypes while preserving the allele frequencies assumed by the model. This is a useful check on both a derivation and a numerical calculation.

Interpreting and Using the Inbreeding Coefficient

The heterozygote equation can be rearranged to show the model’s defining relationship to Hardy–Weinberg heterozygosity. If \(p\) and \(q\) are known and \(2pq>0\), then

Thus, \(F\) measures the proportional reduction in heterozygosity from the Hardy–Weinberg value \(2pq\), under this mixture model. It is not the proportion of individuals who are homozygous. Homozygotes can occur even when \(F=0\), because two independently drawn alleles can match by chance. Conversely, the quantity \(F\) describes the probability of identity by descent, not the probability that the genotype is homozygous.

The same mixture logic extends to a locus with several alleles. If allele \(i\) has frequency \(p_i\), the homozygote frequency is \(p_i^2+Fp_i(1-p_i)\). For two distinct alleles \(i\) and \(j\), the unordered heterozygote frequency is \(2(1-F)p_ip_j\). Conditional on identity by descent, the two alleles must match; conditional on no identity by descent, independent draws produce the usual homozygote and heterozygote probabilities. The biallelic formulas are the special case with allele frequencies \(p\) and \(q\).

Boundary Cases and Common Pitfalls

At \(F=0\), the formulas reduce to the Hardy–Weinberg proportions \(p^2\), \(2pq\), and \(q^2\). At \(F=1\), all pairs are identical by descent in this model, so the heterozygote frequency is zero and the two homozygote frequencies are \(p\) and \(q\). These cases help check the direction and scale of the adjustment.

• Wrong reasoning: “If \(F=0.20\), then 20% of individuals must be homozygous.” Correction: \(F\) is the probability of identity by descent, not the total homozygote proportion. Homozygosity also arises when independent allele draws happen to match.
• Wrong reasoning: “Inbreeding increases the \(A_1A_1\) frequency by \(F\).” Correction: The increase is \(Fpq\), not \(F\); it depends on both allele frequencies. The same amount \(Fpq\) is added to the other homozygote, while \(2pqF\) is removed from heterozygotes.
• Wrong reasoning: “A higher \(F\) changes the allele frequency \(p\).” Correction: In this model, the genotype proportions change but their allele marginal remains \(p\). A change in allele frequency requires a change to the specified population allele frequency, not merely this genotype adjustment.
• Wrong reasoning: “Every homozygote must be identical by descent.” Correction: Two alleles can be identical in state simply because independent draws selected the same allele. Identity by descent is a statement about shared ancestry, not just matching allele labels.