Inbreeding Coefficients From Heterozygosity

From Heterozygosity to an Inbreeding Coefficient

Observed heterozygosity summarizes how often a sample carries two different alleles at a set of loci. To interpret that proportion, it helps to compare it with the heterozygosity expected from allele frequencies in a defined reference population. The heterozygosity-based inbreeding coefficient expresses the difference as a proportion of that expectation.

This coefficient is useful as a compact description of heterozygote deficit or excess relative to a chosen baseline. Its name can invite a stronger interpretation than the calculation supports: a positive value does not, on its own, establish that a particular sample is biologically inbred. The allele-frequency reference, the loci included, and the reliability of the genotype calls all matter.

For a biallelic locus \(j\), if the frequency of one allele in the reference is \(p_j\), the other allele has frequency \(1-p_j\), and expected heterozygosity under random pairing of alleles is \(H_{E,j}=2p_j(1-p_j)\). For a locus with more than two alleles, the corresponding expectation is \(1-\sum_k p_{jk}^2\), where \(p_{jk}\) is the reference frequency of allele \(k\). The observed value at a locus is one for a heterozygous genotype and zero for a homozygous genotype.

If sample \(i\) has called genotypes at a set \(C_i\) of \(m_i\) loci, then \(H_{O,i}\) is the average of those zero-or-one observed indicators. For a straightforward sample-level summary, \(H_{E,i}\) is the average of the expected heterozygosities at those same loci. Thus, the coefficient compares two averages over the same set of called loci. A locus with zero expected heterozygosity contributes no expected variation; if all included loci have zero expected heterozygosity, the coefficient is undefined.

Why the Formula Has This Interpretation

At a biallelic locus with allele frequencies \(p\) and \(q=1-p\), the expected heterozygote proportion under random pairing is \(2pq\). In a standard inbreeding model, an individual’s two alleles are identical by descent with probability \(F\); with probability \(1-F\), they are drawn as under random pairing. The model gives the genotype probabilities \(p^2+Fpq\) for the first homozygote, \(2pq(1-F)\) for the heterozygote, and \(q^2+Fpq\) for the second homozygote.

The heterozygote probability in that model is \(2pq(1-F)\). Since the random-pairing expectation is \(2pq\), dividing the model’s heterozygote probability by the expectation gives \(1-F\). Rearranging yields the coefficient’s defining relationship:

This derivation explains the traditional interpretation of \(F\) under the model. In routine genomic quality control, however, the calculation is often a descriptive sample-level summary across many loci, not a direct estimate of a known pedigree parameter. It behaves like the model’s \(F\) only when the assumptions and reference population are appropriate.

Interpreting the Sign and Magnitude

The sign is relative to the chosen expectation. A positive coefficient means a heterozygote deficit: \(H_O<H_E\). A negative coefficient means a heterozygote excess: \(H_O>H_E\). A coefficient of zero means the two summaries agree. For example, \(F=0.20\) means observed heterozygosity is 20% below its expected value, not that 20% of the sample’s genome is necessarily inbred. The coefficient is dimensionless.

Magnitude is also relative. It depends on the allele frequencies used to compute \(H_E\), the loci included, and the observed genotypes. The ratio does not carry a universal cutoff that turns a sample into “inbred” or “outbred.” In a finite set of loci, the observed proportion can vary substantially, especially when there are few informative loci. Linkage between loci also means the observations need not provide as much independent information as the number of loci suggests.

Assumptions, References, and Calculation Choices

The calculation requires a defined set of loci, valid genotype calls, and a reference allele frequency for each locus. The usual biallelic expectation \(2p_j(1-p_j)\) assumes diploid genotypes and uses the allele frequency in that reference. For the traditional inbreeding interpretation, the model additionally assumes that the reference is appropriate and that departures from random pairing can reasonably be described by the inbreeding model. These assumptions can fail even when the arithmetic is performed correctly.

A pooled reference can combine groups with different allele frequencies. The resulting expectation may then be a poor baseline for any one sample or subgroup. Likewise, allele-frequency estimates from a small or selectively chosen reference can be uncertain. The same genotype calls can therefore yield different coefficients under different reference frequencies; a change in \(F\) need not represent a change in the sample’s genotypes.

Missing calls should not be counted as homozygotes. In the basic calculation, each sample’s observed and expected summaries use loci called for that sample. If samples have different called loci, their summaries may not be directly comparable: they may be averaging over different sets of expected heterozygosities. Genotyping errors can also alter the observed count of heterozygotes, while poor locus selection can make the summary unrepresentative of the intended genomic region.

Implementations can differ in the exact aggregation or frequency-estimation choices. One common summary is the ratio of mean observed heterozygosity to mean expected heterozygosity across loci, as defined here. An average of locus-specific coefficients instead gives each locus a separate ratio before averaging; in general, that is not algebraically identical. When comparing reported values, check the loci used, how allele frequencies were obtained, how missingness was handled, and how the across-locus summary was formed.

How This Summary Differs From Other Checks

A heterozygosity-based coefficient compresses observed and expected heterozygosity across loci into one sample-level summary. A Hardy–Weinberg equilibrium test, by contrast, evaluates genotype-count departures at a locus under a specified model; it supplies a test statistic or p-value for that locus and does not have the same interpretation as a sample-level coefficient. A coefficient can be calculated descriptively without a hypothesis test, and its value alone does not establish statistical significance.

The coefficient is also not interchangeable with a pedigree-based estimate of inbreeding. The latter uses information about relationships and inheritance; the heterozygosity-based summary compares genotype calls with allele-frequency expectations. It is concise and easy to compute, but it is sensitive to the reference and cannot identify the cause of a deficit or excess. Later diagnostic work can assess how individual samples compare with one another; this coefficient by itself does not supply that diagnostic decision.

Common Pitfalls and Limitations

• Reversing the sign. Since \(F=1-H_O/H_E\), a positive value means less observed heterozygosity than expected, not more.
• Calling \(F\) a probability. The coefficient is a ratio-based summary, not the probability that a sample is inbred. It can be negative when there is heterozygote excess.
• Ignoring the reference. Expected heterozygosity depends on allele frequencies. An interpretation that omits the frequency baseline is incomplete.
• Treating a descriptive estimate as a cause. A deficit can arise from several sources, including population mixture, technical error, or departures from the model. The coefficient does not distinguish among them.
• Comparing unlike locus sets. Different missingness patterns or locus filters can change the average expectation and make sample summaries less comparable.
• Overinterpreting a small number of loci. With few loci, each genotype has substantial influence on \(H_O\), so the coefficient can be unstable.

A useful report states the coefficient together with the reference used to calculate expected heterozygosity and enough detail to identify the loci and missing-call handling. The number is best treated as a summary to interpret alongside the underlying data and other quality-control evidence, not as a stand-alone verdict.