Genotype Coding And Allele-Frequency Inputs

Why Genotype Coding and Frequency Choice Matter

A pairwise relatedness calculation begins with genotypes, but a genotype is not yet a numerical input until its alleles have been coded. At each biallelic marker, the usual starting point is a count of copies of one named allele. That count can then be centered relative to an allele frequency and, in many procedures, scaled by the expected genotype variation. Both choices matter: inconsistent allele coding can reverse a marker’s contribution, while a different frequency can change how strongly that marker is weighted.

This tutorial concerns those inputs, not a complete relatedness estimator. It uses diploid autosomal genotypes at biallelic markers and explains how to define the count and frequency before applying a pairwise method. Any result formed from these inputs still depends on the method’s later choices about combining markers and handling data.

Definition: At a biallelic marker, choose one allele as the effect allele and code an individual’s genotype by $g$, the number of copies of that allele. For a diploid individual, $g\in\{0,1,2\}$. Let $p$ be the frequency of that same effect allele in a specified population or reference set. The centered genotype is $g-2p$; a commonly used standardized genotype is $$ z=\frac{g-2p}{\sqrt{2p(1-p)}}, $$ when $0<p<1$.

The words “that same effect allele” are essential. If $g$ counts the alternate allele, then $p$ must be the frequency of the alternate allele. Using the frequency of the other allele with that count does not center the genotype correctly. The effect allele is simply the allele being counted; it need not have a biological effect.

From Allele Count to Centered and Standardized Genotype

Under Hardy–Weinberg equilibrium, the two allele copies are modeled as independent draws, each carrying the effect allele with probability $p$. Thus the genotype count has a binomial distribution, $G\sim\operatorname{Binomial}(2,p)$. Its expected value and variance are

E(G)=2p,\qquad \operatorname{Var}(G)=2p(1-p).

Centering subtracts the expected count $2p$, so $G-2p$ has mean zero under this model. Dividing by the model-based standard deviation $\sqrt{2p(1-p)}$ gives a quantity with variance one. This explains the standardization formula; it does not make every observed genotype equal to a standard normal value. The variable $z$ takes only three possible values for hard-call genotypes at a locus.

Hard calls use $g=0,1,2$. If genotype probabilities or dosage data are provided instead, an expected allele count can be fractional, such as $g=0.7$. The same centering formula can be applied to that dosage, provided the dosage and $p$ refer to the same allele and allele-frequency definition. The binomial variance formula describes the underlying genotype model, not necessarily every source of uncertainty in an estimated dosage.

Worked Example: Standardizing Three Genotype Counts

Suppose a hypothetical biallelic marker has effect-allele frequency $p=0.30$. For an individual with genotype count $g$, calculate the centered and standardized values. Assume diploidy and use the Hardy–Weinberg variance scale.

The expected count is $2p=2(0.30)=0.600$. The standard deviation used in the denominator is $$ \sqrt{2p(1-p)}=\sqrt{2(0.30)(0.70)}=\sqrt{0.420}=0.648074\ldots $$

For $g=0$, the centered count is $0-0.600=-0.600$, giving $z=-0.600/0.648074\ldots\approx-0.9258$. For $g=1$, it is $1-0.600=0.400$, giving $z=0.400/0.648074\ldots\approx0.6172$. For $g=2$, it is $2-0.600=1.400$, giving $z=1.400/0.648074\ldots\approx2.160$.

These values describe each genotype relative to the assumed frequency and variance at this marker. A count of one is not “half a standard deviation” by definition; its standardized value depends on $p$.

Estimating and Specifying the Allele Frequency

With complete hard-call genotypes from $n$ diploid individuals, the sample frequency of the counted allele is the total number of its copies divided by the total number of allele copies:

\hat{p}=\frac{\sum_{i=1}^{n} g_i}{2n}.

This is a count-based frequency estimate. The denominator is $2n$ because each diploid individual contributes two allele copies. If genotypes are missing, the denominator must instead count the allele copies actually observed; the complete-data expression above should not be used unchanged. Which individuals supply the frequency is also part of the analysis specification: it could be the individuals being compared, a larger study cohort, or an external reference set.

Worked Example: Calculating a Cohort Frequency

In a hypothetical cohort of six diploid individuals, the effect-allele genotype counts are $0,1,2,1,0,1$. Calculate the sample frequency and one centered genotype.

The total number of effect-allele copies is $0+1+2+1+0+1=5$, and there are $2(6)=12$ allele copies in total. Therefore, $$ \hat{p}=\frac{5}{12}=0.4167. $$

For the individual with $g=2$, the expected count based on this frequency is $2\hat{p}=0.8333$, so the centered count is $2-0.8333=1.1667$. Its standardized value, if the Hardy–Weinberg scale is used, is $$ z=\frac{2-2(5/12)}{\sqrt{2(5/12)(7/12)}}= \frac{7/6}{\sqrt{35/72}}\approx1.673. $$

The calculation identifies the frequency as a cohort estimate, not a universal property of the marker. Another population or reference panel can have a different frequency and therefore a different expected count and scale.

Allele Orientation Must Be Consistent

A marker may be coded using either of its two alleles. Reversing the coding is valid only if both the genotype count and its frequency are reversed together. If the original count is $g$ and its frequency is $p$, the count of the other allele is $g^*=2-g$, and its frequency is $p^*=1-p$. Then

$$ g^*-2p^*=(2-g)-2(1-p)=-(g-2p). $$

The standardizing denominator is unchanged because $2p^*(1-p^*)=2(1-p)p=2p(1-p)$. Therefore $z^*=-z$. If every individual at the marker is recoded consistently, a product of two standardized genotypes is unchanged:

$$ z_i^*z_j^*=(-z_i)(-z_j)=z_i z_j. $$

This sign reversal is why the choice of counted allele is arbitrary when coding is internally consistent. It is also why inconsistent coding is consequential. If one individual’s allele orientation is reversed but the other’s is not, the product changes sign. In a multi-marker calculation, such errors can make contributions cancel or reinforce one another for reasons unrelated to the individuals’ genetic similarity.

Worked Example: Checking a Reversed Allele Code

At a hypothetical marker, a pair has counts $g_i=2$ and $g_j=1$ for an allele with $p=0.30$. Check the standardized values after reversing the counted allele.

With the original orientation, the denominator is $\sqrt{0.420}=0.648074\ldots$. The values are $$ z_i=\frac{2-0.600}{0.648074\ldots}\approx2.160,\qquad z_j=\frac{1-0.600}{0.648074\ldots}\approx0.6172. $$ Their product is approximately $2.1602469(0.6172134)=1.3333$.

For the reversed orientation, $g_i^*=0$, $g_j^*=1$, and $p^*=0.70$. The denominator remains $\sqrt{2(0.70)(0.30)}=\sqrt{0.420}$. The standardized values are approximately $-2.160$ and $-0.6172$; their product is again approximately $1.3333$. If only one person’s coding had been reversed, the product would instead be approximately $-1.3333$, showing why allele labels must be harmonized across individuals.

In practice, allele alignment requires matching the same biological allele across genotype files and reference-frequency files, not merely matching a column called “alternate.” Strand complements can change the written allele labels. For strand-ambiguous A/T or C/G variants, allele labels alone may not determine whether two files use the same orientation, so a reliable harmonization convention or additional information is needed. Using “minor allele” as the counted allele is not a substitute for alignment: the minor allele can differ between populations or frequency panels.

Sample-Specific and Reference Frequencies

Using a frequency estimated from the analyzed sample makes the centering and scaling reflect that sample’s allele distribution. With complete data, the centered genotypes sum to zero at each locus because

\sum_{i=1}^{n}(g_i-2\hat{p}) =\sum_{i=1}^{n}g_i-2n\left(\frac{\sum_{i=1}^{n}g_i}{2n}\right)=0.

This is an exact sample property when $\hat{p}$ is calculated from those same complete genotypes. It does not show that the cohort represents a single homogeneous population, nor does it imply that every subgroup is centered at zero. Because the frequency is estimated using the analyzed individuals, pairwise inputs can also depend on which other individuals were included in the frequency calculation.

A reference frequency instead fixes the centering and scaling to a stated external set. That can support comparisons across cohorts when the same allele orientation and reference are used. However, if the reference population differs in allele frequency from the analyzed group, the resulting centered values may have nonzero means in that group, and the standardization may not match its genotype variance. Neither choice is universally correct: the intended population interpretation and the estimator’s definition determine the appropriate input. Frequency estimates also have uncertainty, particularly in small reference panels, even if an analysis treats the supplied frequency as fixed.

Worked Example: How the Frequency Changes a Pairwise Input

Two hypothetical individuals have counts $g_i=2$ and $g_j=1$ at one marker. Compare their product of standardized genotypes using a cohort frequency $p=0.30$ and a reference frequency $p=0.20$. This is a one-marker illustration, not a complete relatedness estimate.

With $p=0.30$, the denominator is $\sqrt{0.420}=0.648074\ldots$. The standardized values are approximately $2.1602$ and $0.6172$, so their product is $$ \frac{(2-0.60)(1-0.60)}{0.420} =\frac{0.56}{0.420}=1.3333. $$

With $p=0.20$, the denominator is $\sqrt{2(0.20)(0.80)}=\sqrt{0.32}=0.565685\ldots$. The standardized values are approximately $2.8284$ and $1.0607$, giving $$ \frac{(2-0.40)(1-0.40)}{0.32} =\frac{0.96}{0.32}=3.000. $$

The genotypes did not change, but the marker’s centered values and product did. The reference frequency assigns a different baseline and scale. A downstream method may aggregate such inputs over many markers, so frequency choice can affect its result; this example alone does not establish the direction or size of a whole-genome relatedness change.

Practical Checks and Common Pitfalls

Before calculating pairwise quantities, write down the allele-count and frequency conventions for every marker set. At minimum, check that the counted allele is identified consistently, the frequency refers to that allele and a specified population, the genotype scale is appropriate for diploidy, and the frequency is strictly between zero and one wherever the standardized formula is applied. A frequency of zero or one makes the denominator zero; such a monomorphic marker cannot be standardized by this formula and must be handled according to the chosen method.

Wrong: “I used the alternate-allele count, so the reference-allele frequency is fine.” Correction: Pair the count $g$ with the frequency of the allele counted by $g$; otherwise $g-2p$ is not centered for that coding.
Wrong: “The minor allele is always the same allele in every dataset.” Correction: Minor-allele status depends on the population and sample. Align by allele identity, and reverse both count and frequency when changing orientation.
Wrong: “Any frequency choice gives the same standardized genotype.” Correction: As the numerical comparison showed, changing $p$ changes both the centered count and its scale.
Wrong: “A cohort-centered value proves the cohort has no population structure.” Correction: A pooled frequency forces the pooled centered sum to zero, but subgroup means can differ and structure can remain.

Key takeaway: A genotype input is defined jointly by its allele-count orientation and its allele frequency. Consistent reversal changes the sign of each standardized genotype but preserves pair products; inconsistent reversal does not. State whether frequencies are sample-specific or reference-based, because that choice changes the centering, scaling, and interpretation of pairwise inputs.

Check Your Understanding

Use the definitions and calculations in this tutorial to answer the following.

For a diploid marker with effect-allele frequency $p=0.25$, what are the expected genotype count and the standard deviation used for standardization?
A complete sample of five individuals has genotype counts $0,1,1,2,0$. Calculate the sample frequency of the counted allele.
If a genotype count changes from $g$ to $2-g$, what corresponding change must be made to its allele frequency, and how does the standardized genotype change?
Why can a reference frequency make a cohort’s centered genotypes have a nonzero mean?
Explain why using a minor-allele count in one file and a differently oriented count in another can distort pairwise marker contributions.