Choosing a Coding From Genotype Patterns

Reading a Genotype-Mean Table

A genotype-mean table puts three trait means side by side: one for each genotype at a biallelic locus. Its pattern can suggest a useful coding for the genetic effect. The key is to compare the means, not simply to choose a familiar coding or to focus on which genotype has the largest mean.

Use \(aa\), \(Aa\), and \(AA\) for the three genotypes, and let \(m_0\), \(m_1\), and \(m_2\) denote their respective mean trait values. The dosage of allele \(A\) is 0 for \(aa\), 1 for \(Aa\), and 2 for \(AA\). Throughout, the trait is quantitative and the genotype-specific means are measured on the same scale and in the same units.

These labels describe restrictions on genotype means. They do not, by themselves, establish a biological mechanism or show that a genotype causes a trait difference. In real data, the means are usually estimated from individuals; a close visual pattern is not proof that the corresponding population means are exactly equal.

The Mean Constraints Behind Each Coding

The usual coding schemes make their assumptions visible as equations. Under additive coding, the mean changes by the same amount for each additional copy of \(A\). Under dominant or recessive coding, two genotypes are assigned the same coded value, so the model requires their means to be equal. Unrestricted coding makes no equality constraint.

Here \(\mu\) is the mean for the reference genotype in the stated coding. In the additive model, \(\beta\) is the change in mean per additional \(A\) allele. In the dominant and recessive models, \(\delta\) is the difference between the two coded groups. In the unrestricted model, \(\delta_1\) and \(\delta_2\) are separate differences from the \(aa\) mean. Their units are the same as the trait.

The patterns can also be checked directly using differences. Define the first step as \(m_1-m_0\) and the second as \(m_2-m_1\). Additivity requires the steps to be equal. Dominance for \(A\) requires the second step to be zero, while recessiveness for \(A\) requires the first step to be zero. These are exact statements about means; with estimated means, the observed differences may only be approximately equal to zero or to each other.

When the Three Means Need Separate Effects

Some tables do not satisfy any of the three simpler patterns. For instance, if the heterozygote mean is neither halfway between the homozygote means nor equal to either one, an additive, dominant, or recessive coding would impose a constraint not present in the table. An unrestricted genotypic model then represents the three means directly, using \(aa\) as the reference and separate contrasts for \(Aa\) and \(AA\).

A Short Justification: Why These Checks Work

The patterns follow directly from the coded predictor values. In an additive model, genotype dosage \(x\) takes values \(0,1,2\), and the mean is \(\mu+\beta x\). Thus the three predicted means are \(\mu\), \(\mu+\beta\), and \(\mu+2\beta\). Subtracting consecutive means gives \(\beta\) both times, so equal steps are necessary. Conversely, if the two steps are equal to some value \(\beta\), setting \(\mu=m_0\) gives exactly those three means under the additive formula.

For dominant coding, the predictor is 0 for \(aa\) and 1 for both \(Aa\) and \(AA\). A linear mean model in this predictor must give the same predicted mean to those latter genotypes. For recessive coding, the predictor is 0 for \(aa\) and \(Aa\), and 1 for \(AA\), requiring the first two means to match. In unrestricted coding, use separate indicators for \(Aa\) and \(AA\), with \(aa\) as the reference. The model has two contrasts and can therefore reproduce any three specified means: choose \(\mu=m_0\), \(\delta_1=m_1-m_0\), and \(\delta_2=m_2-m_0\).

Choosing a Coding Carefully

A practical decision begins with the scientific question and the pattern of the means. If the aim is to describe a pattern that follows allele dosage, check whether the adjacent steps are similar. If a carrier effect is the intended representation, check whether the two \(A\)-carrying genotypes have similar means. If an effect is expected only for \(AA\), check whether \(aa\) and \(Aa\) have similar means. Use unrestricted genotypic coding when you need to allow the three genotypes to differ without those constraints.

For estimated means, “similar” is not the same as “exactly equal.” Sampling fluctuation can make estimates differ even when population means are equal, or make them appear equal when they are not. A table is useful for proposing or describing a coding, but deciding whether a restriction is statistically compatible with the data requires accounting for uncertainty. Also check that all three genotype groups are represented and that their means are defined; an absent genotype group cannot supply an observed mean for this comparison.

The allele named in the pattern matters. “Dominant for \(A\)” means that \(Aa\) and \(AA\) share a mean; “recessive for \(A\)” means that \(AA\) differs while \(aa\) and \(Aa\) share a mean. Reversing which allele is named changes which two genotypes are grouped. State the allele and the required equality rather than relying on the label alone.

Common Mistake

A common mistake is to call a pattern additive because the \(AA\) mean is larger than the \(Aa\) mean, or dominant because the two means look close after rounding. Neither conclusion follows from that comparison alone. Additivity requires equal adjacent steps, and dominance for \(A\) requires equality of \(m_1\) and \(m_2\). For example, means of 20, 21, and 29 cm have an increasing trend, but their steps are 1 and 8 cm, and neither dominant nor recessive equality holds. The correction is to write down the relevant differences or equality before choosing the coding.