Genotypic Coding With Two Degrees of Freedom

Why Use Two Genotype Indicators?

A quantitative trait can have three different average values across the three genotypes at a biallelic locus: one mean for each homozygote and one for the heterozygote. A model with only one genotype indicator generally cannot represent all three means independently. A model with two indicators can: it assigns one mean to a chosen reference genotype and estimates a separate difference for each of the other two genotypes.

This is a flexible way to describe genotype-specific means. It does not require the heterozygote mean to lie halfway between the homozygote means, nor does it require either homozygote to share the heterozygote mean. The two coefficients describe contrasts with the reference genotype; they are not automatically per-allele effects.

Definition: Let the three genotype categories be $aa$, $Aa$, and $AA$, and take $aa$ as the reference genotype. For individual $i$, define $I(G_i=Aa)=1$ if that individual has genotype $Aa$, and 0 otherwise; define $I(G_i=AA)$ in the same way. The two-degree-of-freedom genotypic mean model is $$ Y_i=\mu+\beta_{Aa}I(G_i=Aa)+\beta_{AA}I(G_i=AA)+\varepsilon_i. $$ Here $Y_i$ is the quantitative trait, $\mu$ is the mean for the reference genotype, and each $\beta$ is the difference between the corresponding genotype mean and the reference mean.

What the Model Says for Each Genotype

The two indicator variables are mutually exclusive: an individual belongs to one genotype category, not both non-reference categories. Substituting each genotype into the model shows exactly which mean it represents.

\begin{aligned} E(Y_i\mid G_i=aa)&=\mu,\\ E(Y_i\mid G_i=Aa)&=\mu+\beta_{Aa},\\ E(Y_i\mid G_i=AA)&=\mu+\beta_{AA}. \end{aligned}

Thus, $\mu$ is the mean trait value for $aa$, $\beta_{Aa}$ is the $Aa$-minus-$aa$ mean difference, and $\beta_{AA}$ is the $AA$-minus-$aa$ mean difference. If the trait is measured in centimetres, both coefficients are measured in centimetres. A positive coefficient means that the corresponding genotype mean is higher than the reference mean; a negative coefficient means it is lower.

The model has three parameters for the three genotype-specific means: an intercept and two coefficients. The phrase “two degrees of freedom” refers to the two genotype effects beyond the intercept. These two effects can take any values. Consequently, the three means can have any pattern, provided the model’s parameters are defined. This is a model for means, not a claim that genotype causes the trait difference.

Worked Example: Reading the Coefficients as Mean Differences

Suppose a hypothetical study reports trait means of 10.0 units for $aa$, 13.0 units for $Aa$, and 8.0 units for $AA$. Use $aa$ as the reference and write the fitted mean model.

The intercept is the reference mean, so $\hat{\mu}=10.0$ units. The heterozygote coefficient is its mean minus the reference mean: $$ \hat{\beta}_{Aa}=13.0-10.0=3.0\text{ units}. $$ The alternate-homozygote coefficient is: $$ \hat{\beta}_{AA}=8.0-10.0=-2.0\text{ units}. $$

The fitted model is therefore $ \hat{Y}_i=10.0+3.0I(G_i=Aa)-2.0I(G_i=AA)$, in trait units. For $aa$, both indicators are zero and the fitted mean is $10.0$. For $Aa$, the first indicator is one and the second is zero, giving $10.0+3.0=13.0$. For $AA$, the indicators are zero and one, giving $10.0-2.0=8.0$. The coefficients reproduce all three means without imposing an equal-step pattern.

Model Assumptions and Estimation

The indicator representation is an algebraic way to encode genotype categories. To interpret it as a statistical model for a quantitative trait, specify the conditional mean and the error assumptions. A standard starting point is that, conditional on the observed genotypes, each individual’s trait equals the appropriate genotype-specific mean plus an error with mean zero:

E(\varepsilon_i\mid G_i)=0.

For the usual least-squares model-based standard errors and tests, one commonly additionally assumes that errors from different individuals are independent and have a common variance $\sigma^2$. Normally distributed errors are needed for exact small-sample $t$- and $F$-inference under this model, but not to define the coding or to calculate the fitted group means. These assumptions concern the trait observations and inferential procedure; they do not follow from the choice of genotype indicators.

With an intercept and both indicators, ordinary least squares estimates the three genotype-specific means when all three genotype categories are observed. Each fitted mean is the sample mean within its genotype group. The coefficients are then differences between these group means and the reference group mean. The derivation below makes this relationship explicit.

Identifiability condition: To estimate all three genotype-specific means from the data using this model, each of $aa$, $Aa$, and $AA$ must occur at least once. If a genotype category is absent, its mean—and therefore its contrast with the reference—cannot be determined from the observed data alone.

Why Least Squares Gives the Genotype Group Means

Write $m_g$ for the fitted mean assigned to genotype group $g$, and let $y_{ig}$ denote the observed trait for individual $i$ in that group. The sum of squared residuals separates into one sum for each genotype:

S=\sum_g\sum_{i=1}^{n_g}(y_{ig}-m_g)^2,

where $n_g$ is the number of observations in group $g$. For each group, define its sample mean as $\bar{y}_g=\frac{1}{n_g}\sum_{i=1}^{n_g}y_{ig}$. Subtract and add this group mean inside each squared residual. The cross-product term sums to zero because $\sum_i(y_{ig}-\bar{y}_g)=0$:

\begin{aligned} \sum_{i=1}^{n_g}(y_{ig}-m_g)^2 &=\sum_{i=1}^{n_g}\left[(y_{ig}-\bar{y}_g)+(\bar{y}_g-m_g)\right]^2\\ &=\sum_{i=1}^{n_g}(y_{ig}-\bar{y}_g)^2 +n_g(\bar{y}_g-m_g)^2. \end{aligned}

The first term does not depend on $m_g$, and the second is minimized when $m_g=\bar{y}_g$. Since the three group means can be set independently by the intercept and two indicators, least squares fits each observed genotype group at its sample mean. In the reference parameterization, this gives $\hat{\mu}=\bar{y}_{aa}$, $\hat{\beta}_{Aa}=\bar{y}_{Aa}-\bar{y}_{aa}$, and $\hat{\beta}_{AA}=\bar{y}_{AA}-\bar{y}_{aa}$.

Worked Example: Estimating Effects From Hypothetical Observations

Consider hypothetical trait measurements in units for three genotype groups: $aa$: 4, 6, and 5; $Aa$: 7, 8, and 6; $AA$: 3, 5, and 4. Take $aa$ as the reference. Assume the observations are independent and that the conditional mean model is appropriate. Calculate the group means and model coefficients.

For $aa$, the sample mean is $(4+6+5)/3=15/3=5.0$ units. For $Aa$, it is $(7+8+6)/3=21/3=7.0$ units. For $AA$, it is $(3+5+4)/3=12/3=4.0$ units.

The reference mean estimates the intercept: $\hat{\mu}=5.0$ units. The $Aa$ contrast is $\hat{\beta}_{Aa}=7.0-5.0=2.0$ units, and the $AA$ contrast is $\hat{\beta}_{AA}=4.0-5.0=-1.0$ units. Therefore the fitted model is $ \hat{Y}_i=5.0+2.0I(G_i=Aa)-1.0I(G_i=AA)$, with all terms in trait units.

Checking the fitted values: $aa$ has fitted mean $5.0$, $Aa$ has fitted mean $5.0+2.0=7.0$, and $AA$ has fitted mean $5.0-1.0=4.0$. These match the three sample means. The individual observations still vary around those means; the model does not assert that every person with a given genotype has exactly the same trait value.

When the Effects Cannot Be Estimated

The formula can be written even when a genotype category is absent, but that does not mean every parameter can be estimated. If no individual has genotype $AA$, then $I(G_i=AA)=0$ for everyone. The data contain no information about $\beta_{AA}$: changing it leaves every fitted value unchanged. This is a rank-deficient design, and the $AA$-specific mean cannot be estimated without additional assumptions or information.

A very small genotype group is different from an absent one. If at least one observation occurs in each category, all three means are algebraically estimable, though estimates based on small groups may be imprecise. Estimability and precision are separate questions: having a parameter that can be calculated does not guarantee that it is estimated accurately.

Worked Example: Recognizing an Unobserved Genotype

Suppose a hypothetical sample contains five $aa$ individuals and four $Aa$ individuals, but no $AA$ individuals. Their observed trait values are sufficient to calculate the $aa$ and $Aa$ means. Can the model estimate all three genotype means?

For every observed individual, $I(G_i=AA)=0$. Thus each fitted value has the form $\mu+\beta_{Aa}I(G_i=Aa)$; the term involving $\beta_{AA}$ is always zero. For any two proposed values $b_1$ and $b_2$ of $\beta_{AA}$, the fitted values on this sample are identical, because $$ \mu+\beta_{Aa}I(G_i=Aa)+b_1(0) = \mu+\beta_{Aa}I(G_i=Aa)+b_2(0). $$

Therefore $\beta_{AA}$ is not identified by these observations. The sample can estimate the means for $aa$ and $Aa$, but it cannot estimate the $AA$ mean or its contrast with $aa$. Assigning a numerical $AA$ effect would require an extra constraint or external data, neither of which is provided by the two-indicator model itself.

Interpreting the Two Degrees of Freedom

The two coefficients answer two specific questions: how far the $Aa$ mean is from the $aa$ mean, and how far the $AA$ mean is from the $aa$ mean. Together they describe the genotype pattern relative to that baseline. For example, one contrast can be positive while the other is negative, as in the first worked example. No ordering of the three means is built into the coding.

If the scientific question is whether the genotype-specific means differ at all, the corresponding null statement is that both genotype contrasts are zero:

H_0:\beta_{Aa}=0\ \text{and}\ \beta_{AA}=0.

Under this null, all three conditional means equal $\mu$. A joint test of these two restrictions addresses the overall genotype association in the specified mean model; a test of just one coefficient addresses only that coefficient’s contrast with the reference. Any inferential conclusion also depends on the sampling design and the error assumptions used for the test. A detected difference is an association in the modeled data, not by itself evidence that genotype caused the trait difference.

The reference genotype determines the interpretation of the intercept and coefficients. Here the intercept is the $aa$ mean, and both effects compare another genotype with $aa$. State the reference genotype whenever reporting the estimates, so that the baseline and contrasts are unambiguous.

Common Mistakes

Interpreting a coefficient as an allele effect. The $\beta$ coefficients are genotype-versus-reference mean differences. They are not automatically changes per additional copy of an allele.
Assuming the heterozygote must be intermediate. The model imposes no ordering or equal-spacing constraint. The three means can take any values.
Counting the intercept as a genotype degree of freedom. The intercept sets the reference mean; the two indicator coefficients are the two genotype contrasts beyond that baseline.
Ignoring an absent genotype category. A coefficient whose indicator is always zero cannot be estimated from the sample. A software output cannot supply information that the data do not contain.
Confusing a fitted group mean with every individual’s value. Individual trait values can differ within a genotype group; the model describes conditional means and residual variation.

Key takeaway: With a reference genotype and one indicator for each of the other two genotypes, the intercept and two coefficients represent three unrestricted genotype-specific means. Interpret each coefficient as a mean contrast with the stated reference, and verify that all three categories are observed before estimating the full model.

Check Your Understanding

Use the two-indicator model with $aa$ as the reference genotype.

If the three genotype means are 6, 9, and 11 trait units for $aa$, $Aa$, and $AA$, respectively, what are the intercept and two coefficients?
What does a value of $-2.5$ for $\beta_{AA}$ mean in context?
Why can the model represent any three genotype-specific means without requiring equal differences between adjacent genotype categories?
What happens to estimability of $\beta_{AA}$ if no $AA$ individual is observed?
Does a nonzero genotype contrast establish that genotype causes the trait difference? Explain.