Two-Degree-of-Freedom Genotypic Models

Why Use an Unrestricted Genotype Model?

A per-allele additive model summarizes genotype with a single dosage slope: each additional copy of the allele is assumed to change the outcome by the same amount on the model's scale. That can be a useful and economical description, but it rules out some patterns. For example, the heterozygote could have a higher mean than either homozygote, or the two nonreference genotypes could differ in opposite directions from the reference.

An unrestricted genotypic model avoids that particular shape constraint. It estimates a separate mean or risk for each of the three genotypes. The price is that it uses two genotype parameters beyond the reference level, so the overall test of genotype association has two degrees of freedom. This tutorial develops that model and the corresponding omnibus test; it does not assume that the genotype-specific pattern follows a straight line in allele dosage.

Definition: Let $G_i$ denote the genotype of individual $i$, coded as 0, 1, or 2 copies of a specified allele. An unrestricted three-genotype model assigns a separate outcome mean or risk to each genotype. With genotype 0 as the reference, its mean-scale form is $$ E(Y_i\mid G_i)=\beta_0+\beta_1 I(G_i=1)+\beta_2 I(G_i=2), $$ where $I(\cdot)$ equals 1 when its condition is true and 0 otherwise. Thus the genotype-specific means are $\mu_0=\beta_0$, $\mu_1=\beta_0+\beta_1$, and $\mu_2=\beta_0+\beta_2$.

The reference genotype is a coding choice, not a claim that it is biologically normal or has no effect. Under this coding, $\beta_1$ is the difference between the mean for genotype 1 and the mean for genotype 0, and $\beta_2$ is the corresponding difference for genotype 2. Changing the reference changes the coefficient labels and values, but not the three fitted genotype-specific means or the overall equality hypothesis.

The unrestricted model's global null hypothesis is $H_0:\beta_1=\beta_2=0$, equivalently $H_0:\mu_0=\mu_1=\mu_2$. Its alternative is that at least one genotype-specific mean differs. This is one joint test with two degrees of freedom, not two separate one-degree-of-freedom tests. The same structure applies to genotype-specific risks for a binary outcome, with $p_g=P(Y=1\mid G=g)$ in place of $\mu_g$.

Worked Example: Reading the Reference-Genotype Coefficients

Suppose a hypothetical sample has a quantitative trait measured in millimetres. The sample means are 12 mm for genotype 0, 15 mm for genotype 1, and 11 mm for genotype 2. Use genotype 0 as the reference and express the fitted genotype means in the unrestricted model.

The reference mean sets the intercept: $\hat{\beta}_0=\bar{Y}_0=12$ mm. The first nonreference coefficient is the difference from that mean: $\hat{\beta}_1=\bar{Y}_1-\bar{Y}_0=15-12=3$ mm. The second is $\hat{\beta}_2=\bar{Y}_2-\bar{Y}_0=11-12=-1$ mm.

\widehat{E}(Y_i\mid G_i) =12+3I(G_i=1)-1I(G_i=2)\quad\text{mm}.

Substitution gives fitted means of $12$ mm for genotype 0, $12+3=15$ mm for genotype 1, and $12-1=11$ mm for genotype 2. These are descriptive sample estimates; this calculation alone does not test whether the population means differ. No distributional assumption is needed to calculate the sample means, although inference about population means requires an appropriate sampling and outcome model.

Testing Genotype Means for a Quantitative Outcome

For a quantitative trait, a standard unadjusted test is the one-way analysis-of-variance $F$-test. In its usual form, assume that individuals are independent, that the outcome within each genotype group is normally distributed, and that the three groups share a common within-group variance. The null is that all three population means are equal. The model allowing three separate means is compared with the null model that has one common mean.

Let $n_g$ and $\bar{x}_g$ be the sample size and sample mean in genotype group $g$, and let $N=n_0+n_1+n_2$. The overall mean is $\bar{x}=\sum_g n_g\bar{x}_g/N$. The between-group sum of squares measures how far the three sample means lie from the overall mean; the within-group sum of squares measures variation among individuals around their own genotype mean. The test statistic is the ratio of their mean squares.

SS_{\text{between}}=\sum_{g=0}^{2}n_g(\bar{x}_g-\bar{x})^2,\qquad SS_{\text{within}}=\sum_{g=0}^{2}\sum_{i:G_i=g}(x_i-\bar{x}_g)^2,

F=\frac{SS_{\text{between}}/2}{SS_{\text{within}}/(N-3)}.

There are two numerator degrees of freedom because estimating three group means uses two more parameters than estimating one common mean. The denominator has $N-3$ degrees of freedom because the unrestricted model estimates three group means. Under the null and the stated assumptions, this statistic follows an $F$ distribution with 2 and $N-3$ degrees of freedom.

Worked Example: An Omnibus Test of Three Quantitative Means

Consider a hypothetical trait measured in arbitrary units, with four independent individuals in each genotype group:

Genotype 0: 2, 4, 4, 6; genotype 1: 4, 6, 6, 8; genotype 2: 1, 3, 5, 7. Assume for this example that the group distributions are sufficiently compatible with the normal, equal-variance model for the $F$-test.

The group means are $\bar{x}_0=4$, $\bar{x}_1=6$, and $\bar{x}_2=4$. The overall mean is $\bar{x}=(16+24+16)/12=56/12=4.667$ units. Therefore,

SS_{\text{between}} =4(4-4.667)^2+4(6-4.667)^2+4(4-4.667)^2 =10.667.

Within genotype 0, the squared deviations from 4 sum to $4+0+0+4=8$. Within genotype 1, the squared deviations from 6 also sum to 8. Within genotype 2, the squared deviations from 4 sum to $9+1+1+9=20$. Thus $SS_{\text{within}}=8+8+20=36$, with $12-3=9$ denominator degrees of freedom.

F=\frac{10.667/2}{36/9}=\frac{5.3335}{4}=1.333.

For an $F$ distribution with 2 and 9 degrees of freedom, the upper-tail $p$-value is approximately 0.311. At significance level $\alpha=0.05$, this sample does not provide sufficient evidence that the three population means differ. This does not establish that the means are equal; the sample is small, and a nonsignificant result may reflect limited information as well as small differences.

Testing Genotype-Specific Risks

For a binary outcome, define the genotype-specific risk $p_g=P(Y=1\mid G=g)$. Its direct sample estimate is the proportion with outcome $Y=1$ in genotype group $g$: $\hat{p}_g=y_g/n_g$, where $y_g$ is the number with the outcome and $n_g$ is the group size. The unrestricted risk model allows each of $p_0,p_1,p_2$ to take its own value. The null of no genotype association is $H_0:p_0=p_1=p_2$.

An unadjusted Pearson chi-square test of homogeneity compares the observed counts in the genotype-by-outcome table with the counts expected if the risk is the same across genotypes. If the total number with the outcome is $Y$, the pooled risk under the null is $\hat{p}=Y/N$. For genotype group $g$, the expected outcome and non-outcome counts are $n_g\hat{p}$ and $n_g(1-\hat{p})$. Summing $(O-E)^2/E$ over the six cells gives the test statistic. When expected counts are adequate, it is compared with a chi-square distribution with $(3-1)(2-1)=2$ degrees of freedom.

Worked Example: Comparing Three Hypothetical Risks

In a hypothetical study, each of three genotype groups contains 100 individuals. The numbers with a binary outcome are 20 in genotype 0, 40 in genotype 1, and 20 in genotype 2. The estimated risks are $\hat{p}_0=20/100=0.20$, $\hat{p}_1=40/100=0.40$, and $\hat{p}_2=20/100=0.20$.

There are 80 outcomes among 300 individuals, so the pooled risk under the null is $\hat{p}=80/300=0.2667$. The expected outcome count is $100(0.2667)=26.667$ in each group, and the expected non-outcome count is $100(1-0.2667)=73.333$ in each group. All expected counts are comfortably above 5, so the usual large-sample chi-square approximation is reasonable for this table.

\begin{aligned} X^2 &=\left(\frac{(20-26.667)^2}{26.667}+\frac{(80-73.333)^2}{73.333}\right)\\ &\quad+\left(\frac{(40-26.667)^2}{26.667}+\frac{(60-73.333)^2}{73.333}\right)\\ &\quad+\left(\frac{(20-26.667)^2}{26.667}+\frac{(80-73.333)^2}{73.333}\right)\\ &=2.273+9.091+2.273=13.636. \end{aligned}

The test has 2 degrees of freedom and a $p$-value of approximately 0.0011. At $\alpha=0.05$, the data provide evidence that the risks are not all equal across genotypes. In this sample, the estimated risk is 0.20 for genotype 0, 0.40 for genotype 1, and 0.20 for genotype 2. The omnibus test alone does not establish which population risks differ; that requires appropriately designed follow-up comparisons.

Why the Test Has Two Degrees of Freedom

The parameter count gives a direct explanation. A model with one common genotype-independent mean has one mean parameter. The unrestricted model has three means, one for each genotype. The difference is $3-1=2$. In reference-indicator form, those two extra parameters are $\beta_1$ and $\beta_2$. The joint null that both are zero is exactly the statement that all three means equal the reference mean.

For the quantitative-trait $F$-test, the same comparison appears as a difference in residual variation. The null model's residual sum of squares is the total variation around $\bar{x}$:

SS_{\text{total}}=\sum_{g=0}^{2}\sum_{i:G_i=g}(x_i-\bar{x})^2.

For each individual, write $x_i-\bar{x}=(x_i-\bar{x}_g)+(\bar{x}_g-\bar{x})$. Squaring and summing within each group yields a cross-product term proportional to $\sum_{i:G_i=g}(x_i-\bar{x}_g)$, which equals zero because deviations from a group sample mean sum to zero. Consequently,

SS_{\text{total}}=SS_{\text{within}}+SS_{\text{between}}.

Thus the unrestricted model reduces the residual sum of squares by $SS_{\text{between}}$, and the $F$-statistic assesses that reduction relative to the within-group variation. This decomposition also clarifies why a large difference between group means is more persuasive when within-group variability is modest.

Interpretation and Common Pitfalls

The unrestricted model is useful when the data should be allowed to show a non-linear or otherwise non-additive pattern across the three genotypes. It estimates the genotype-specific quantities directly and tests the broad question of whether genotype groups are associated with the outcome. For a quantitative outcome, those quantities are means on the outcome scale; for a binary outcome, they are risks. The specific test and assumptions must match the outcome and design.

Mistake: treating the two coefficients as a per-allele slope. The coefficients $\beta_1$ and $\beta_2$ compare each nonreference genotype separately with genotype 0. They are not constrained to be equal, and neither is automatically the effect of one additional allele copy.
Mistake: interpreting a significant omnibus test as proof that every genotype differs. The joint test only supports the conclusion that not all three means or risks are equal. It does not identify which pairs differ.
Mistake: saying a nonsignificant test proves no association. A large $p$-value means the data did not provide sufficient evidence against the joint null under the chosen model. It does not establish exact equality.
Mistake: overlooking sparse genotype groups. Small group sizes can make mean estimates unstable and can undermine the chi-square approximation for risks. Check group counts and the relevant model assumptions before relying on a test result.

Key takeaway: With three genotypes, the unrestricted model estimates three genotype-specific means or risks. Using one genotype as the reference expresses the model with two additional coefficients, and the omnibus null that both coefficients are zero has two degrees of freedom.

Check Your Understanding

Try these questions before reviewing the worked examples.

With genotype 0 as the reference, what do $\beta_1$ and $\beta_2$ represent in the unrestricted mean model?
Why does the omnibus test of equality across three genotype-specific means have 2 numerator degrees of freedom?
A quantitative-trait $F$-test is significant. What conclusion is supported, and what conclusion is not established by that result alone?
For a binary outcome, a genotype group has 18 outcomes among 60 individuals. Calculate its estimated risk.
Name two assumptions or conditions that should be checked before using the quantitative $F$-test or the binary Pearson chi-square test.