Recessive Coding for Homozygote Effects

Why Use Recessive Coding?

A variant may be hypothesized to affect a quantitative trait only in people with two copies of the variant allele. Recessive coding represents that hypothesis with one binary predictor: variant-allele homozygotes receive a value of 1, while reference-allele homozygotes and heterozygotes both receive 0. The resulting model compares the mean trait value in the variant homozygotes with the mean among the other two genotypes combined.

This is a particular way of describing genotype groups, not evidence by itself that a biological effect is recessive. The coding makes a specific statistical assumption: the two genotype groups assigned 0 have the same modeled mean. Whether that assumption is a useful representation depends on the scientific question and the observed genotype-specific trait pattern.

From Genotype to Indicator and Model

The indicator is determined by genotype, not by the observed trait value. For example, if the reference allele is A and the variant allele is G, then AA and AG each receive \(R_i=0\), while GG receives \(R_i=1\). The labels A and G are only illustrative; the rule is always to assign 1 to the homozygote carrying two copies of the variant allele.

The model can be written for each individual as

Here \(Y_i\) is the quantitative trait for individual \(i\), \(\beta_0\) is the intercept, \(\beta_R\) is the recessive-coded coefficient, and \(\varepsilon_i\) is the individual’s deviation from the modeled mean. The coefficient has the same units as \(Y_i\). A positive \(\beta_R\) means the modeled trait mean is higher in variant homozygotes; a negative value means it is lower.

For this unadjusted model, the mean-model assumption is that the expected trait value is \(\beta_0\) for each of the two genotypes coded 0 and \(\beta_0+\beta_R\) for the genotype coded 1. Equivalently, the residual has mean zero within each coding group. To use ordinary least squares for the coefficient estimates, both coding groups must be represented in the data. If there are no variant homozygotes, the recessive effect cannot be estimated from that sample. If the groups have very different residual variability, the group mean estimates remain defined, but standard errors and inference need to account for that variability rather than silently assuming it is equal.

What the Coefficient Estimates

In the unadjusted model, ordinary least squares estimates the intercept as the sample mean among individuals with \(R_i=0\). It estimates the mean for \(R_i=1\) as the sample mean among variant homozygotes. The coefficient is the difference between those two sample means. Thus the two genotypes coded 0 are pooled before making the comparison.

This pooling is weighted by the number of observations in each genotype group. It is not generally the simple average of the reference-homozygote mean and heterozygote mean. If there are \(n_0\) individuals coded 0 and \(n_1\) individuals coded 1, define their sample means as \(\bar{x}_0\) and \(\bar{x}_1\), respectively. Then the fitted intercept is \(\bar{x}_0\), and the fitted coefficient is \(\bar{x}_1-\bar{x}_0\).

Why the Mean-Difference Result Holds

The equivalence between this regression and a two-group mean comparison follows directly from least squares. Write the residual sum of squares as separate sums for the \(R_i=0\) and \(R_i=1\) groups. The fitted mean for the first group is \(\beta_0\); the fitted mean for the second is \(\beta_0+\beta_R\). These are two freely varying group means, expressed through a baseline and a difference.

For a group with observations \(y_1,\ldots,y_m\), the sum of squared deviations from a proposed mean \(a\) is minimized at the sample mean. To verify this, differentiate \(\sum_{j=1}^{m}(y_j-a)^2\) with respect to \(a\); setting the derivative equal to zero gives \(-2\sum_{j=1}^{m}(y_j-a)=0\), so \(a=\sum_j y_j/m\). Applying this fact to the two terms in \(S\), the minimizing fitted mean for \(R_i=0\) is \(\bar{x}_0\), and the minimizing fitted mean for \(R_i=1\) is \(\bar{x}_1\). Since the second fitted mean is \(\beta_0+\beta_R\), it follows that

This derivation also shows why both indicator groups must be observed: without observations in one group, its mean cannot be estimated, so the difference between the two fitted means is not identifiable.

Assumptions, Uses, and Limits

Recessive coding is useful when the scientific question specifically concerns a trait difference for variant homozygotes relative to the other two genotypes together. Its simple coefficient is easy to describe, but that simplicity comes from the grouping restriction. The model does not estimate three separate genotype means. It represents two means: one for the pooled non-homozygote group and one for variant homozygotes.

For the coding to be interpreted as intended, genotype categories must be assigned correctly and the variant allele must be identified consistently. The trait must be measured on a scale for which a mean difference is meaningful. The model also presumes that, in the population or analysis being described, reference homozygotes and heterozygotes have the same expected trait value under this coding. If they differ systematically, the pooled mean still can be calculated, but the recessive model does not describe both genotype-specific means separately.

A coefficient from an unadjusted model describes a sample or population mean difference, not a causal effect. Other characteristics may differ between genotype groups, and a difference in group means alone does not rule out confounding, selection, or measurement error. If covariates are added in a later analysis, coefficient interpretation depends on that adjusted model; the simple pooled-mean calculation here applies to the unadjusted model.

A further practical concern is that variant homozygotes may be uncommon. A small number of such individuals can make the estimated mean and coefficient imprecise, while no variant homozygotes make the coefficient unestimable. The fact that the code is binary does not guarantee a precise estimate: precision depends on sample size and trait variability in both groups.

Common Mistakes

• Coding every carrier as exposed. That assigns 1 to heterozygotes as well as variant homozygotes. Recessive coding assigns 1 only to variant homozygotes; both other genotypes receive 0.
• Comparing variant homozygotes only with reference homozygotes. The coefficient in this model compares variant homozygotes with the pooled group of reference homozygotes and heterozygotes.
• Taking an unweighted average of the two comparison means. The pooled mean weights each genotype mean by its sample size. The unweighted average is correct only when the two group sizes are equal.
• Calling a negative coefficient a reduction caused by the allele. A negative coefficient describes a lower modeled mean in the variant homozygotes in this comparison. It does not establish that the genotype caused the difference.
• Treating the coding assumption as a biological conclusion. Recessive coding imposes a shared mean for the two genotypes coded 0; it does not prove that their true means are identical or establish a biological mechanism.