Choosing Additive, Dominant, Or Recessive Coding

Why Genotype Coding Changes the Question

A biallelic variant has three possible genotypes. Once an effect allele has been designated, let \(G\) be the number of copies of that allele: \(G=0,1,\) or \(2\). A regression analysis must represent these three values with one or more predictors. The choice is not merely a computational convenience: it determines which patterns of genotype-specific trait means are represented by the model and, therefore, what association pattern is being tested.

This tutorial compares three common choices for a quantitative trait: additive dosage, dominant, and recessive coding. The aim is to understand what each coding says about the means, not to assume that any one genetic pattern is true in advance. Genotype indicators for unrestricted nonadditive effects provide a useful reference: they allow the heterozygote and the two homozygote groups to have separate means. Each one-predictor coding considered here makes a more specific restriction.

For clarity, the basic models below have an intercept and one genotype predictor, with no additional covariates. Their mean patterns can be read directly from the predicted mean at each genotype. In an applied GWAS, other variables may also be included, but that does not change the meaning of the genotype coding itself.

Here \(\beta_A\) is a change in mean trait value per additional effect-allele copy under the additive model. The dominant coefficient \(\beta_D\) is the mean difference between carriers and noncarriers under the dominant model. The recessive coefficient \(\beta_R\) is the mean difference between individuals with two effect-allele copies and those with zero or one. These coefficients have trait units, but they describe different contrasts and should not be compared as if they were the same effect.

Assumptions and What Each Test Addresses

Each coding is commonly fitted by ordinary least squares. For a model with predictor \(X\), the conditional mean is specified as \(E[Y\mid X]=\beta_0+\beta_1X\), where \(X\) is \(G\), \(D\), or \(R\), respectively. The standard regression test of no association tests \(H_0:\beta_1=0\) against a two-sided alternative. Under the specified model, this null says that the mean does not change with the coded predictor. It does not establish that the biological effect is absent in every possible model.

For the usual small-sample \(t\)-test and confidence interval from ordinary least squares, assume independent observations, a correctly specified conditional mean, and errors with constant variance and a normal distribution conditional on the predictor. With larger samples, inference can often rely on approximate sampling theory, but dependence between participants or unequal residual variances can still make standard errors inappropriate if ignored. The coding also requires the genotype groups that define its contrast to be represented: a dominant contrast needs both carriers and noncarriers, while a recessive contrast needs both \(G=2\) individuals and individuals with \(G=0\) or \(1\).

These models describe association, not causation. Population structure, relatedness, selection, or other factors associated with both genotype and trait can complicate interpretation. Genotype coding by itself does not control those issues. Nor does a significant result prove that the chosen pattern is the unique or biologically correct one.

Why the Binary-Coding Coefficient Is a Group Difference

The dominant and recessive models each use a binary predictor. With an intercept and a binary predictor \(Z\), the fitted model is \(E[Y\mid Z]=\beta_0+\beta_1Z\). If both groups are observed, the joint least-squares fit gives each group its own fitted mean: \(\hat\beta_0=\bar{Y}_{Z=0}\) and \(\hat\beta_0+\hat\beta_1=\bar{Y}_{Z=1}\). Subtracting these fitted means gives the coefficient interpretation.

This result follows from the joint fit, not from minimizing over the intercept while holding an arbitrary slope fixed. To see why, write the residual sum of squares as a sum over the two groups. For any fixed fitted value in a group, the sum of squared deviations is minimized when that fitted value is the group sample mean. Since the model with an intercept and binary \(Z\) can choose both group fitted values freely, the minimum occurs at the two group means. The slope is their difference. For dominant coding, \(Z=D\); for recessive coding, \(Z=R\).

Worked Examples: Reading the Mean Patterns

Choosing a Coding Before Looking at Results

Each coding is a different one-degree-of-freedom test of association. Additive coding tests for a linear change in mean across effect-allele dosage. Dominant coding tests whether the carrier mean differs from the noncarrier mean under the constraint that the two carrier groups share a mean. Recessive coding tests whether the two-copy group differs from the combined zero- and one-copy groups under the constraint that those latter groups share a mean. A coding can be useful when its pattern is scientifically motivated, but it can fit poorly if the actual genotype means do not follow its restriction.

Choosing whichever coding produces the smallest \(p\)-value after inspecting all three results creates a selection problem. Even if each test is calibrated at significance level \(\alpha=0.05\), the chance of at least one false positive is not generally 0.05 when several tests are examined and the most favorable result is reported. If three tests were independent, that chance under a global null would be \(1-(1-0.05)^3=0.1426\). The tests here are usually correlated because they use the same individuals and genotype groups, so 0.1426 is an illustration under independence, not the actual error rate for every analysis.

A defensible approach is to specify a primary coding before examining association results, based on the scientific question or a justified prior model. If multiple encodings are explored, report that exploration transparently and use an inferential strategy that accounts for the selection or confirm the pattern in independent data. Do not describe the smallest unadjusted \(p\)-value as though only one prespecified test had been performed.

Diagnostics and Limitations

• Inspect all three genotype counts and means. A coding imposes equality constraints on some means. Large departures from those constraints suggest that the chosen simple pattern may not describe the data well.
• Check for sparse groups. A small \(G=2\) group can make the recessive contrast unstable; few noncarriers can similarly undermine the dominant contrast. If a defining group is absent, its contrast cannot be estimated from these data.
• Keep the allele definition fixed. Switching which allele is counted changes the interpretation of \(G\), including the direction of an additive slope and which group is labeled homozygous for the counted allele. State the counted allele and genotype mapping.
• Do not treat a coding as a diagnostic cure. A more favorable \(p\)-value does not demonstrate that a model's mean constraints are correct, and a nonsignificant result does not prove equal genotype means.
• Remember what this comparison leaves out. These are one-predictor quantitative-trait models. They do not by themselves handle population structure, relatedness, covariate adjustment, nonconstant residual variance, or case-control outcomes.

The main practical trade-off is simplicity versus flexibility. A one-predictor coding gives a compact, interpretable contrast and uses one regression coefficient. Genotype indicators can represent three distinct means and avoid the additive, dominant, or recessive equality constraint, but they use more parameters and may be imprecise when groups are small. The appropriate choice depends on the scientific target and analysis plan, not on which encoding looks best after testing.