Within-Family and Between-Family Components in QTDT

Why Separate Within-Family and Between-Family Variation?

A genotype–trait association can arise because people with different genotypes have different trait values within their families, because families with different average genotypes also differ in their average trait values, or both. These patterns have distinct interpretations. In particular, between-family differences may reflect ancestry or family environment rather than an effect of genotype on the trait.

The quantitative transmission disequilibrium test (QTDT) uses a decomposition of genotype dosage to distinguish these sources of variation. The within-family component compares an individual’s genotype with the average genotype in that individual’s family. The between-family component compares family-average genotypes across families. The distinction is about the source of genotype variation used for inference, not a claim that either association is automatically causal.

Definition: Let \(G_{ij}\) be the allele dosage for individual \(i\) in family \(j\), typically coded as 0, 1, or 2 copies of a specified allele. For the analyzed individuals in family \(j\), let \(\bar{G}_j\) be their mean dosage. The within-family component is \(G_{ij}-\bar{G}_j\), and the between-family component is \(\bar{G}_j\). Thus \(G_{ij}=(G_{ij}-\bar{G}_j)+\bar{G}_j\).

The family mean is calculated over the individuals included in the analysis for that family. It need not be an integer. For example, if a family has observed dosages 0, 1, and 2, its mean dosage is 1. The corresponding within-family deviations are −1, 0, and 1; they sum to zero within the family.

The QTDT Mean Model and Its Coefficients

A standard linear QTDT mean model for a quantitative trait \(Y_{ij}\) is

Y_{ij}=\mu+\boldsymbol{C}_{ij}^{\mathsf T}\boldsymbol{\gamma} +\beta_W(G_{ij}-\bar{G}_j)+\beta_B\bar{G}_j+u_j+\varepsilon_{ij}.

Here, \(\mu\) is an intercept, \(\boldsymbol{C}_{ij}\) is a vector of measured covariates with coefficients \(\boldsymbol{\gamma}\), \(u_j\) is a family-level random effect, and \(\varepsilon_{ij}\) is an individual residual. The model is commonly paired with mean-zero random effects and residuals, with a specified variance structure; the conventional random-effects interpretation also assumes the family effect is independent of the modeled genotype and covariates. Inference additionally relies on an appropriate model for the trait and on accurate genotypes and family relationships. If covariates or residual distributions are modeled incorrectly, the fitted association can be misleading.

The coefficient \(\beta_W\) describes the expected trait difference, in trait units, associated with a one-copy increase in dosage relative to the family mean, holding the modeled covariates and between-family component fixed. It is estimated from genotype differences among members of the same family. The coefficient \(\beta_B\) describes the expected trait difference associated with a one-copy increase in the family mean dosage, comparing families under the model. Its interpretation depends more strongly on assumptions about differences between families.

For example, if \(Y\) is measured in millimeters, \(\beta_W=1.2\) means that a one-copy increase in within-family dosage is associated with an expected 1.2 mm increase in the trait, conditional on the model. It does not mean that every individual’s trait changes by 1.2 mm when their genotype is changed, nor does it establish a causal effect.

Worked Example: Calculating Family-Centered Dosages

Suppose three hypothetical members of one family have dosages \(0,1,2\), and their trait values are \(10,12,14\) units. Calculate the family mean dosage and the within-family components.

Family mean dosage.

\bar{G}_j=\frac{0+1+2}{3}=1.

Within-family components. Subtract 1 from each dosage: \(0-1=-1\), \(1-1=0\), and \(2-1=1\). Their sum is \(-1+0+1=0\), as it must be for deviations from that family’s mean.

The trait values rise by 2 units for each additional allele copy in this hypothetical family, so the within-family pattern is consistent with a positive association. One family alone, however, is not enough to establish a general association or quantify its uncertainty.

What the Decomposition Separates

The decomposition has a useful algebraic property: within-family deviations sum to zero. This makes the within-family and between-family contributions to a pooled genotype–trait covariance separable. To see this, write \(G_{ij}-\bar{G}\) as \((G_{ij}-\bar{G}_j)+(\bar{G}_j-\bar{G})\), where \(\bar{G}\) is the overall mean dosage. Likewise, write \(Y_{ij}-\bar{Y}\) as \((Y_{ij}-\bar{Y}_j)+(\bar{Y}_j-\bar{Y})\). On multiplying these expressions and summing over individuals, the cross-products vanish: within each family, the deviations from the family mean sum to zero.

\sum_j\sum_i(G_{ij}-\bar{G})(Y_{ij}-\bar{Y}) = \sum_j\sum_i(G_{ij}-\bar{G}_j)(Y_{ij}-\bar{Y}_j) + \sum_j n_j(\bar{G}_j-\bar{G})(\bar{Y}_j-\bar{Y}).

Here \(n_j\) is the number of analyzed individuals in family \(j\). The first term measures within-family covariation; the second measures between-family covariation, weighted by family size. The identity shows why a pooled association can combine distinct patterns. It does not, by itself, make either term a causal effect or account for covariates and the correlation among relatives.

Worked Example: Estimating a Within-Family Slope

Consider these hypothetical observations. Each family has a different average trait value, so estimate the within-family slope after centering dosage and trait within each family. Assume a common linear within-family slope and no additional covariates.

Family 1 has dosages \(0,1,2\) and trait values \(10,12,14\). Family 2 has dosages \(0,1,2\) and trait values \(30,31,32\). Family 3 has dosages \(1,1,2\) and trait values \(20,21,22\).

For a model with family-specific intercepts, the within-family slope is the sum of within-family cross-products divided by the sum of squared within-family dosage deviations:

\hat{\beta}_W= \frac{\sum_j\sum_i(G_{ij}-\bar{G}_j)(Y_{ij}-\bar{Y}_j)} {\sum_j\sum_i(G_{ij}-\bar{G}_j)^2}.

In Family 1, the dosage deviations are \((-1,0,1)\) and the trait deviations are \((-2,0,2)\). Its cross-product sum is \(2+0+2=4\), and its squared-dosage-deviation sum is \(1+0+1=2\). In Family 2, the corresponding deviations are \((-1,0,1)\) and \((-1,0,1)\), giving cross-product sum \(2\) and squared-deviation sum \(2\). In Family 3, the dosage mean is \(4/3\), so the dosage deviations are \((-1/3,-1/3,2/3)\); the trait mean is 21, so the trait deviations are \((-1,0,1)\). Its cross-product sum is \(1/3+0+2/3=1\), and its squared-deviation sum is \(1/9+1/9+4/9=2/3\).

Combining the three families gives

\hat{\beta}_W=\frac{4+2+1}{2+2+2/3} =\frac{7}{14/3}=1.5.

The estimated within-family association is 1.5 trait units per additional allele copy relative to the family mean. This calculation uses within-family information; the large differences in average trait values across the three families do not enter its numerator as between-family contrasts.

Why the Within-Family Component Limits Stratification Bias

Population stratification occurs when groups with different ancestry have different allele frequencies and also differ in a trait for reasons unrelated to the tested genotype. A between-family association can reflect this pattern: families with higher average dosage may also have higher average trait values because their ancestry, environment, or both differ.

The within-family component instead uses deviations around each family’s own genotype mean. A factor shared by all members of a family, such as broad ancestry or a family-level environment, cannot by itself create a within-family genotype contrast. Under Mendelian segregation, the transmitted alleles vary among relatives conditional on parental genotypes; this gives within-family comparisons a basis that is less exposed to differences in allele frequency between ancestry groups. In the QTDT model, the within-family coefficient can therefore be informative even when the between-family coefficient is vulnerable to such differences.

This protection is not absolute. It depends on correct genotyping and family structure, and on the absence of relevant confounding that varies among relatives and is associated with their genotypes. For example, a sibling-specific environmental factor related to both genotype and trait could still distort a within-family association. Nor does resistance to population stratification turn an association into proof of causation.

Worked Example: A Between-Family Pattern Without a Within-Family Pattern

Suppose three hypothetical families each contribute two members. Their dosages are \(0,1\) in Family A, \(1,2\) in Family B, and \(2,3\) in Family C. Both members of each family have the same trait value: 10 units in A, 15 in B, and 20 in C. This simplified example represents a situation where family-level background differs across families.

The family mean dosages are \(0.5,1.5,2.5\), and the family mean trait values are \(10,15,20\). Regressing the three trait means on the three dosage means gives a between-family slope of

\hat{\beta}_B= \frac{(-1)(-5)+0(0)+(1)(5)} {(-1)^2+0^2+1^2} =\frac{10}{2}=5

trait units per additional copy in family mean dosage. Yet within each family the two trait values are identical. Thus every within-family trait deviation is zero, and the within-family cross-product is zero. The apparent between-family pattern does not demonstrate a within-family genotype association: family-level ancestry or environment could produce the same pattern.

Interpreting the Two Results Together

The within-family and between-family estimates answer different questions. A clear within-family association is evidence that genotype differences among relatives track trait differences under the model. A between-family association indicates that family-average dosage and family-average trait covary; it may include genetic association, population structure, shared environment, or combinations of these. Disagreement between the estimates is therefore not automatically a contradiction. It can signal that between-family structure matters.

A practical QTDT analysis should report which component is being tested, its estimated effect and uncertainty, the allele-dosage coding, and the trait units. It should also describe the families and individuals contributing genotype variation: families with no variation in dosage contribute no direct information to the within-family slope. Inference must account for related observations and use a variance model appropriate to the design.

Common Mistake

Wrong reasoning: “The overall genotype–trait association is significant, so the result cannot be due to population stratification.”
Correction: A pooled association may combine within-family and between-family covariance. Examine the components separately. The within-family component limits confounding from factors shared within families, but it does not rule out all bias or establish causation.
Wrong reasoning: “The between-family coefficient is the same as the within-family genetic effect.”
Correction: The coefficients use different sources of variation and can differ. The between-family coefficient is especially sensitive to family-level structure and requires stronger assumptions for a genetic interpretation.

Key takeaway: QTDT decomposes dosage into \(G_{ij}-\bar{G}_j\) and \(\bar{G}_j\). The first supports a within-family association interpretation and reduces vulnerability to population stratification; the second captures between-family association and can reflect ancestry or shared environment as well as genetic effects.

Check Your Understanding

Answer each question using the distinction between within-family and between-family variation.

A family has dosages 0, 1, and 2. What is its mean dosage, and what are the within-family components?
In the QTDT model, interpret \(\beta_W\) and \(\beta_B\), including the units of each coefficient.
Why can a between-family genotype–trait association be confounded by population stratification?
What property of within-family deviations makes the covariance decomposition separate into within-family and between-family terms?
Does a within-family association prove that genotype causally changes the trait? Explain one assumption or limitation relevant to the interpretation.