Fixed Effects and Confounding of Variance Components

Why Mean Structure Matters in a Family Model

Family members may resemble one another because they share alleles, but resemblance can also reflect age, sex, treatment, household, or other shared conditions. A variance-component analysis separates these explanations only to the extent that its mean and covariance models represent them adequately. If a systematic mean difference is left out, the resulting residual pattern can look like covariance. If a shared environment follows the same family pattern as a genetic component, the data may not contain enough information to distinguish the two.

The key is to model covariate effects in the expected phenotype while representing relatedness and shared conditions in the covariance matrix. These are separate parts of the model, but errors in either part can affect estimated genetic variance.

Definition: For a vector $y$ of quantitative phenotypes from $n$ pedigree members, a fixed-effects variance-component model is $$ y=X\beta+\varepsilon,\qquad \varepsilon\sim N(0,V(\theta)). $$ Here $X$ is a specified design matrix of covariates, $\beta$ is a vector of fixed-effect coefficients, and $V(\theta)$ is the covariance matrix parameterized by variance components $\theta$. Thus $E(y)=X\beta$ and $\operatorname{Cov}(y)=V(\theta)$. Conditional on the covariates and pedigree information, the model assumes a multivariate normal phenotype vector.

For example, $X$ might contain a column of ones for the intercept, centered age, and a sex indicator. A common covariance specification is

V(\theta)=\sigma_A^2K+\sigma_Q^2Q+\sigma_C^2C+\sigma_e^2I.

In this notation, $K$ is a specified additive-genetic relationship kernel, $Q$ is a specified locus-specific kernel based on identity-by-descent information, $C$ represents shared-environment covariance, and $I$ is the identity matrix. The kernels are treated as known; their scaling conventions must be stated before interpreting the associated variance components. For illustration here, take $K_{ii}=Q_{ii}=C_{ii}=1$, with $K_{ij}$ the additive genetic correlation, $Q_{ij}$ the locus-specific sharing quantity on the chosen scale, and $C_{ij}=1$ for members sharing the modeled environment and 0 otherwise. The residual component $\sigma_e^2I$ represents individual-specific variation. All component variances are nonnegative, and $V$ must be positive definite for the ordinary Gaussian likelihood.

The fixed-effect coefficients describe mean differences, in phenotype units per covariate unit or relative to the reference category. They do not, by themselves, induce covariance between relatives. Covariance components describe dependence remaining after accounting for the mean. This distinction does not guarantee that the components can be separated: that depends on the patterns in the kernels and on the pedigree and covariate data.

Fitting Covariates Alongside Pedigree Covariance

Fixed effects and covariance components are estimated together under maximum likelihood or restricted maximum likelihood. At any proposed covariance parameter value, the fixed effects describe the mean pattern, while $V(\theta)$ describes how observations vary jointly around that mean. In generalized least squares, the estimate of the fixed effects for a given $V$ is

\hat{\beta}(\theta)=(X'V(\theta)^{-1}X)^{-1}X'V(\theta)^{-1}y.

This expression requires $X$ to have full column rank. In practice, the likelihood or restricted likelihood is optimized over the covariance parameters while accounting for the fixed effects. A fixed covariate should be included because it belongs in the scientific mean model, not merely because it changes a variance estimate in a preferred direction.

Worked Example: Age and Sex in a Family-Trait Model

Suppose four hypothetical relatives have ages 20, 30, 40, and 50 years. Let the sex indicator be 1 for male and 0 for female. The fitted mean model is $E(Y_i)=\beta_0+\beta_{\text{age}}(\text{age}_i-35)+\beta_{\text{sex}}\text{male}_i$. For illustration, suppose the estimated coefficients are $\hat{\beta}_0=50$ trait units, $\hat{\beta}_{\text{age}}=0.40$ trait units per year, and $\hat{\beta}_{\text{sex}}=3.0$ trait units. The four hypothetical individuals are female aged 20, male aged 30, female aged 40, and male aged 50.

Their fitted means are, respectively, $50+0.40(20-35)+3(0)=44$, $50+0.40(30-35)+3(1)=51$, $50+0.40(40-35)+3(0)=52$, and $50+0.40(50-35)+3(1)=59$ trait units. The age coefficient is a modeled mean difference of 0.40 trait units per year, holding sex fixed; the sex coefficient is a modeled 3.0-unit difference between the coded groups, holding age fixed.

Now suppose the first two people are siblings with $K_{12}=0.5$, $Q_{12}=0.75$, and $C_{12}=1$. If the illustrative component estimates are $\hat{\sigma}_A^2=4$, $\hat{\sigma}_Q^2=2$, $\hat{\sigma}_C^2=1$, and $\hat{\sigma}_e^2=3$ trait-units squared, then their modeled covariance is $4(0.5)+2(0.75)+1(1)+3(0)=4.5$ trait-units squared. Their modeled marginal variance is $4+2+1+3=10$ trait-units squared. These calculations assume the specified kernels and fitted model; the fixed age and sex effects change the expected values, not these covariance entries.

Why Residualization Changes the Covariance

A useful justification shows why one must be careful when describing residual products after fitting fixed effects. First consider ordinary least squares with design matrix $X$, and define $M_X=I-X(X'X)^{-1}X'$. The residual vector is $r=M_Xy$, and $M_XX=0$. Under the correctly specified mean model, $y=X\beta+\varepsilon$, so $r=M_X\varepsilon$. Therefore

\operatorname{Cov}(r)=M_XV M_X', \qquad E(r_ir_j)=(M_XV M_X')_{ij}.

The second identity follows because $E(r)=0$, so the expected product equals the corresponding covariance entry. It is generally not $V_{ij}$: estimating fixed effects has projected the random variation as well as the mean. The matrix $M_X$ is symmetric for ordinary least squares, giving the often-written form $M_XVM_X$.

For generalized least squares, the residual operator is instead $R=I-X(X'V^{-1}X)^{-1}X'V^{-1}$. The residuals are $Ry$, and their covariance is $RVR'$, not $M_XVM_X$. Restricted likelihood uses contrasts that remove the fixed-effect space; its covariance calculations likewise account for that removal. These distinctions matter when using residual patterns to explain or approximate covariance.

Worked Example: An Intercept Estimate Changes a Pairwise Residual Product

Consider four hypothetical observations in two independent pairs. Each observation has variance 5 trait-units squared, the covariance within either pair is 4 trait-units squared, and covariance between pairs is zero. Thus $V$ has two $2$-by-$2$ blocks with diagonal entries 5 and off-diagonal entries 4. The mean is a common intercept, which is estimated from all four observations by ordinary least squares.

For this intercept-only fit, $M=I-J/4$, where $J$ is the $4$-by-$4$ matrix of ones. The expected product of the two residuals in the first pair is $(MVM)_{12}$. Each row sum of $V$ is 9, so $(VJ/4)_{12}=9/4$, $(JV/4)_{12}=9/4$, and $(JVJ/16)_{12}=36/16=9/4$. Consequently,

(MVM)_{12} =V_{12}-\frac{9}{4}-\frac{9}{4}+\frac{36}{16} =4-2.25-2.25+2.25 =1.75\ \text{trait-units squared}.

The expected residual product after estimating the intercept is therefore 1.75, not 4. If the residuals were instead formed around the known true intercept, their expected product would be $E(\varepsilon_1\varepsilon_2)=V_{12}=4$. The difference arises because the estimated intercept uses all four observations. Treating a fitted-intercept residual product as if it had expectation $V_{12}$ would ignore that projection.

Omitted Mean Structure Can Mimic Covariance

Suppose the true mean is $X\beta$, but the fitted mean omits a covariate. Write the omitted part of the true mean as a vector $\delta$, so the actual data can be represented as $y=X\beta+\delta+\varepsilon$. Under ordinary least squares, fitted residuals are $r=M_X\delta+M_X\varepsilon$. Since the errors have mean zero,

E(rr')=M_XVM_X'+(M_X\delta)(M_X\delta)'.

The second term is a systematic contribution from the omitted mean pattern. If members of the same family have similar covariate values, their entries in $M_X\delta$ may have the same sign. Their residual products can then be positive even when the true covariance component is smaller. A variance-component model may attribute some of this pattern to genetic or shared-environment covariance.

Worked Example: An Omitted Age Trend Creates Positive Pair Products

Use the two-pair covariance setting from the preceding example: marginal error variance 5, within-pair error covariance 4, and zero covariance between pairs. Suppose the true mean increases by 1 trait unit per year of age, but the fitted model includes only an intercept. The four hypothetical ages are 20 and 22 years in the first pair, and 40 and 42 years in the second pair.

The mean age is 31 years, so after fitting the intercept the omitted age contribution is $M\delta=(-11,-9,9,11)'$ trait units. Its product for the first pair is $(-11)(-9)=99$ trait-units squared. The random-error contribution to the expected residual product is still $(MVM)_{12}=1.75$ trait-units squared, calculated above. Hence the expected first-pair residual product under the misspecified mean is

E(r_1r_2)=99+1.75=100.75\ \text{trait-units squared}.

The large positive value is not evidence that the true within-pair error covariance is 100.75. Most of it comes from the omitted age trend: both younger relatives have negative fitted residual means after centering, so their residuals tend to have the same sign. Including age in the mean model removes this systematic source of resemblance, subject to the assumed linear age effect being adequate.

Shared Environments and Identifiability

A covariance component is identifiable only when its kernel contributes a pattern that can be distinguished from the other modeled patterns, given the available pedigree structure. If the genetic and shared-environment kernels are identical, their coefficients cannot be separately recovered from the likelihood: only their sum is identified. Even when kernels are not exactly identical, limited family structures can make them difficult to distinguish, leading to unstable estimates and strong dependence on modeling assumptions.

Worked Example: Genetic and Household Components in Sibling Pairs

Suppose the only observed families are pairs of siblings. Use the kernel conventions above: each person has $K_{ii}=C_{ii}=1$, siblings have $K_{ij}=0.5$, and siblings in the same household have $C_{ij}=1$. Assume there is no locus-specific component in this example. Then the marginal variance and within-pair covariance are $V_{ii}=\sigma_A^2+\sigma_C^2+\sigma_e^2$ and $V_{ij}=0.5\sigma_A^2+\sigma_C^2$.

For one possible covariance matrix, take $\sigma_A^2=4$, $\sigma_C^2=4$, and $\sigma_e^2=4$, all in trait-units squared. This gives diagonal variance $4+4+4=12$ and sibling covariance $0.5(4)+4=6$. A different set, $\sigma_A^2=8$, $\sigma_C^2=2$, and $\sigma_e^2=2$, gives the same diagonal variance $8+2+2=12$ and the same sibling covariance $0.5(8)+2=6$.

Thus these sibling-pair covariance patterns alone cannot distinguish the two parameter sets: they imply the same $2$-by-$2$ covariance matrix. More varied relationships, additional environmental information, or other design features may help separate components, but adding a component to the model does not by itself make it identifiable.

Assumptions, Diagnostics, and Common Mistakes

Interpretation depends on the model assumptions. The covariate form should adequately describe the conditional mean; the kernels should reflect the intended genetic and environmental relationships; and the residual distribution and covariance specification should be suitable for the likelihood method. The phenotype observations must also correspond to the modeled individuals and pedigree structure. A fitted variance component is conditional on these choices; it is not automatically a pure biological quantity.

Useful checks include examining whether age or other continuous covariates need nonlinear terms, whether categorical covariates and reference groups are coded correctly, and whether residual patterns remain associated with family, age, sex, batch, or household. One should also inspect whether covariance kernels are highly similar in the actual sample and whether estimates are unstable under reasonable changes in mean or covariance specification. Such checks diagnose model sensitivity; they do not prove that a genetic interpretation is correct.

Mistake: treating fitted residual covariance as the original error covariance. The claim “the expected residual product is $V_{ij}$” is generally wrong after fixed effects are estimated. For ordinary least squares, use $(M_XVM_X')_{ij}$; for generalized least squares, use $(RVR')_{ij}$.
Mistake: interpreting every positive family residual pattern as genetic. An omitted age trend, batch effect, or shared household condition can also produce resemblance. Correct the mean model and consider plausible covariance sources before assigning a genetic interpretation.
Mistake: assuming all variance components are separately estimable. If covariance kernels are identical, or nearly so in the observed pedigree design, the associated components are not separately identified, or may be estimated imprecisely.

Key takeaway: Fixed covariates specify the expected phenotype; pedigree and environmental kernels specify covariance around that expectation. Omitted mean structure can create apparent residual dependence, and shared-environment covariance can be confounded with genetic covariance. Interpret each component in light of residualization, kernel scaling, and identifiability.

Check Your Understanding

Answer each question by referring to the model and assumptions stated above.

In a model with centered age and sex in $X$, what does the age coefficient describe, and which part of the model represents covariance between relatives?
For ordinary least squares with an intercept, why is the expected fitted-residual product generally not equal to the corresponding entry of $V$? State the correct matrix expression.
In the omitted-age example, what produced the term 99 trait-units squared in the expected pairwise residual product?
For sibling pairs with $K_{ij}=0.5$ and $C_{ij}=1$, write the formulas for marginal variance and within-pair covariance when only additive-genetic, shared-environment, and individual-specific components are included.
What feature of the pedigree or data would be needed to help distinguish genetic from shared-environment covariance?