Why Check for Residual Ancestry Structure?
Adjusting for principal components can reduce confounding by broad population structure, but including components in a model does not guarantee that all relevant structure has been captured. A component may have been omitted, the relationship with a component may not be adequately represented, or a technical covariate may track ancestry and remain imbalanced. Diagnostics ask a focused question: after the chosen adjustment, do outcomes or genotypes still show systematic patterns across ancestry-related variables?
The goal is not to prove that ancestry has been completely removed. Principal components summarize patterns in the analyzed genotypes; they are not a complete description of ancestry, and a weak diagnostic result cannot establish that no residual structure exists. Instead, several complementary checks can reveal model inadequacy and help determine whether conclusions are sensitive to reasonable alternative adjustments.
What to Examine After Adjustment
For a phenotype \(Y_i\), a common starting model includes an intercept, selected principal components, and relevant covariates \(C_i\). The residual is the observed phenotype minus its fitted value. If the adjustment is adequate for the diagnostic question, residuals should not show a strong systematic trend across omitted components or ancestry-group labels. Plot residuals against each component, and examine group summaries as well as formal conditional comparisons.
The genotype side is also informative. For variant \(j\), let \(G_{ij}\) be individual \(i\)'s genotype dosage, usually coded 0, 1, or 2 copies of an allele. Regressing \(G_{ij}\) on the components and covariates can show whether dosage still varies with a component after adjustment for the others. Across many variants, widespread residual relationships may indicate unresolved broad structure, but isolated signals can also reflect chance, local genetic effects, or technical problems.
The diagnostic should match the model being assessed. If the association analysis adjusts for two components, for example, examine residual patterns with those components and consider whether additional components or group terms change the result. Avoid treating a group label as a definitive biological classification: labels are summaries, and individuals may lie along continuous or admixed gradients.
Worked Example: Testing an Additional Component for a Phenotype
Suppose a hypothetical continuous phenotype is modeled for 40 unrelated participants. The reduced model contains an intercept, PC1, PC2, and age. Its residual sum of squares is 180 phenotype-units squared. Adding PC3 gives a residual sum of squares of 156. The question is whether PC3 contributes information conditional on the included terms.
For an exact classical nested-model \(F\)-test, assume the conditional mean is correctly specified and the errors are independent and normally distributed with common variance. The reduced model has 4 coefficients and the full model has 5, so the full-model residual degrees of freedom are \(40-5=35\). The partial \(F\) statistic is
Under the stated assumptions, this has an \(F\) distribution with 1 and 35 degrees of freedom under the null that the PC3 coefficient is zero. The two-sided equivalent test gives \(p\approx 0.026\). Thus the hypothetical data provide evidence that PC3 is associated with the phenotype conditional on PC1, PC2, and age. This is a diagnostic signal to review the adjustment, not proof that ancestry caused the phenotype pattern. Residual plots and checks of model assumptions remain important.
Conditional Genotype Checks
A broad structure diagnostic can test whether genotype dosage remains associated with ancestry variables after adjustment. One simple approach is a linear regression of \(G_{ij}\) on the components and covariates, comparing a model with the current adjustment to one that adds a component or group term. The regression is convenient for dosage data, but dosage is discrete and bounded. Consequently, the ordinary linear-model \(F\)-test is generally an approximation here, not an exact test based on normally distributed errors. Variance patterns, sample size, and group balance affect how reliable that approximation is.
Use these tests as part of a diagnostic set rather than relying on a single variant. For a genome-wide view, examine whether many variants show residual component associations, whether patterns are concentrated in particular technical batches, and whether the findings persist under sensible changes to the component adjustment. A large number of nominal signals may reflect unresolved structure, but significance counts alone are not a diagnosis: dependence among variants and multiple testing matter.
Worked Example: A Dosage Association With an Omitted Component
For a hypothetical variant in 120 participants, regress dosage \(G\), coded 0, 1, or 2, on an intercept, PC1, and PC2. The residual sum of squares is 48.0 dosage-units squared. Adding PC3 reduces it to 45.6. The added component contributes one coefficient, and the full model has \(120-4=116\) residual degrees of freedom.
The conventional nested-model statistic is
The nominal \(F_{1,116}\) tail probability is approximately 0.015. This suggests that dosage remains associated with PC3 conditional on PC1 and PC2 in this hypothetical sample. Because the response takes only the values 0, 1, and 2, this \(p\)-value is an approximation; the discrete dosage errors are not exactly normal, so the classical exact \(F\)-test justification does not apply. Inspect dosage patterns and model residuals, and use an inference procedure suitable for the data if a confirmatory conclusion is needed. One variant's result alone does not establish genome-wide residual structure.
Group Comparisons and Covariate Patterns
A continuous component may not capture every relevant difference, so a conditional comparison across broad ancestry groups can be a useful sensitivity check. Fit a reduced model containing the chosen components and covariates, then a full model adding group indicators. The test asks whether the group terms jointly improve fit conditional on the reduced-model terms. This differs from comparing unadjusted group averages, which can reflect age, site, or other covariate differences.
Also check covariates that may themselves align with ancestry, especially technical variables such as collection site or processing batch. A residual pattern across groups can arise because the phenotype model omitted a relevant technical covariate, because the ancestry adjustment is inadequate, or for other reasons. The pattern does not distinguish these explanations on its own. Examine design information and compare defensible models rather than automatically adding every available variable.
Worked Example: Adding an Ancestry-Group Term
Consider a hypothetical continuous phenotype from 36 participants in three ancestry groups. A reduced model includes an intercept, age, and two principal components, with residual sum of squares 50.4. A full model adds two group indicators, with residual sum of squares 38.4. Assume the conditional errors are independent, normally distributed with common variance, and that the conditional mean model is correct. The full model has six coefficients, so its residual degrees of freedom are \(36-6=30\). The group effect uses two degrees of freedom.
The joint partial \(F\) statistic is
For \(F_{2,30}\), the upper-tail probability is approximately 0.0169. Under the stated model assumptions, this is evidence that the group terms jointly improve the conditional fit. It does not show that group membership caused the phenotype difference. The next diagnostic question is whether the pattern persists after reviewing covariate balance, technical factors, and the adequacy of the component adjustment.
Derivation and Interpretation of the Nested-Model Test
The partial \(F\) calculation compares the reduction in residual variation from adding terms with the residual variation left by the larger model. Suppose the reduced and full models are nested, the full model has \(q\) additional coefficients, and its residual degrees of freedom are \(df_F\). Under the null that all \(q\) added coefficients are zero, and under the classical linear-model assumptions of independent normal errors with common variance and correctly specified conditional mean, the scaled reduction and the full-model residual mean square estimate the same error variance.
The numerator is the reduction in residual sum of squares per added coefficient; the denominator is the full-model residual mean square. A large value indicates that the added terms reduce residual variation substantially relative to the remaining variation. If the errors are not normal, are dependent, have nonconstant variance, or the mean model is misspecified, the exact reference distribution need not hold. For discrete genotype dosage, the ordinary linear-model test is therefore usually treated as an approximation, or replaced by a method whose assumptions fit the response and sampling design.
Diagnostics, Alternatives, and Limitations
No single plot or \(p\)-value settles whether ancestry adjustment is adequate. Use plots to see the shape and location of patterns; use conditional tests to quantify model comparisons; and use sensitivity analyses to assess whether substantive results change. A simple linear term can miss curvature, so inspect residuals against components for bends or group-specific patterns. A significant added-component test can also be driven by a few influential observations, so inspect those observations and verify data quality before interpreting the result.
If a component trend appears nonlinear, consider whether a prespecified flexible representation is scientifically and statistically defensible. If group differences persist, check group balance in age, site, batch, and other relevant covariates. If dosage residual variance changes across fitted values or groups, do not assume the ordinary \(F\)-test is exact. Robust or response-appropriate inference may be preferable, while preserving the distinction between a diagnostic exploration and a confirmatory test.
These checks have limits. Components are estimated from genotypes, and the selected components may not capture every ancestry pattern. Group labels can obscure gradients, and small groups have limited power. A lack of significant residual association may mean the diagnostic had little power rather than that structure is absent. Conversely, a significant result may reflect an omitted technical factor or model form, not ancestry alone. Related observations can also violate independence; the present diagnostics do not substitute for an appropriate model of dependence.
The main alternatives are complementary rather than interchangeable: plots reveal shape and outliers; conditional component tests assess added linear terms; group-term comparisons test a specified categorical contrast; and sensitivity analyses compare conclusions under plausible adjustment choices. Report what was examined, the model and assumptions, and whether the evidence was exploratory or confirmatory.
Common Mistakes
- Assuming included components remove all structure. Check residual patterns and sensitivity to additional relevant terms rather than treating adjustment as a guarantee.
- Calling a dosage-model \(F\)-test exact. Dosage is discrete and bounded; ordinary linear-model inference is generally approximate for this response.
- Interpreting a group association as a causal ancestry effect. The test identifies a conditional pattern, not its cause.
- Treating a nonsignificant test as proof of no residual structure. Consider sample size, group balance, and the power of the diagnostic.
- Ignoring technical covariates. Site or batch patterns can resemble ancestry patterns; inspect the study design before attributing a residual association.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A phenotype model's residuals show a curved trend against PC2, although the linear PC2 coefficient is not significant. What should you investigate?
- A dosage regression gives a nominally significant test for an added component. Can its ordinary linear-model \(F\)-test be called exact? Why or why not?
- A group-term comparison is significant after adjustment for two components. What does this establish, and what does it not establish?
- A residual group comparison is not significant in a study with a very small group. What is the appropriate interpretation?
Solutions. (1) Inspect the residual plot for curvature and consider whether the linear mean specification is inadequate; do not rely only on the linear coefficient test. (2) Generally, no. Dosage takes discrete values 0, 1, and 2, so its regression errors are not exactly normally distributed; the conventional test is an approximation unless a suitable model and inference justify another reference distribution. (3) It establishes evidence that the group terms improve the conditional fit under the model and test assumptions. It does not establish that ancestry caused the phenotype pattern or rule out omitted technical covariates. (4) The data provide insufficient evidence in that test to establish a group difference. Low power is plausible, so the result is not proof that residual structure is absent.