Diagnosing Residual Population Structure

Why Diagnose Structure After Adjustment?

A model can include principal components, ancestry groups, or a genetic relatedness term and still leave ancestry-related variation unexplained. The practical question is not simply whether an adjustment was included, but whether the fitted model has removed the outcome patterns that could create spurious genetic associations. Residual diagnostics address that question by looking for systematic structure left over after fitting.

No single residual plot establishes that confounding is present or absent. A pattern can arise from an omitted ancestry effect, but also from model misspecification, sampling variation, or genuine genetic effects. Conversely, a quiet-looking diagnostic does not prove that every tested variant is free of confounding. The goal is to find coherent evidence that warrants checking the adjustment strategy and the association results.

What Residuals Can Reveal

For a quantitative phenotype, a basic adjustment model might be \(y_i=\beta_0+\mathbf{c}_i^\mathsf{T}\boldsymbol{\beta}+\varepsilon_i\), where \(y_i\) is the outcome for person \(i\), \(\mathbf{c}_i\) contains included covariates and chosen structure terms, and \(\varepsilon_i\) is the remaining error. The fitted residual is \(e_i=y_i-\hat{y}_i\). Under a correctly specified conditional-mean model, residuals should not show a systematic mean trend with an omitted ancestry feature, after accounting for the included covariates. Random scatter is expected; a smooth trend or consistent group differences are reasons to investigate.

For a binary outcome, subtracting a fitted probability gives a raw residual \(y_i-\hat{p}_i\), where \(y_i\) is 0 or 1 and \(\hat{p}_i\) is the model-fitted probability. The Pearson residual is \(r_i=(y_i-\hat{p}_i)/\sqrt{\hat{p}_i(1-\hat{p}_i)}\). Comparing observed and expected case counts across ancestry groups is often easier to interpret than inspecting individual binary residuals. A group with more cases than expected contributes positive residuals; a group with fewer contributes negative residuals.

The choice of residual matters. A raw residual plot for a binary outcome is constrained by the 0/1 observations and can be hard to read, so fitted probabilities, grouped observed-versus-expected counts, or Pearson residual summaries are more informative. In mixed models, residuals also reflect the assumed covariance. For a model with genetic random effects, marginal residuals are correlated according to the fitted covariance structure. They should not be treated as independent observations; use diagnostics that account for that covariance, such as model-based residual checks or appropriately whitened residuals.

Why an Omitted Ancestry Trend Appears in Residuals

A short linear-model calculation explains the diagnostic logic. Suppose the true quantitative-outcome model includes a matrix \(Z\) of ancestry features that was left out of the fitted model:

Here \(C\) contains the intercept and covariates that were fitted, and the errors have conditional mean zero given \(C\) and \(Z\). Let \(M_C=I-C(C^\mathsf{T}C)^{-1}C^\mathsf{T}\), assuming \(C\) has full column rank. The residuals from regressing \(y\) on \(C\) are \(r=M_Cy\). Substituting the true model gives:

because \(M_C C=0\). Therefore, if the omitted ancestry features still vary after removing their linear relationship with the included covariates, they contribute a systematic component to the residuals. This does not mean every residual trend is confounding: the derivation assumes the stated linear mean model, and random errors can produce apparent patterns in a finite sample. It does explain why coherent residual associations with ancestry are a warning signal.

Check Calibration Across Ancestry Groups

A group-level check asks whether the model predicts outcomes comparably across ancestry groups after adjustment. For a binary phenotype, add the fitted probabilities within each group to obtain the expected case count, then compare it with the observed count. This is a calibration diagnostic: it asks whether predicted risks correspond to observed frequencies in those groups. It is not, on its own, a test of whether an individual genetic association is causal or confounded.

Groups can differ in sample size, covariate distributions, and fitted risk. Therefore, comparing raw case counts alone is not useful. Compare observed counts with the group-specific expected counts from the fitted model, and interpret differences in light of their uncertainty. A consistent excess in one ancestry group and a deficit in another can indicate residual outcome structure, especially when it is also visible along PCs or among association statistics.

Look for Coherent Patterns Across the Genome

Residual structure can also appear in the association results. After fitting the chosen structure adjustment at each variant, examine whether association statistics are systematically larger in genomic regions or variant sets that share ancestry-related properties. For example, variants with large allele-frequency differences across ancestry groups may be especially sensitive to residual population structure. A concentration of elevated statistics in regions whose variants are strongly related to ancestry can therefore be informative.

The comparison should be descriptive and carefully constructed. Linkage disequilibrium makes nearby variants correlated, so thousands of SNPs in one region are not thousands of independent checks. Summaries can use LD-pruned variants or region-level summaries, with the dependence and the number of regions considered in interpretation. Also compare relevant variant properties where possible: allele frequency, imputation quality, and sample size can differ across sets and affect statistics for reasons unrelated to residual structure.

An elevated genome-wide distribution is not specific to confounding. A genuinely polygenic trait can produce many non-null associations, and selection of the analysis sample can alter the distribution as well. Likewise, a localized association peak may reflect a real locus rather than ancestry. Stronger diagnostic evidence comes from a pattern that aligns with ancestry—for example, elevated statistics repeatedly among ancestry-differentiated variants or across multiple regions with similar ancestry loadings—and that is consistent with residual phenotype or group-calibration patterns.

A Practical Diagnostic Sequence

Useful diagnosis combines checks at different levels rather than relying on one summary. The following sequence keeps the interpretation focused on model adequacy:

Common Mistake: Treating a Flat Plot as Proof

A common mistake is to say, “The residuals have no trend against the PCs in my adjusted model, so population structure is fully controlled.” If those same PCs were included as linear predictors, ordinary least-squares residuals are mechanically uncorrelated with them. The plot cannot independently validate the adjustment. The correction is to assess aspects not forced by the fit: group calibration, omitted or held-out ancestry features, nonlinear patterns, model-based diagnostics for mixed models, and the distribution of association statistics across relevant genomic sets.

The reverse mistake is to treat any residual group difference as proof of confounding. A finite sample can show imbalances by chance; model misspecification or genuine ancestry-specific outcome effects can also produce patterns. Interpret diagnostics together, preserve the distinction between association and causation, and avoid making model changes solely to remove a visually inconvenient result.