Flexible Adjustment for Nonlinear Ancestry Effects

Why Allow Ancestry Effects to Be Nonlinear?

Principal components (PCs) summarize major axes of genetic variation. In association regression, adjusting for selected PCs can reduce confounding when ancestry is related to both genotype and phenotype. A common model enters each PC as a linear term. That is a useful starting point, but it imposes a particular shape: for a given PC, the model assumes the conditional mean phenotype changes at a constant rate across the PC’s range.

The relationship may instead curve or change slope. If the true ancestry–phenotype relationship is nonlinear, a linear PC term may leave systematic variation unmodeled. A spline represents that relationship using several smooth basis functions, allowing the fitted mean to bend while retaining a regression framework. The purpose is not to make adjustment arbitrarily complicated: it is to model plausible curvature and examine whether the genetic association estimate is sensitive to that choice.

This tutorial focuses on a quantitative phenotype and an additive genotype dosage. The model and comparison below concern the mean phenotype conditional on genotype, PCs, and other included covariates. They do not by themselves establish a causal genetic effect.

Linear and Spline-Adjusted Models

Definition: Let $Y_i$ be the quantitative phenotype for individual $i$, $G_i$ the genotype dosage coded as 0, 1, or 2 copies of a specified allele, $C_{ij}$ the value of PC $j$, and $Z_i$ a vector of other covariates. A linear-PC model can be written $$ Y_i=\beta_0+\beta_G G_i+\sum_{j=1}^{J}\theta_j C_{ij}+\boldsymbol{\gamma}^{T}Z_i+\varepsilon_i. $$ A spline-PC model replaces each linear PC term with a function represented by spline basis terms: $$ Y_i=\beta_0+\beta_G G_i+\sum_{j=1}^{J} f_j(C_{ij})+\boldsymbol{\gamma}^{T}Z_i+\varepsilon_i. $$

In both models, $\beta_G$ is the adjusted mean phenotype difference for a one-copy increase in genotype dosage, conditional on the included ancestry terms and covariates. Its units are phenotype units per allele copy. This interpretation assumes an additive dosage effect and a common genotype effect across the modeled PC values; the models above do not include genotype-by-PC interactions.

A spline function $f_j$ is a weighted sum of known basis functions of PC $j$. The weights are estimated from the data. One transparent option is a cubic regression spline. For one PC value $c$ and internal knots $k_1,\ldots,k_K$, one basis uses

f(c)=a_1c+a_2c^2+a_3c^3+\sum_{r=1}^{K}d_r(c-k_r)_+^3, \qquad (u)_+=\max(u,0).

The polynomial terms allow curvature, and each truncated-cubic term adds flexibility after its knot. The function and its first two derivatives remain continuous at each knot, so the fitted curve does not jump or acquire a sharp corner there. In practice, software often uses a numerically stable reparameterization of a spline basis; the fitted function space, rather than a particular coding of its columns, is the central idea.

Knots are chosen within the observed PC range, often at prespecified quantiles. Their number and locations control flexibility. Too few may fail to represent important curvature; too many can produce an unstable fit, particularly where few individuals are observed. With several PCs, an additive model gives each PC its own spline function. It does not automatically represent interactions between PCs.

Worked Example: Evaluating a Cubic Spline Basis

Suppose a PC has been centered and scaled, and a cubic regression spline uses internal knots at $-0.5$ and $0.5$. For an individual with PC value $c=0.6$, calculate the polynomial and truncated-cubic basis values.

The polynomial terms are $c=0.6$, $c^2=0.6^2=0.36$, and $c^3=0.6^3=0.216$. The first truncated term is $(0.6-(-0.5))_+^3=1.1^3=1.331$. The second is $(0.6-0.5)_+^3=0.1^3=0.001$. Thus the basis values are $0.6,\ 0.36,\ 0.216,\ 1.331,$ and $0.001$.

For illustration, suppose the estimated weights for these terms are $1.2,\ -0.4,\ 0.15,\ -0.08,$ and $0.3$. The fitted ancestry contribution from these terms, excluding the model intercept, is $$ 1.2(0.6)-0.4(0.36)+0.15(0.216)-0.08(1.331)+0.3(0.001) =0.72-0.144+0.0324-0.10648+0.0003 =0.50222. $$

If the phenotype is measured in milligrams per liter, this contribution is in milligrams per liter. It is one component of the fitted conditional mean, not a standalone estimate of an individual’s phenotype.

Assumptions and Interpretation

A flexible PC model still relies on a statistical model. For the usual linear-regression interpretation, the conditional mean of $Y_i$ must be represented adequately by the chosen genotype, spline, and covariate terms. The residuals $\varepsilon_i$ are assumed to have conditional mean zero. Classical standard errors and the exact partial $F$ test described below additionally assume independent errors with common variance; normal errors give the exact finite-sample test distribution. With large samples, Wald inference is often based on an approximate normal distribution, and other standard-error methods may be used when their assumptions are justified.

The genotype term is assumed to act linearly in dosage, and the model assumes that this effect does not vary with ancestry unless such interactions are added. The PCs and covariates must also be measured and included appropriately. A spline can accommodate a nonlinear mean relationship with the PCs in the model; it cannot correct confounding due to omitted ancestry dimensions, an inappropriate covariate set, relatedness that requires a different model, or other violations.

For a given PC, a linear model is a special case of the cubic spline model: set the coefficients of the nonlinear basis terms to zero. Thus, with the same individuals and other covariates, the spline model can be compared with the linear-PC model as a larger nested regression. The spline fit can capture curvature, but it also uses more degrees of freedom and may have less precise coefficient estimates.

Worked Example: Comparing Genotype Estimates

Consider a hypothetical sample of 2,400 unrelated participants with a quantitative phenotype measured in milligrams per liter. Both regressions use the same individuals, genotype dosage, selected PCs, and clinical covariates. A linear-PC fit reports a genotype estimate of $0.18$ milligrams per liter per allele copy with $SE=0.055$. A spline-PC fit reports $0.11$ milligrams per liter per allele copy with $SE=0.050$. Assume the reported standard errors are appropriate and use large-sample two-sided Wald inference.

For the linear-PC fit, the Wald statistic is $$ z=\frac{0.18}{0.055}=3.273. $$ The corresponding two-sided normal-approximation p-value is approximately $0.0011$. A 95% Wald confidence interval is $$ 0.18\pm1.96(0.055)=0.18\pm0.1078=(0.0722,\ 0.288). $$

For the spline-PC fit, the statistic is $$ z=\frac{0.11}{0.050}=2.200, $$ with two-sided p-value approximately $0.0278$. Its 95% Wald confidence interval is $$ 0.11\pm1.96(0.050)=0.11\pm0.0980=(0.0120,\ 0.208). $$

In these hypothetical fits, allowing nonlinear PC effects changes the estimated conditional phenotype difference from $0.18$ to $0.11$ milligrams per liter per allele copy. Both intervals exclude zero at the 95% level, but the estimate and its uncertainty depend on the ancestry model. This sensitivity is a reason to examine the PC relationship and report the adjustment used. It does not prove that the spline estimate is unbiased, nor does it show that the association is causal.

Testing Whether the Added Spline Terms Improve Fit

A nested-model test can assess whether the nonlinear terms improve fit beyond a linear PC term. For one PC represented by the cubic basis above with $K$ internal knots, the spline model adds $K+2$ terms beyond the linear $c$ term: $c^2$, $c^3$, and the $K$ truncated-cubic terms. The null hypothesis is that all these added coefficients are zero. For several PCs, the test can jointly assess all nonlinear terms added to the corresponding linear-PC model.

Theorem: Let $RSS_L$ and $RSS_S$ be the residual sums of squares from the linear-PC and spline-PC models fitted to the same $n$ observations. Let the spline model add $q$ parameters and estimate $p_S$ parameters in total, including its intercept. Under the null hypothesis that all added terms have coefficient zero, and under the normal linear-model assumptions of independent, equal-variance errors, $$ F=\frac{(RSS_L-RSS_S)/q}{RSS_S/(n-p_S)} $$ has an $F$ distribution with $q$ and $n-p_S$ degrees of freedom.

The numerator measures the reduction in residual variation per added parameter. The denominator estimates the residual variance in the larger model. A large value of $F$ is evidence against the null that the added spline terms are all unnecessary, conditional on the specified model and assumptions. This test concerns improvement in fit; it is not a direct test that the genotype association is confounded, and it does not establish that the larger model is the best possible ancestry adjustment.

Worked Example: Testing Added Curvature

In a hypothetical sample of $n=800$, the linear-PC model has residual sum of squares $RSS_L=760$. A cubic spline with one internal knot has $RSS_S=748$, adding $q=3$ nonlinear terms. Suppose the spline model has $p_S=11$ total estimated parameters.

The residual degrees of freedom in the spline model are $800-11=789$. Substitution into the partial $F$ statistic gives $$ F=\frac{(760-748)/3}{748/789} =\frac{12/3}{0.9480} =\frac{4}{0.9480} =4.219. $$

The reference distribution is $F(3,789)$, giving a p-value of approximately $0.0058$. Under the stated regression assumptions, this is evidence that the added nonlinear PC terms improve fit over a linear PC term. The result alone does not tell us whether the genotype coefficient changed substantially; that comparison requires examining the genotype estimates and their uncertainty in both models.

Practical Model Comparison

Keep the comparison fair.
Fit both models on the same individuals with the same genotype coding and non-PC covariates. Otherwise, a changed genotype estimate could reflect a changed analysis sample or covariate set rather than spline flexibility.

Specify the spline deliberately.
Choose a modest number of knots within the observed PC range. Check that the basis can be estimated stably, especially near the extremes where data may be sparse.

Compare estimates and fit.
Report the genotype estimate, standard error or confidence interval from each model. A nested-model test can quantify evidence for nonlinear terms, but should be interpreted alongside the effect-estimate comparison and model assumptions.

A spline model is not automatically preferable just because it is more flexible or has a smaller residual sum of squares; adding parameters cannot increase ordinary least-squares residual sum of squares. The practical question is whether the added flexibility is supported, stable, and relevant to controlling ancestry-related structure without obscuring the association of interest. The same knot choice and adjustment strategy should be applied consistently rather than selected solely to produce a preferred association result.

Common Mistake

Wrong reasoning: “The spline-adjusted genotype estimate is smaller, so the linear model was definitely confounded and the spline estimate is the true genetic effect.”
Correction: A changed estimate shows sensitivity to the model for ancestry effects. It is compatible with residual nonlinear confounding under the linear model, but it can also reflect sampling variability, unnecessary flexibility, or model misspecification. Neither estimate is guaranteed to be unbiased merely because of its form. Interpret the comparison with the design, covariates, PC distributions, and uncertainty in view.

A related pitfall is to interpret a small p-value for the spline terms as evidence that the genotype association is real. The spline-term test asks whether the modeled phenotype relationship with ancestry is better represented by added curvature. The genotype coefficient’s test asks a different question: whether the adjusted phenotype association with dosage differs from zero under that fitted model.

Key takeaway: Spline terms let PC-adjusted regression represent smooth nonlinear ancestry–phenotype relationships. Compare spline and linear adjustments on the same sample, examine the genotype-effect estimates and uncertainty, and treat a change in association as a modeling sensitivity—not proof that either estimate is the truth.

Check Your Understanding

Try these questions without looking back.

In the spline-PC model, what does the genotype coefficient represent, and what units should it have?
For a cubic regression spline with two internal knots, how many terms are added beyond a linear PC term in the basis shown here?
Why must the linear-PC and spline-PC models use the same analysis sample for a direct comparison of their genotype estimates?
What null hypothesis is tested by the partial $F$ test for added spline terms?
A spline-adjusted genotype estimate differs from the linear-adjusted estimate. Give one interpretation that is justified and one conclusion that is not justified by that difference alone.