Continuous Ancestry Models Versus Population Labels

Why the Representation of Ancestry Matters

An ancestry proportion and a population label describe related but different information. A proportion records gradations: for example, an estimated genome-wide ancestry fraction of 0.35 differs from 0.65. A label assigns an individual to a category, such as a cluster or a study-defined population group. Replacing proportions with labels may make a model easier to describe, but it discards differences among people assigned to the same category.

That loss matters in association analysis when ancestry is related both to genotype dosage and to the outcome. A label may control broad differences between groups while leaving a residual association between dosage and ancestry within each group. Conversely, a continuous linear term may be too restrictive if the relationship with ancestry is nonlinear. Neither representation is automatically correct: the choice depends on the scientific meaning of the measure, its relationship with the variables, and whether the model fits adequately.

Two Regression Representations

Let \(Y_i\) be an outcome for individual \(i\), \(X_i\) a genetic dosage, and \(A_i\) a continuous ancestry proportion between 0 and 1. For a quantitative outcome, a simple continuous-ancestry model is

Here, \(\beta_X\) is the expected change in outcome units per one-unit increase in dosage, conditional on the ancestry proportion. The coefficient \(\gamma\) describes the fitted outcome difference for a change of 1.0 in the ancestry proportion, holding dosage fixed. This is a linear conditional-mean model: it assumes the association of the outcome with \(A_i\) can be represented by a straight line, and that the dosage slope is constant across ancestry values. Other prespecified covariates can be added.

For \(K\) population labels, a label-adjusted model can instead be written as

where \(L_i\) is the label and \(I(L_i=k)\) equals 1 when individual \(i\) has label \(k\), and 0 otherwise. The omitted label is the reference. The \(\delta_k\) terms permit different group intercepts. This specification assumes a common dosage slope across labels and treats individuals within each label as having the same fitted ancestry contribution to the mean outcome.

For standard coefficient tests and confidence intervals, a common set of regression conditions is needed: the conditional mean is correctly specified, observations are independent (or dependence is handled in the variance estimate), and there is adequate variation in dosage after adjustment. Normal errors with constant variance support exact small-sample \(t\)-inference; with larger samples, robust standard errors can address some variance misspecification, but do not repair a wrong conditional mean or dependence that was ignored. Ancestry proportions are estimates, so substantial measurement error can also weaken or distort adjustment.

What Label Adjustment Removes—and What It Leaves

The distinction can be made precise by considering a model with one continuous ancestry measure. For clarity, suppose the models include an intercept and no other covariates. Let \(S_{UV}=\sum_i(U_i-\bar U)(V_i-\bar V)\) denote a centered cross-product. In the continuous model, the dosage coefficient is

The numerator and denominator are the cross-product and sum of squares remaining after each variable is linearly adjusted for ancestry. With additional covariates, the same interpretation applies after first residualizing the relevant variables with respect to those covariates.

In the label model, the corresponding calculation first subtracts each label-specific mean. Write \(X_i-\bar X_{L_i}\) for the dosage deviation from its label mean, and define the analogous deviations for \(Y_i\) and \(A_i\). The label-adjusted dosage slope is the within-label cross-product of dosage and outcome divided by the within-label dosage sum of squares. If the conditional mean contains a continuous ancestry term, then within labels the outcome has a component \(\gamma(A_i-\bar A_{L_i})\). Thus the label-adjusted slope contains the term

This term vanishes only if the within-label dosage and ancestry deviations have zero cross-product (or if \(\gamma=0\)). A label can therefore remove between-label differences without removing within-label ancestry confounding.

Worked Comparison: Categorizing a Smooth Gradient

Choosing, Checking, and Comparing the Models

A useful comparison begins with what the variable represents. A continuous proportion is appropriate when the scientific question concerns variation along an ancestry gradient and the estimates are sufficiently informative. A label may be appropriate when the categories correspond to the study's sampling strata or when the estimand is explicitly a within-category association. A label is not made more informative simply by being familiar or easy to communicate.

For multiple ancestry proportions \(A_{i1},\ldots,A_{iK}\) that sum to 1, including all \(K\) proportions with an intercept causes exact collinearity. Use \(K-1\) proportions with a reference component, or another identifiable parameterization. This is distinct from replacing proportions by a label: a \(K\)-category label also needs a reference category, but it represents group membership rather than a full quantitative composition.

Diagnostics should target the assumptions that distinguish the representations. For a continuous model, inspect outcome residuals across the range of each ancestry proportion and check whether a straight-line conditional mean is plausible. For a label model, examine ancestry-proportion distributions within each label and check for residual outcome patterns across those within-label values. In either model, assess dosage variation within ancestry strata, influential observations, and whether the fitted design matrix is nearly collinear. Very little dosage variation after adjustment makes the dosage coefficient imprecise even if the model is otherwise reasonable.

A practical alternative is to include both ancestry proportions and label indicators, if the design and data support estimating their separate contributions. This can accommodate broad category differences while adjusting for within-label gradients, but it can also increase collinearity and uncertainty. A flexible nonlinear function of the proportion may be preferable to arbitrary categorization when a linear trend is implausible and data cover the range adequately. Each extension requires enough observations and dosage variation to estimate its parameters.

Common Pitfall and Limitations

• Wrong reasoning: “The model includes population labels, so ancestry confounding is removed.” Correction: Labels adjust between-label means; they do not automatically adjust ancestry variation within a label. Check within-label covariance and residual patterns.
• Wrong reasoning: “A continuous proportion must be better because it contains more numerical detail.” Correction: A misspecified straight-line relationship, uncertain proportions, or poor overlap can make a simple continuous model inadequate. Evaluate fit and measurement quality.
• Wrong reasoning: “The adjusted dosage coefficient is a causal genetic effect.” Correction: Regression adjustment defines a conditional association under the model. Causal interpretation requires additional design and identification assumptions.

Both representations have limits. Labels depend on how groups were defined and can hide diversity within categories; continuous proportions depend on estimation and can be difficult to interpret as a single gradient when ancestry is multidimensional. Either model may have weak overlap in ancestry or dosage across the sample, limiting what comparisons the data support. Report the ancestry representation, coding, adjustment set, and diagnostics so that the coefficient's meaning is explicit.