Choosing K in Ancestry Clustering

Why Choosing K Requires More Than One Number

In model-based ancestry clustering, \(K\) is the number of ancestral components represented by the fitted model. Choosing \(K\) means deciding how many components are useful for describing the genotype data under that model. A larger \(K\) gives the model more flexibility, so its likelihood on the data used to fit it will generally not decrease. That alone is not evidence that the larger value is preferable: added flexibility can fit noise or small, sample-specific features.

Three kinds of evidence are commonly compared: prediction of genotypes that were not used for fitting, likelihood criteria that penalize model complexity, and reproducibility across independent runs. They answer related but different questions. A defensible choice considers how they agree, whether the data and model assumptions make the comparison meaningful, and what scientific purpose the components are meant to serve.

For orientation, let \(g_{il}\) be the genotype count, 0, 1, or 2, for individual \(i\) at locus \(l\). Write \(q_{ik}\) for individual \(i\)’s proportion assigned to component \(k\), and \(f_{kl}\) for the allele frequency at locus \(l\) in component \(k\). The proportions satisfy \(q_{ik}\geq 0\) and \(\sum_{k=1}^{K}q_{ik}=1\). In a simple model, the allele probability for an individual at a locus is \(p_{il}=\sum_{k=1}^{K}q_{ik}f_{kl}\), and genotype probabilities are modeled using that probability under the model’s assumptions. The fitted likelihood summarizes how well the model represents the observed genotypes. Different software can use different likelihoods, estimation procedures, and treatment of genotype data, so comparisons should be made within a consistent analysis setup.

Cross-Validation: Predicting Genotypes Left Out of Fitting

Cross-validation assesses how well a fitted model predicts data not used to estimate it. In ancestry clustering, one common design holds out some genotype observations, fits the model using the remaining observations, and scores the held-out genotypes using their predictive probabilities. The held-out observations must not be used to estimate the parameters used for their own prediction.

A convenient score is the mean negative log predictive probability, sometimes called held-out log loss. Lower values indicate better predictive performance. If \(H\) is the set of held-out genotype observations and \(\hat{P}_{-H}(g_{il})\) is the predictive probability estimated without those observations, the score is:

For multiple folds, each observation is held out in one fold, and scores are combined across held-out observations. This score is an estimate of predictive performance on data like those held out, not a direct test of whether the model’s components correspond to real populations. The estimate is most relevant when the held-out data resemble the data on which the model will be used.

The design of the folds matters. If the purpose is prediction at new loci, holding out loci is relevant; if the purpose is prediction for new individuals, holding out individuals may be more relevant. Linkage between nearby markers, relatives appearing in different folds, and uneven representation of groups can all affect the score. State the validation design and keep it consistent across candidate values of \(K\).

Likelihood Criteria: Fit Balanced Against Complexity

The maximized likelihood cannot by itself choose \(K\), because a more flexible model can usually achieve at least as high a likelihood on the fitted data. The Akaike information criterion (AIC) and Bayesian information criterion (BIC) penalize the maximized likelihood according to the number of fitted parameters. Smaller values are preferred within the set of models being compared.

Both criteria trade fit against complexity, but they use different penalties. AIC adds \(2d\); BIC adds \(d\log(n)\). When \(n\) is sufficiently large, BIC penalizes each additional parameter more heavily. These criteria are not probabilities that a model is true, and neither guarantees that the chosen model has a literal population interpretation.

The parameter count and sample size require care in genetic data. Count only free parameters: for example, proportions constrained to sum to one do not contribute \(K\) independent parameters per individual. Also, genotype observations across linked loci may not act like independent observations. The appropriate \(n\) in BIC depends on the likelihood and its sampling assumptions; using the raw count of genotype cells without justification can give a misleading comparison. Criteria are meaningful only when likelihoods are calculated for comparable data under a consistent model.

Run-to-Run Stability

Many ancestry models use iterative or stochastic algorithms, so different initial values or random seeds can lead to different fitted solutions at the same \(K\). Repeating the fit helps distinguish a reproducible pattern from one that appears only under a particular run. Useful comparisons include the maximized likelihood across runs and how similar the estimated ancestry proportions are after corresponding components have been matched.

A practical comparison may summarize the average pairwise correlation of individuals’ ancestry-proportion profiles across runs, or the variation in estimated proportions for individuals of interest. Component labels can be permuted between runs without changing the fitted model, so comparisons must match corresponding components before measuring agreement. This matching is a bookkeeping step; a low raw correlation before matching need not mean that the solutions differ substantively.

When comparing stability across candidate values, use several runs per \(K\), the same data and settings, and consistent summaries. A stable solution can still reflect a model that is too simple, systematic sampling features, or limitations of the model. Conversely, poor stability can indicate that the data do not strongly distinguish alternative fits at that \(K\), rather than proving that the value should never be used.

What K Does—and Does Not—Mean

A selected \(K\) describes the resolution at which a particular model and comparison procedure represent the data. It need not equal a count of discrete, isolated populations. Human genetic variation may include admixture, shared ancestry, geographic gradients, and other patterns that do not divide neatly into separate groups. A clustering model can approximate such variation using components even when no sharp biological boundaries exist.

For that reason, a component should not automatically be named as a specific population just because its estimated allele frequencies or ancestry proportions distinguish it in a fitted model. Interpretation requires relevant scientific context and careful attention to how the sample was collected and which individuals and markers were included. Choosing \(K\) answers a model-comparison question; it does not by itself settle a taxonomic, historical, or causal question.

A Practical Comparison Strategy

Common Mistakes

• Choosing the largest fitted likelihood. The likelihood generally improves as flexibility increases. Compare predictive performance or use an explicitly penalized criterion rather than treating the maximum as decisive.
• Calling the cross-validation minimum “the true K.” It is the best score among the tested candidates under a particular validation design. It is not proof of a unique number of biological populations.
• Ignoring assumptions behind BIC. The choice of \(n\) and the likelihood’s independence assumptions matter. Linked genotype observations cannot automatically be counted as independent contributions.
• Treating stability as proof of correctness. Reproducibility across runs shows that an algorithm repeatedly finds similar fits; it does not establish that the model is adequate or that its components are discrete populations.
• Comparing component labels without matching them. Component names are arbitrary across runs. Match corresponding components before comparing their estimated proportions.