Why Fit and Stability Need Separate Checks
An ancestry model can return plausible-looking proportions and still be unreliable. A different initialization might produce substantially different estimates, a small change in sampled individuals or genomic regions might alter the components, or the fitted model might systematically predict the wrong genotypes. These are related but distinct concerns: stability asks whether estimates change under reasonable perturbations of the analysis, while fit asks whether the model describes or predicts the observed genotype data adequately.
A check of one kind does not replace the others. Repeated runs can converge to similar estimates even when the model fits poorly. Conversely, a model may predict genotypes reasonably well while giving uncertain ancestry proportions for particular individuals. The diagnostics below therefore examine reproducibility, sampling sensitivity, residual patterns, and prediction on genotypes excluded from the relevant estimation step.
Model and Diagnostic Quantities
For a precise discussion, consider \(n\) diploid individuals at \(L\) biallelic loci and a model with \(K\) ancestry components. Let \(g_{il}\in\{0,1,2\}\) be the allele count for individual \(i\) at locus \(l\), \(q_{ik}\) the estimated proportion of individual \(i\)'s ancestry assigned to component \(k\), and \(f_{kl}\) the allele frequency for component \(k\) at locus \(l\). The proportions satisfy \(q_{ik}\geq 0\) and \(\sum_{k=1}^{K}q_{ik}=1\), while \(0\leq f_{kl}\leq 1\).
In the genotype-count model used here, the allele probability implied for individual \(i\) at locus \(l\) is \(p_{il}=\sum_{k=1}^{K}q_{ik}f_{kl}\). Conditional on the estimated parameters, the model treats the two allele copies as independent draws with probability \(p_{il}\), and treats loci as independent for the likelihood calculation. This gives \(g_{il}\sim\operatorname{Binomial}(2,p_{il})\). In practice, linkage disequilibrium, relatedness, model misspecification, and parameter estimation can all affect the diagnostics; independence is an approximation, not a guarantee.
The fitted allele probability gives an expected genotype count of \(2p_{il}\). A raw residual compares the observed count with that expectation. Under the stated binomial model, the genotype-count variance is \(2p_{il}(1-p_{il})\), which motivates a standardized residual when the variance is nonzero.
The standardized residual \(z_{il}\) expresses the discrepancy in units of the model-based genotype-count standard deviation. It is a diagnostic, not automatically a standard-normal test statistic: the parameters were estimated, residuals can be dependent, and model assumptions may fail. If \(p_{il}\) is zero or one, the formula has a zero denominator; such a prediction assigns probability zero to at least one genotype count, so the observed data and model require direct scrutiny rather than this standardization.
Replicate Runs and Component Alignment
An iterative ancestry algorithm may be sensitive to starting values. Fit the same model several times with different initializations, then compare estimates only after matching component labels. Component 1 in one run need not correspond to component 1 in another: the component numbers are labels, not fixed biological names. Matching can use similarity in the component allele-frequency profiles, ancestry-proportion profiles, or both. A permutation should be applied consistently to all parameters for a component.
Once aligned, useful summaries include the range or standard deviation of each \(q_{ik}\) across runs and differences between the \(f_{kl}\) profiles. Similar estimates across runs suggest algorithmic reproducibility for those data and settings. They do not establish that \(K\) is appropriate, that the model fits, or that the estimates are unaffected by the sampled loci or people.
Worked Example: Comparing Two Runs After Label Alignment
Suppose two hypothetical runs estimate ancestry proportions for three individuals with \(K=2\). In run B, the component labels are reversed relative to run A. After matching the components using their allele-frequency profiles, the estimates are:
| Individual | Run A: component 1 | Run A: component 2 | Aligned run B: component 1 | Aligned run B: component 2 |
|---|---|---|---|---|
| A | 0.72 | 0.28 | 0.70 | 0.30 |
| B | 0.45 | 0.55 | 0.44 | 0.56 |
| C | 0.18 | 0.82 | 0.20 | 0.80 |
For individual A, the aligned component-1 difference is \(0.72-0.70=0.02\); for B it is \(0.45-0.44=0.01\), and for C it is \(0.18-0.20=-0.02\). The component-2 differences have the same absolute values because each pair of proportions sums to one. Across all six aligned entries, the mean absolute difference is
The estimates are close across these two runs after alignment. Comparing the unaligned entries would have made the runs appear very different for the wrong reason: the component labels had switched. This comparison addresses repeatability across these initializations only; it does not check prediction or sensitivity to the data sample.
Bootstrap Resampling and Sensitivity
A bootstrap assesses how estimates vary under resampling from the observed data. Resampling individuals with replacement asks how estimates respond to variation in the sampled people. Resampling genomic blocks with replacement asks how estimates respond to variation in the sampled regions; blocks are preferable to isolated loci when nearby loci are correlated. The resampling unit should match the source of sensitivity being investigated. In each resample, refit the model and align component labels before summarizing \(q\) or \(f\).
For an individual-level summary, one might examine the bootstrap distribution of a component proportion, report its median and a percentile interval, or report the fraction of replicates in which that proportion exceeds a prespecified threshold. For population-level components, examine whether their frequency profiles and assignments recur across resamples. If components split, merge, or become difficult to match, that instability is itself informative. A bootstrap interval summarizes variation under the chosen resampling scheme and model; it does not account automatically for every source of uncertainty or correct a misspecified model.
Worked Example: Interpreting Bootstrap Variation
Suppose five hypothetical block-bootstrap refits, after label alignment, estimate component 1 for one target individual as \(0.41,\ 0.44,\ 0.43,\ 0.39,\ 0.45\). Their mean is
The estimates range from \(0.39\) to \(0.45\), a range of \(0.06\), or six percentage points. This suggests modest variation across these five particular resamples. Five resamples are far too few for a reliable percentile interval, and this range is not a confidence interval. With many resamples, a percentile interval could be calculated from the empirical quantiles, but its interpretation would still depend on the block definition, refitting procedure, and model assumptions. These values do not imply that the individual's ancestry proportion is known to within six percentage points in every relevant sense.
Residual Patterns as a Model-Fit Check
Residuals are most useful when inspected for structure rather than reduced to a single extreme value. For example, many residuals with the same sign in a group of individuals may indicate that the model systematically underpredicts or overpredicts their allele counts at certain loci. Repeated patterns across neighboring loci may also reflect linkage disequilibrium or unmodeled genomic structure. Residuals can be summarized by locus, individual, or groups defined without using the residuals themselves. A plot of \(r_{il}\) or \(z_{il}\) against locus position, predicted probability, or a relevant sample grouping can reveal patterns hidden by an overall average.
Worked Example: Calculating Genotype Residuals
For one hypothetical individual, suppose the model predicts allele probabilities \(p_{il}=0.40,\ 0.25,\ 0.60,\ 0.50\) at four loci, while the observed genotype counts are \(2,\ 0,\ 2,\ 0\). At the first locus, the expected count is \(2(0.40)=0.80\), so the residual is \(2-0.80=1.20\). Its model-based standard deviation is
At the second locus, the expected count is \(2(0.25)=0.50\), and the residual is \(0-0.50=-0.50\). The standard deviation is \(\sqrt{2(0.25)(0.75)}=\sqrt{0.375}\approx0.6124\), giving \(z\approx-0.8165\). At the third locus, the residual is \(2-2(0.60)=0.80\), and \(z=0.80/\sqrt{0.48}\approx1.155\). At the fourth locus, the residual is \(0-1.00=-1.00\), and \(z=-1.00/\sqrt{0.50}\approx-1.414\).
These four residuals alone do not show a clear systematic pattern. An isolated residual is not sufficient evidence that the model is poor; useful evidence would come from recurring discrepancies, evaluated across appropriate individuals or loci and with dependence in the genotype data considered.
Held-Out Genotypes: A Predictive Check
A held-out assessment asks whether a model predicts genotypes that were not used in the estimation step relevant to those predictions. The split must respect the fact that \(f_{kl}\) is locus-specific. If loci are held out, their component frequencies are not supplied by estimates at other loci. Using training-locus frequencies as though they were frequencies for held-out loci does not produce a valid prediction under this model.
One valid design holds out individuals while retaining the loci in the training data. Estimate component frequencies at the scoring loci using training individuals. For a held-out individual, estimate \(q\) using a separate set of genotype loci, then predict genotypes at scoring loci using that \(q\) and the component frequencies estimated from training individuals at those same scoring loci. Do not use the held-out individual's genotypes at the scoring loci to estimate that individual's \(q\). This design keeps both required ingredients available without using the scored genotypes to construct their own predictions.
Another valid design holds out loci but estimates their component frequencies using other individuals at those same loci, excluding the held-out genotypes being scored. In either design, specify which individuals and loci are used for estimating \(q\), estimating \(f\), and scoring predictions. If model choices are compared, use the same held-out data and a comparable split for each candidate model.
For observed held-out genotype counts \(g_{il}\), the binomial predictive probability is \(p_{il}^{g_{il}}(1-p_{il})^{2-g_{il}}\binom{2}{g_{il}}\). Summing the logarithms of these probabilities gives a held-out log likelihood. A larger value means better prediction for those genotypes under that scoring setup, not proof that the model is true.
Worked Example: Scoring Held-Out Genotypes Without Leakage
Consider a hypothetical held-out individual and three scoring loci. The individual's ancestry proportions were estimated using separate loci, not these scoring genotypes. Component frequencies at each scoring locus were estimated from training individuals. Combining those quantities gives \(p=0.50,\ 0.50,\ 0.40\) at the three scoring loci. The held-out genotype counts are \(0,\ 2,\ 1\), respectively.
The probabilities of the observed counts are
Under the conditional independence assumption across these loci, the joint probability of the observed three-genotype pattern is \(0.25(0.25)(0.48)=0.0300\). Its log likelihood is \(\log(0.0300)\approx-3.507\). As a comparison, suppose another candidate model, fitted using the same training design, predicts \(p=0.30,\ 0.70,\ 0.50\). Its probabilities for the same observed genotypes are \(0.70^2=0.49,\ 0.70^2=0.49,\) and \(2(0.50)(0.50)=0.50\). The product is \(0.12005\), with log likelihood \(\log(0.12005)\approx-2.120\). The second candidate predicts this small held-out pattern better, but three genotypes are not enough to establish broad superiority. Repeated held-out splits and careful attention to linkage and sample design are needed for a more informative comparison.
A Short Justification for the Residual Scale
The standardized residual uses the variance implied by the model, rather than treating a one-allele discrepancy as equally surprising at every predicted frequency. To see the scale, write \(G=A_1+A_2\), where the two allele indicators are independent Bernoulli random variables with probability \(p\). Each has mean \(p\) and variance \(p(1-p)\). Therefore
The variance addition follows from the model's conditional independence of the two allele copies. Thus the denominator in \(z_{il}\) is the model-implied standard deviation of the genotype count. If allele copies are not independent under the chosen genotype model, this variance justification does not apply as written.
Common Pitfalls and Practical Interpretation
- Comparing component numbers without alignment. A label switch can make equivalent runs look different. Match the components before computing differences.
- Treating repeatability as model validation. Similar runs show algorithmic reproducibility, not necessarily good fit or a suitable value of \(K\).
- Using held-out locus frequencies that were never estimated. A locus-specific frequency at a scored locus must be estimated from appropriate data at that locus, without using the held-out genotype being scored.
- Calling every large residual a formal rejection. Residuals are diagnostics whose scale and dependence rely on model assumptions. Look for patterns and account for estimation and linkage rather than applying an automatic threshold.
- Interpreting bootstrap spread as total uncertainty. A bootstrap reflects the specified resampling scheme; it does not automatically include uncertainty from model choice, reference-panel mismatch, or assumptions not represented in the resampling.
Together, aligned runs, carefully chosen bootstrap resampling, residual inspection, and held-out prediction provide complementary evidence about an ancestry model. Report the design and the limits of each check. The central question is not whether one diagnostic declares the model “good,” but whether estimates and predictions are sufficiently reproducible and adequate for the intended analysis.
Check Your Understanding
Answer each question using the model assumptions and diagnostics described above.
- Why must component labels be aligned before calculating differences between replicate ancestry-proportion estimates?
- What source of sensitivity is examined by resampling genomic blocks rather than individuals?
- If \(p_{il}=0.30\) and the observed genotype count is \(g_{il}=2\), calculate the raw residual and the standardized residual under the binomial model.
- In a held-out-individual design, why should an individual's ancestry proportions be estimated using loci separate from the scoring loci?
- Does a higher held-out log likelihood prove that one ancestry model is biologically true? Explain what it does support.