What Does “Ancestry Outlier” Mean?
A sample can look unusual in a principal component analysis (PCA) plot without being erroneous, belonging to a uniquely defined population, or having a particular ancestry. “Outlier” is always a comparison: the sample is poorly represented relative to a specified reference set, marker panel, and analysis procedure. Changing any of these can change the result.
Here the goal is to assess how well a chosen reference set represents samples under study. PCA gives a low-dimensional view of genetic variation, while allele-frequency comparisons provide a way to check whether differences are also present in the genotype data at particular markers. The two kinds of evidence can agree, but neither turns a statistical distance into a definitive ancestry label.
Projecting Samples Into Reference PCA
PCA finds directions that capture variation in a genotype data set. For reference comparisons, the reference samples define the centering, scaling, and principal-component axes. A sample to be assessed is then projected onto those same axes. Recomputing PCA with the sample included can rotate or shift the axes, making the comparison less direct.
For a biallelic marker \(j\), let \(g_{ij}\) be the allele dosage for sample \(i\), coded as 0, 1, or 2, and let \(p_j\) be the allele frequency estimated in the reference. One common reference-based standardization is
This standardization centers dosage at the reference expectation \(2p_j\) and scales by the expected standard deviation under a binomial dosage model. Markers with \(p_j=0\) or \(p_j=1\) have a zero denominator and cannot be used in this form. After forming the reference matrix of standardized genotypes, PCA yields loading vectors \(v_1,\ldots,v_K\). The projected score for sample \(i\) on component \(k\) is
This projection is meaningful only if the same marker coding, allele orientation, reference frequencies, and loadings are used for both the reference and the assessed sample. PCA axes can have their signs reversed without changing the analysis, but scores from separately fitted PCAs are not automatically comparable. The number of retained components \(K\) also matters: a sample can appear typical on the first two components while differing on later components.
Measuring Distance From the Reference
A plot is useful for seeing patterns, but visual judgment alone is not a reproducible outlier rule. A simple quantitative approach measures the distance of a sample’s score vector from the reference distribution. Let \(\mathbf{t}_i\) be its vector of \(K\) scores, \(\boldsymbol{\mu}_R\) the reference mean score vector, and \(\Sigma_R\) the reference score covariance matrix. The squared Mahalanobis distance is
This distance accounts for different variances along score directions and, if present, correlations among them. A large value means that the sample is far from the reference center relative to the reference’s score variation. It does not, on its own, establish a formal probability or threshold.
Under the specific model that scores are independent samples from a \(K\)-dimensional multivariate normal distribution with known mean and covariance, \(D_i^2\) has a chi-square distribution with \(K\) degrees of freedom. In practice, the mean and covariance are estimated, scores may not be normally distributed, and individuals in a reference can be related. Thus, a chi-square cutoff is best treated as a model-based guide unless its calibration is justified. Empirical cutoffs from held-out reference samples can be more appropriate, provided the held-out samples are representative and the threshold is chosen in advance.
Worked Example: A Large Distance in Two PCA Components
Suppose a hypothetical reference set has mean score vector \((0,0)\) and covariance matrix with variances 1 and 0.25 and covariance 0. An assessed sample has scores \((2.4,1.8)\). Assume for this illustration that the reference scores follow a bivariate normal distribution and that the stated mean and covariance are known.
The squared Mahalanobis distance is
For two components, the 95th percentile of a chi-square distribution with 2 degrees of freedom is approximately 5.991. Since \(18.72>5.991\), the score is unusually far from the reference under the stated model. The model-based upper-tail probability is approximately \(e^{-18.72/2}=0.0000861\). This calculation does not show that the sample has a particular ancestry; it shows that its projected position is poorly represented by this reference distribution, assuming the model is suitable.
Checking the Signal With Allele Frequencies
PCA compresses information across many markers. An allele-frequency comparison examines differences more directly at selected markers or across a defined marker set. For a biallelic marker, the allele frequency is the proportion of allele copies carrying the allele of interest. A difference between two reference groups can be summarized as \(\hat{p}_1-\hat{p}_2\). Under independent sampling of allele copies, an approximate standard error is
where \(n_1\) and \(n_2\) are the numbers of sampled allele copies. This estimated standard error is appropriate for describing uncertainty in the difference under the independent-binomial approximation. Related individuals, dependence between allele copies, or a sampling scheme that does not represent the reference population can make that approximation unreliable. The result at one marker is also not a genome-wide ancestry conclusion.
Worked Example: Comparing Allele Frequencies
In a hypothetical comparison, two reference groups each contribute 200 allele copies at a biallelic marker. The allele of interest appears 40 times in group 1 and 70 times in group 2, giving \(\hat{p}_1=40/200=0.200\) and \(\hat{p}_2=70/200=0.350\). Assume allele copies are sampled independently within each group for this approximate calculation.
The estimated difference is \(0.200-0.350=-0.150\), or a 15-percentage-point lower frequency in group 1. Its estimated standard error is
The standardized difference from zero is \(-0.150/0.04402\approx -3.408\). Under a normal approximation, the two-sided p-value is about 0.00065. This is evidence against equal allele frequencies at this marker under the sampling assumptions. It is not the probability that the null hypothesis is true, and it does not by itself establish a genome-wide difference or explain why the groups differ.
Combining Evidence Without Overstating It
PCA and allele-frequency comparisons answer related but distinct questions. PCA indicates whether a sample’s overall pattern, as summarized by chosen components, is unusual relative to a reference. Frequency comparisons show where allele frequencies differ and can help investigate whether a PCA displacement corresponds to broad genetic differences or is driven by a narrower set of markers. Agreement across several well-chosen checks strengthens the evidence that the reference is incomplete for the sample; it still does not turn the label into an ancestry diagnosis.
| Approach | What it summarizes | Main caution |
|---|---|---|
| Projected PCA scores | Overall variation along selected reference-derived axes | Results depend on the reference, markers, scaling, and retained components |
| Allele-frequency contrasts | Differences at specified markers or marker sets | A few markers may not represent genome-wide patterns; uncertainty depends on sampling assumptions |
| Visual inspection alone | Patterns apparent in a score plot | Can miss outliers in higher dimensions and does not define a reproducible cutoff |
A useful diagnostic sequence is to verify that the projected sample and reference use compatible marker alleles and coding; inspect its position across several components rather than only a two-dimensional plot; compare its distance with the spread and shape of reference scores; and examine whether frequency differences are coherent across markers. If the score distribution is skewed or contains subgroups, a single ellipsoidal model may be misleading. If many reference samples are themselves outlying, the chosen reference may be heterogeneous or unsuitable for a single center-and-distance summary.
Other explanations for an unusual position include a reference panel that does not represent the sample’s genetic background, an intermediate or mixed profile, marker selection effects, or technical differences between data sets. A sample can also be unusual because it lies along a continuous gradient rather than belonging to a discrete cluster. A distance statistic cannot distinguish these explanations by itself. Investigators should report the reference set, marker and projection choices, components assessed, distance rule, and any threshold used.
Why Outlier Labels Need Limits
An outlier rule is a screening device, not a universal biological boundary. A threshold based on a normal model can flag valid samples when the reference is non-normal; an empirical threshold can fail when its calibration set is too small or unrepresentative. A sample may be far from the reference on a component omitted from a plot, while a two-dimensional plot can make a non-outlier look isolated because of its display scale. These are reasons to inspect the assumptions and data, not reasons to automatically discard flagged samples.
Reference choice also determines the meaning of the comparison. A sample can be an outlier relative to a narrow reference set but lie within a broader one. Conversely, a broad reference can obscure meaningful substructure. The introductory material on admixture gradients and intermediate PCA positions establishes why an intermediate score need not indicate a discrete group; here, the practical implication is that a reference-relative outlier may reflect a gradient or a combination of genetic backgrounds.
The main pitfall is to reason, “This point lies outside the reference cluster, so the sample has an unusual or incorrect ancestry.” The conclusion does not follow. A defensible statement is narrower: “Under this marker set and reference-trained projection, the sample’s scores are unusually distant from the reference distribution.” Follow-up comparisons and data-quality checks are needed before interpreting the source or practical consequence of that distance.
Worked Example: Interpreting Concordant and Discordant Checks
In a hypothetical analysis, one sample has a large distance from a reference on the first four projected components. Across a preselected set of 30 common markers, its allele frequencies also differ consistently from the reference estimates. The marker coding and projection procedure have been checked, and the reference scores do not show a strong skew.
The PCA result indicates that the sample is poorly represented by the reference in the four-dimensional score space. The frequency comparisons provide a second, related indication that the difference is not merely an artifact of viewing one two-dimensional plot. The appropriate conclusion is that this reference may not adequately represent the sample’s genetic profile under these analyses. Neither result establishes a specific ancestry, and the 30 markers do not constitute a complete explanation of the genome-wide pattern.
Now suppose instead that the sample is visually separated on the first two PCs but has an unremarkable distance when all four prespecified components are used, and the selected frequency comparisons are small relative to their uncertainty. A plot-based outlier label would be weakly supported. The discrepancy calls for checking the component selection and plot scale, not choosing whichever display gives the stronger claim.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A sample has unusual scores relative to one reference but not a broader reference. What can be concluded?
- Why should an assessed sample be projected using reference-derived loadings rather than casually comparing coordinates from two separately fitted PCAs?
- A sample’s squared Mahalanobis distance is 7.2 on two components. Using the 95th-percentile chi-square cutoff of 5.991, how does it compare with that cutoff, and what qualification is needed?
- At a marker, two hypothetical groups have frequencies 0.10 and 0.15, with 100 independent allele copies in each. Calculate the difference and its approximate standard error.
- Why is “the sample is an outlier, therefore its ancestry is incorrect” an invalid interpretation?
Solutions. (1) The sample is poorly represented relative to the narrower reference under that analysis; the conclusion is reference-dependent and does not identify ancestry. (2) Separate PCA fits can have different axes, scales, and origins, so their scores do not share a common coordinate system without a defined alignment. (3) Since \(7.2>5.991\), it exceeds the model-based 95% cutoff for two components; this interpretation requires a suitable score-distribution model and a justified calibration of the reference parameters. (4) The difference is \(0.10-0.15=-0.050\). The approximate standard error is \(\sqrt{0.10(0.90)/100+0.15(0.85)/100}=\sqrt{0.002175}\approx0.04664\). (5) Outlier status describes distance from a chosen reference, not the sample’s identity or the cause of its genetic difference; valid variation, reference limitations, or technical factors may contribute.