Common Errors in Ancestry and Relatedness Analysis

Why These Errors Matter

An ancestry or relatedness analysis can produce plausible-looking plots and estimates while answering the wrong question. A cluster in a PCA plot may reflect a technical artifact or a heavily sampled genomic region; a large relationship estimate may partly reflect allele-frequency differences between populations. Such errors are consequential because downstream analyses can then adjust for the wrong structure, misidentify close relatives, or treat redundant genetic information as independent evidence.

The central discipline is to keep four choices distinct: how genotypes are encoded and scaled, how marker dependence is handled, what a PCA axis represents, and whether a similarity reflects ancestry or kinship. These choices interact, but none is a substitute for the others.

Genotype Scaling: Know What Each Marker Contributes

Definition: For diploid individual \(i\) and biallelic marker \(j\), let \(G_{ij}\in\{0,1,2\}\) be the number of copies of a chosen allele, or an estimated dosage between 0 and 2. If that allele has frequency \(p_j\), the centered dosage is \(G_{ij}-2p_j\). Under Hardy–Weinberg equilibrium, its variance is \(2p_j(1-p_j)\); the variance-standardized genotype is \[ Z_{ij}=\frac{G_{ij}-2p_j}{\sqrt{2p_j(1-p_j)}}. \]

Raw dosage, centered dosage, and standardized genotype answer different computational needs. Centering removes a marker's average dosage under the chosen frequency estimate. Standardization additionally gives markers with different expected genotype variances comparable scale under the stated model. For example, a common-marker dosage varies more in raw units than a dosage at a very rare marker, so centering alone does not make their variances equal.

The standardization formula relies on a suitable allele-frequency estimate and uses the Hardy–Weinberg variance as its scale. Population mixture, inbreeding, genotyping error, and very small frequency estimates can make that scale a poor description of the observed data. Standardization is therefore a modeling choice, not an automatic correction for every problem. In particular, dividing by a very small standard deviation can magnify noisy rare-allele calls.

Worked Example: Comparing Two Standardized Dosages

Suppose two hypothetical markers have estimated allele frequencies \(p_1=0.50\) and \(p_2=0.05\). For one individual, the dosages are \(G_{i1}=2\) and \(G_{i2}=2\). Calculate the centered and standardized values.

At marker 1, the expected dosage is \(2p_1=1\), so the centered dosage is \(2-1=1\). Its standard deviation under the model is \(\sqrt{2(0.50)(0.50)}=\sqrt{0.50}\approx0.7071\). Thus \(Z_{i1}=1/0.7071\approx1.414\).

At marker 2, the expected dosage is \(2p_2=0.10\), so the centered dosage is \(2-0.10=1.90\). Its model standard deviation is \(\sqrt{2(0.05)(0.95)}=\sqrt{0.095}\approx0.3082\). Thus \(Z_{i2}=1.90/0.3082\approx6.165\).

The rare-allele dosage is much farther from its frequency-based expectation in standardized units. That is the intended consequence of this scaling, but it also shows why an erroneous or unstable frequency estimate can give a rare marker excessive influence. The calculation does not establish that this individual has unusual ancestry or is related to anyone.

A common implementation error is to use \(\sqrt{p_j(1-p_j)}\) in the denominator while treating \(G_{ij}\) as a diploid dosage. That denominator omits the factor of 2 in the genotype variance. The resulting standardized values are larger by \(\sqrt{2}\) than the values defined above. A global scale change may leave some PCA directions unchanged, but it changes reported eigenvalues and can produce inconsistent results when combined with other marker weights. Check the dosage convention, allele-frequency convention, and denominator together.

Linkage Disequilibrium and Unequal Marker Influence

Linkage disequilibrium (LD) means that alleles at different markers are statistically associated in the analyzed sample or population. PCA and many genomic relationship calculations combine information across markers. When many markers in one region are strongly correlated, they do not provide as many independent pieces of information as the same number of independent markers. If each is included with equal weight, that region can contribute repeatedly to the same underlying signal.

For illustration, with \(m\) retained markers and standardized genotypes, a common sample relationship summary has entries

K_{ik}=\frac{1}{m}\sum_{j=1}^{m} Z_{ij}Z_{kj}.

This expression is a useful illustration, not a universal definition of every relationship estimator. It makes the weighting issue clear: each marker contributes a product. A block of correlated markers can therefore give its shared signal more total weight than a block represented by one marker.

Worked Example: Duplicating a Correlated Marker Block

Consider three hypothetical individuals and one centered, standardized marker represented by the column vector \(z=(1,-1,0)^\mathsf{T}\). Its contribution to the unnormalized individual cross-product matrix is \(zz^\mathsf{T}\):

zz^\mathsf{T}= \begin{pmatrix} 1&-1&0\\ -1&1&0\\ 0&0&0 \end{pmatrix}.

If the same signal is represented by \(k\) identical markers, the summed cross-product contribution is \(kzz^\mathsf{T}\), because \[ \sum_{j=1}^{k}zz^\mathsf{T}=kzz^\mathsf{T}. \] For \(k=5\), the first diagonal contribution is 5 rather than 1, and the off-diagonal contribution for individuals 1 and 2 is \(-5\) rather than \(-1\). If the matrix is averaged over all markers, the final size also depends on how many other markers are included; nevertheless, the repeated signal receives five marker contributions.

Real LD markers are rarely exact duplicates, but the same logic applies approximately. A dense correlated region can dominate a component or relationship summary. This example demonstrates weighting, not a rule that every LD block must be deleted.

LD pruning—retaining a subset of markers with reduced pairwise correlation—is one practical way to limit redundant influence, especially when PCA is intended to summarize broad population structure. It can discard useful information, however, and the appropriate pruning thresholds depend on the analysis. Alternatives include analyzing a prespecified set of approximately independent markers, weighting markers to account for LD, or using an estimator designed for LD. Applying no correction is also a choice, and should be justified rather than treated as neutral.

Useful diagnostics include comparing PCA results before and after pruning, checking whether leading loadings concentrate in one genomic region, and examining whether a few long-range LD regions drive the apparent separation. If a major axis changes substantially when a correlated region is removed, describe that sensitivity. Do not conclude that the pruned result is automatically correct: pruning may remove real structure as well as redundancy.

PCA Axes Are Summaries, Not Ancestry Labels

PCA finds directions that summarize variation in the analyzed genotype matrix. With centered genotype matrix \(X\), a principal component is a direction chosen to capture sample variation; successive directions are constrained to be orthogonal to earlier ones. If the matrix is standardized or the marker set changes, the resulting directions can also change. PCA does not, by itself, identify a person's ancestry, identify a causal process, or determine whether two people are close relatives.

An axis can correlate with ancestry, but it can also reflect batch effects, missingness, LD structure, or the particular individuals and markers included. The sign of an axis is arbitrary: multiplying all its scores by \(-1\) yields the same component. The numerical position and orientation of a PCA plot therefore have no inherent labels such as “northwest ancestry” or “more related.” Interpretation requires external context and quality checks.

Worked Example: Reading Variance Explained Without Overclaiming

Suppose a hypothetical PCA analysis reports eigenvalues 4.0, 2.0, and 1.0 for the first three components, and the total variance represented by all components is 20.0. The variance proportions are \[ 4.0/20.0=0.20,\qquad 2.0/20.0=0.10,\qquad 1.0/20.0=0.05. \] Thus PC1 accounts for 20% of the variance measured in this analysis, PC2 for 10%, and PC3 for 5%.

These percentages do not mean that PC1 represents 20% of a person's ancestry, nor that PC1 is necessarily the most biologically meaningful axis. Suppose, further, that samples processed in two laboratory batches separate on PC1. The association could reflect batch-related variation rather than ancestry. A plot colored by batch, missingness summaries, and sensitivity to marker selection would be relevant diagnostics. If an independently justified ancestry label also aligns with PC1 after those checks, the axis may be useful as a summary of that contrast—but the interpretation is supported by the evidence, not supplied by PCA itself.

Ancestry Similarity Is Not the Same as Kinship

Kinship concerns genetic relationships through shared descent. In particular, identity by descent (IBD) refers to alleles inherited from a common ancestor. Ancestry describes population history and allele-frequency patterns. People from a similar ancestry background may resemble each other at many markers because their allele frequencies are similar, even when they are not close relatives. Conversely, close relatives can have different positions in a PCA plot if their broader population backgrounds differ.

This distinction matters for genomic relatedness estimation. Many estimators compare observed genotype sharing with an expectation based on allele frequencies. If the reference frequencies do not adequately represent population structure, ancestry differences can distort the estimate. The earlier material on separating ancestry similarity from close relatedness motivates modeling or adjusting for structure; the key point here is that neither a PCA distance nor an unadjusted genotype similarity alone establishes kinship.

Worked Example: Interpreting a Kinship Estimate in Context

Suppose a hypothetical analysis reports estimated kinship \(\hat{\phi}=0.012\) for two people. A commonly used expectation for first cousins is \(\phi=1/16=0.0625\), while unrelated people in a well-matched reference population are often expected to have a value near zero under the estimator's convention. The reported value is below the first-cousin expectation, but that comparison alone does not classify the pair.

Now suppose the pair also lies close together on the first two PCs, and both belong to an ancestry group whose allele frequencies differ from the reference used for estimation. The PCA proximity is consistent with similar population background, but it is not independent proof of a close relationship. The estimated kinship may also be biased by the frequency mismatch. Appropriate checks include verifying allele-frequency choices, examining estimates with a structure-robust method, checking genotype quality, and comparing with other evidence such as IBD-based diagnostics when available.

A defensible conclusion is that the pair shows a modest estimated genotype similarity under the specified analysis, with ancestry-related bias requiring assessment. It is not justified to call them first cousins or to claim that PCA proximity establishes kinship.

Diagnostics, Alternatives, and Limits

A reliable analysis makes preprocessing and interpretation decisions visible. Before treating PCA scores or kinship estimates as evidence, inspect the genotype encoding, allele-frequency estimates, missingness, marker set, and LD structure. Then check whether conclusions are stable under reasonable alternatives. Stability is evidence of robustness, not proof that all model assumptions hold.

Verify genotype conventions.
Confirm that dosages use a consistent allele definition, that diploid dosage is on the 0-to-2 scale, and that centering and standardization use the intended frequency and variance.

Audit marker dependence.
Check LD and genomic regions with high PCA loadings. Compare the target summary across an LD-pruned marker set and a justified alternative, documenting what changed.

Challenge the PCA interpretation.
Compare scores with technical covariates and relevant external labels. Treat axes as sample- and preprocessing-dependent summaries, not intrinsic ancestry coordinates.

Evaluate kinship separately.
Check frequency sensitivity and population structure, and use evidence suited to close relationships. Do not convert PCA proximity directly into a relationship category.

Each diagnostic has limits. LD pruning may remove a region that genuinely carries population information; unpruned data may let correlated regions dominate. Standardization can make rare alleles influential, while raw or merely centered genotypes can let common, high-variance markers dominate. PCA is descriptive and can expose patterns without explaining their cause. Kinship estimators depend on the genotype data, allele-frequency model, and assumptions about population structure. Report the choices and uncertainty that matter for the intended interpretation.

Common Mistakes to Avoid

Using a familiar formula with the wrong dosage scale. A variance formula for allele counts is not interchangeable with one for diploid genotype dosage. Match the denominator to the genotype definition.
Assuming thousands of markers mean thousands of independent signals. Correlated markers may repeatedly weight the same signal. Inspect LD and compare justified marker-selection or weighting approaches.
Naming a PC as an ancestry category from its plot alone. A component maximizes variation in the analyzed data; validate its interpretation against technical and external information.
Calling similar ancestry “relatedness.” Shared population background can create allele-frequency similarity without recent shared descent. Use kinship evidence and account for structure.
Treating one diagnostic as a verdict. A stable PCA plot or plausible estimate does not guarantee that scaling, LD, missingness, and frequency assumptions are all adequate.

Key takeaway: Genotype scale determines marker contribution, LD determines how redundant contributions accumulate, PCA describes variation rather than labeling its cause, and ancestry similarity is not equivalent to kinship. Check each issue directly before interpreting results.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

For \(p=0.20\) and dosage \(G=2\), calculate the centered dosage and variance-standardized genotype using the diploid dosage convention.
Why can a highly correlated block of 20 markers affect a PCA more than one independent marker, even if it represents only one broad signal?
A PC separates samples and explains 15% of the total variance. What does that percentage establish, and what does it not establish?
Two people are close in a PCA plot and have an estimated kinship value above zero. Give one reason not to conclude immediately that they are close relatives.

Solutions. (1) The centered dosage is \(2-2(0.20)=1.60\). The standard deviation is \(\sqrt{2(0.20)(0.80)}=\sqrt{0.32}\approx0.5657\), so \(Z=1.60/0.5657\approx2.828\). (2) Each correlated marker contributes a term to the covariance or relationship summary, so repeated correlated contributions can give the shared signal greater weight. (3) It establishes that the component accounts for 15% of the variance under that analysis; it does not establish that the component measures ancestry, a biological cause, or 15% of anyone's ancestry. (4) Their similarity may reflect shared ancestry or a frequency-reference mismatch rather than close IBD sharing; the kinship estimate and its assumptions need separate evaluation.