Population Structure Artifacts In Relatedness Estimates

Why Population Structure Can Look Like Relatedness

A pairwise relatedness estimate is intended to summarize genetic sharing, but many estimators also depend on how genotypes are centered and scaled using allele frequencies. If the two people come from groups with different allele frequencies, a frequency reference for the combined sample may center neither group correctly. Their genotype deviations can then align systematically, even when the individuals are unrelated within their groups.

This is a feature of the estimator, not evidence that ancestry groups are pedigrees. Identity by state describes alleles that match in observed genotypes; identity by descent describes alleles inherited from a shared ancestor. A population-level frequency difference can create excess or reduced identity by state without establishing recent shared ancestry. The distinction matters whenever a relatedness estimate is used to describe a pair, rather than merely to summarize genotype similarity.

Definition: A population structure artifact in a relatedness estimate occurs when allele-frequency differences among sampled groups shift a pairwise estimate away from its interpretation under a common-population reference, even though the shift is not due to the pair's pedigree sharing. The direction and size of the shift depend on the estimator, the frequency reference, the groups being compared, and the markers included.

How Frequency Centering Creates a Shift

Consider a biallelic marker \(l\). Let \(g_{il}\) be the number of counted alleles for individual \(i\), so \(g_{il}\) is 0, 1, or 2. Let \(p_l\) be the allele frequency used as the reference. A common frequency-standardized off-diagonal genomic relationship estimate has the form

\widehat{G}_{ij} = \frac{\sum_l (g_{il}-2p_l)(g_{jl}-2p_l)} {\sum_l 2p_l(1-p_l)}.

This is a marker-weighted average of centered genotype products. Its intended baseline is near zero for unrelated individuals sampled from a population whose allele frequencies are represented by \(p_l\), subject to the estimator's assumptions and sampling variation. It is not a direct count of shared ancestors. In particular, a nonzero value can result when the frequency reference is inappropriate for the pair.

To see the mechanism, suppose individuals \(i\) and \(j\) are independently sampled from groups \(A\) and \(B\). At one marker, write their group-specific allele frequencies as \(p_A\) and \(p_B\), and suppose the estimator uses a pooled frequency \(\bar p\). Under random mating within each group, the conditional mean genotype counts are \(2p_A\) and \(2p_B\). Conditional independence gives

\begin{aligned} E[(g_i-2\bar p)(g_j-2\bar p)\mid A,B] &= E[g_i-2\bar p\mid A]\ E[g_j-2\bar p\mid B]\\ &=4(p_A-\bar p)(p_B-\bar p). \end{aligned}

Thus the expected product is positive when both groups lie on the same side of the pooled frequency and negative when they lie on opposite sides. If the groups contribute equally to the pooled sample, \(p_A-\bar p\) and \(p_B-\bar p\) have opposite signs for a cross-group pair. Across many markers, these small contributions can accumulate. This calculation assumes independent individuals within their groups at the marker; it isolates the frequency-centering effect and does not model pedigree sharing, linkage between markers, or other sources of dependence.

Worked Example: A Pooled Reference Makes Unrelated Pairs Look Different

Suppose a hypothetical cohort has two equally sized groups. At each of two independent markers, the counted-allele frequency is 0.8 in group A and 0.2 in group B. The pooled reference is therefore \(\bar p=(0.8+0.2)/2=0.5\). Assume the individuals in each pair are unrelated and independently sampled within their stated groups.

For a pair from group A, the expected centered product at either marker is \[ 4(0.8-0.5)(0.8-0.5)=4(0.3)(0.3)=0.36. \] The denominator contribution is \(2(0.5)(1-0.5)=0.50\). With two markers, the expected numerator is \(2(0.36)=0.72\), and the denominator is \(2(0.50)=1.00\). The expected standardized estimate is \(0.72/1.00=0.720\).

For a cross-group pair, the expected product at either marker is \[ 4(0.8-0.5)(0.2-0.5)=4(0.3)(-0.3)=-0.36. \] The two-marker estimate is \([2(-0.36)]/[2(0.50)]=-0.720\). These are expected values under the simplified model, not guaranteed estimates for any realized pair. Their contrast shows how the same pooled reference can yield positive within-group and negative cross-group estimates without pedigree sharing.

Choosing a Frequency Reference

A frequency reference should match the population represented by the comparison as closely as is defensible. A pooled reference is convenient, but it can be misleading when the analyzed sample contains groups with appreciably different allele frequencies. A group-matched reference can reduce this centering artifact for comparisons within a well-defined group. It does not make every estimate universally comparable: estimates standardized with different references may have different scales, and a reference estimated imprecisely can add noise.

For comparisons across groups, simply selecting one group's frequencies is not a general solution. It centers one group at its expected genotype mean but leaves the other group's genotypes shifted. An ancestry-aware approach may use group-specific frequencies or model ancestry contributions, but it needs reliable ancestry information and a clearly defined target for the estimate. In admixed samples, a single global label may not capture the ancestry composition relevant at each marker. The ancestry-estimation material earlier in the course emphasizes why labels and estimates are not interchangeable with a person's full genetic background.

Changing the reference is therefore a diagnostic and modeling decision, not a way to turn an artifact into a pedigree coefficient automatically. Before comparing estimates, check which frequencies were used, which individuals informed them, and whether the same reference and marker set were applied to all pairs.

Worked Example: Comparing Pooled and Group-Matched Centering

Consider one hypothetical marker with frequency 0.7 in group A and 0.3 in group B. The two groups have equal sample sizes, so the pooled frequency is 0.5. For two independently sampled, unrelated individuals from group A, the expected unstandardized product under the pooled reference is \[ 4(0.7-0.5)^2=4(0.2)^2=0.16. \] The standardized contribution is \(0.16/[2(0.5)(0.5)]=0.16/0.50=0.320\).

If the same pair is instead centered using the group-A frequency, the expected product is \[ 4(0.7-0.7)^2=0. \] The denominator is \(2(0.7)(0.3)=0.42\), so the expected standardized contribution is \(0/0.42=0\). The group-matched reference removes this particular between-group centering shift for a within-A pair under the stated assumptions.

This does not imply that group matching is always preferable. If group A is small, its frequency estimate may be unstable; if the pair is admixed or its group assignment is uncertain, the assumed reference may not fit. Nor does a zero expectation guarantee a zero realized estimate. The example isolates the change in expectation caused by the reference choice.

Robust Estimators: Helpful, Not Structure-Proof

A robust pairwise estimator can reduce dependence on an external pooled allele-frequency reference. KING-robust kinship estimation, introduced earlier in the course, uses genotype-sharing patterns designed to be less sensitive to population structure than frequency-standardized methods. Its appeal is that the estimate is not centered by inserting one pooled \(p_l\) into every genotype product.

“Robust” does not mean immune to all structure or suitable for every comparison. Performance depends on the estimator's assumptions, the ancestry composition of the pair, marker properties, and the degree of genetic differentiation. A robust estimate and a frequency-standardized estimate may disagree because they use different information and assumptions; disagreement is a diagnostic signal to investigate, not proof that one number is correct. In particular, neither estimate should be translated directly into a pedigree relationship without considering its method and context.

Worked Example: Using Estimator Disagreement as a Diagnostic

Suppose a hypothetical cohort analysis reports a pooled-frequency genomic relationship estimate of 0.24 for a pair, while a robust estimator reports 0.01. Before interpreting the first value as evidence of close pedigree sharing, the analyst checks the pair's ancestry estimates, allele-frequency reference, and marker set. The individuals are assigned to different sampled groups, and the pooled frequencies differ substantially from the frequencies within either group.

The pooled estimate is an observed output, not a direct proof of relatedness. The earlier expectation calculation shows that frequency shifts can produce nonzero centered products for unrelated cross-group individuals. The robust estimate is closer to zero and is less dependent on the pooled reference, which makes the structure explanation plausible. But this comparison alone does not establish that the individuals are unrelated: the estimators have different assumptions, and the data's uncertainty and marker dependence still matter.

A defensible conclusion is therefore conditional: the pooled estimate may be inflated or distorted by population structure, so the pair requires review under a reference and estimator appropriate to the analysis. It is not defensible to report that the pair shares a particular pedigree relationship based only on 0.24, or to treat 0.01 as definitive proof of no relationship.

A Practical Diagnostic Sequence

When a pairwise estimate appears unexpectedly large, small, or negative, separate the question “what does this estimator calculate?” from “what biological relationship does this value support?” A useful review is:

Check the reference frequencies.
Identify whether frequencies are pooled, group-specific, or estimated in another reference sample. Ask whether that reference represents both members of the pair.

Inspect group composition.
Compare the pair's ancestry estimates and the cohort's group composition. A broad ancestry label is a diagnostic summary, not a pedigree statement or a complete description of ancestry at every marker.

Compare a suitable alternative.
Where appropriate, compare the frequency-standardized result with a robust estimator or a result using a defensible group-matched reference. Keep track of changes in scale and marker set.

Report the limitation explicitly.
Describe the estimate as method- and reference-dependent when structure remains plausible. Do not recast an ancestry-driven genotype pattern as evidence of pedigree sharing.

Common Mistake: Treating a Nonzero Estimate as a Family Relationship

The wrong reasoning is: “The pair's genomic relationship estimate is positive, so they must be relatives.” This skips the estimator's frequency reference and confuses a pattern of genotype similarity with identity by descent. A positive estimate can arise because both individuals' genotypes deviate in the same direction from a pooled frequency reference. Conversely, a negative cross-group estimate is not evidence of biological antagonism or a negative pedigree relationship; it may reflect opposite deviations from that reference.

The correction is to interpret the estimate conditionally: name the estimator and frequency reference, check whether the pair's group composition makes the reference appropriate, and compare results using a defensible alternative when possible. Then distinguish evidence of population-level structure from evidence about the pair's pedigree. Structure diagnostics do not themselves establish a relationship; they identify when the estimate needs a more careful interpretation.

Key takeaway: Frequency-standardized relatedness estimates can reflect both pairwise sharing and systematic group differences. A pooled reference can shift within-group estimates upward and cross-group estimates downward. Reference choice and robust estimators help diagnose this problem, but no estimate should be treated as ancestry-independent proof of pedigree sharing.

Check Your Understanding

Use the assumptions and distinctions in this tutorial to answer the following.

For independently sampled individuals from groups with frequencies \(p_A\) and \(p_B\), what is the expected centered genotype product when the reference is \(\bar p\)?
Why can a pooled reference yield a positive within-group estimate for unrelated individuals?
What limitation remains when a frequency reference is matched to one group but used for a cross-group pair?
What does disagreement between a pooled-frequency estimate and a robust estimate suggest, and what does it not prove?
Why is a negative cross-group estimate not evidence of a negative pedigree relationship?